[FEAT] Should include more comprehensive benchmarking (primarily performance)

[EricCousineau-TRI]

Motivation

At present (b7dfe5c), pybind11 proper only benchmarks compile-time and and artifact size for one given test setup (which tests arguments, simple inheritance, but that's about it, I think); the results of which can be seen here:
https://pybind11.readthedocs.io/en/stable/benchmark.html
https://github.com/pybind/pybind11/blob/v2.6.1/docs/benchmark.rst

However, it may difficult to objectively and concretely judge the performance impact of a PR, and weigh that against the value of the feature / issue resolution. Generally, benchmarking is done on an ad-hoc basis (totes works, but may make it difficult for less creative people like myself ;)

Primary motivating issues / PRs:

• Detect and fail if using mismatched holders #2644 (comment)
• [FEAT] Rework casting #2646
• [BUG] Problem when creating derived Python objects from C++ (inheritance slicing) #1333

Secondary:

• Would you consider merging a patch that optionally splits pybind11 into declarations vs definitions in a backwards compatible opt-in manner? #2322
• why pybind11 is slower than cython #1227 / why pybind11 is slower than Python and boost python; the benchmark does not include performance test? #1825
• Python multiple inheritance #693 (comment)
• Iterators efficiency and instance tracking #376

Fuzzy Scoping + Steps

1. Establish a baseline benchmarking setup that touches on the core "hotpath" features for performance. Only track metrics for a given version of pybind11 (out of scope: other binding approaches)

• Initialization time (e.g. dlopen - ish stuff, pybind11 internals upstart, binding registration, ...)
• Run time (function calls, type conversions, casting, etc.)
• Comparison "axes": later CPython versions, pybind11 versions / PR branches / forks
• OS: For now, just Linux ('cause that's all I use ;)

2. Add additional metrics (e.g. memory leaks / usage, do a redux on compile-time + size)
3. Ideally, provide guidance on what pybind11 finds the most important (how to weigh compile-time, size, speed, memory, etc.)
4. (Stretch) Possibly compare against other approaches (Boost.Python, Cython, SWIG, cppyy / Julia / LLVM-esque stuff, etc.)

Given that performance benchmarks can be a P.I.T.A. (e.g. how to OS + interrupts, hardware capacity / abstractions, blah blah), ideally decisions should be made about relative performance on the same machine. Ideally, we should also publish some metrics for a given config to give people a "feel" for the performance, as was done for compile time.

Suggested Solution Artifacts

• Identify code repository
  • Perhaps github.com/pybind/pybind-benchmarks ?
  • Alt.: In-tree (which may make it hard to compare across versions...)
• Identify good tooling for performance benchmarking
  • @henryiii suggested pytest-benchmark

@wjakob @rwgk @rhaschke @YannickJadoul @bstaletic @henryiii @ax3l
Can I ask what y'all think? Is this redundant w.r.t. what we already have?

[rwgk]

This code was used for Google-internal micro-benchmarking:
https://drive.google.com/file/d/1EOGU_A28oBvzoLwdmo2RpImjyL6bv2rl/view?usp=sharing
The link is accessible only to select people. pybind11 maintainers have access already.
The original author is @kkimdev. He generously gave us permission to reuse what's useful for our purposes (under the OSS pybind11 org). I think we will need major adaptions for the code to work outside the Google environment.

[rwgk]

Here is a doc with background and results of the micro-benchmarking:
https://docs.google.com/document/d/1UieJ9WZ9YVJLt_EsIYj4Ahw8OIb3fH6mOIAPQRzj46k/
The link is accessible only to the same group of people. The doc is meant to inform our work on benchmarks. Please do not make copies.
As before, the original author is @kkimdev.

[kkimdev]

A benchmarking tip: I think combining C++ sampling profiler with Python benchmark is extremely useful. My latest internal benchmark script (not the above one) supports --run_pprof=True flag that runs pprof sampling profiler (but not sure if the open source version is as good as the internal, and probably there are other good open source C++ sampling profilers as well) and reports it. The flame graphs in my doc are from that.

[YannickJadoul]

This is a bit annoying if external contributors would feel a calling to help out.

Future people without access, contributions and ideas are still welcome; sharing that code is up to the authors, but ideas can be discussed :-)

[rwgk]

Yes, sorry, but getting clearance to make this info fully public is likely quite a bit of trouble. But we can add interested people, and we can summarize or report what we learn here.

…

On Wed, Dec 30, 2020 at 14:57 Yannick Jadoul ***@***.***> wrote: This is a bit annoying if external contributors would feel a calling to help out. Future people without access, contributions and ideas are still welcome; sharing that code is up to the authors, but ideas can be discussed :-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2760 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFUZAAKRWQM57UBQPTANOLSXOV6HANCNFSM4VOVTTTA> .

[YannickJadoul]

No worries, @rwgk, I completely understand!
Just trying to make sure that interested users/contributors (and maybe future maintainers ;-) ) aren't scared off :-)

[EricCousineau-TRI]

Also looking at the following to gauge how to possibly benchmark the CPython / PyPy low-level bits - for stuff like #2050-type stuff, in addition Kibeom's suggestion for pprof:

• https://pyperformance.readthedocs.io/,
• https://speed.python.org/

[wlav]

Since cppyy was mentioned: I use pytest-benchmark as well (see: https://bitbucket.org/wlav/cppyy/src/master/bench/). It's hard to write good benchmarks, though, as features and defaults differ. For example, releasing the GIL by default is costly for micro-benches (and for large, costly, C++ functions, the bindings overhead doesn't matter). Another big expense is object tracking for identity matching on returns, which not all binders do (and is useless for micro-benches).

For real high performance, the processor matters as well. For example, PyPy has guards on object types in its traces, based on which a specific C++ overload selected by cppyy will be compiled in. On a processor with good branch prediction and a deep out-of-order execution queue, that overhead will not show up in wall clock time (assuming no hyper-threading, of course), but it will be measurable on a processor with simpler cores.

When sticking to CPython only, consider also that CFunction objects have seen a massive amount of support in the form of specialized tracks through the CPython interpreter since release 3. (This is what makes SWIG in "builtin" mode, not the default, absolutely smoke everything else.) Only since 3.8 have closures seen some love, with the API stabilized in 3.9. There's a 30% or so reduction in call overhead in there somehow (for cppyy), but it's proving to be quite a lot of work to implement.

That last point, CPython internal developments and the need to track/make use of them, is also why I'd be interested if the proposed benchmarks end up being made public. The only way of measuring the usefulness of such changes is by having a historic record to compare against and setting that up is quite some effort (esp. when switching development machines regularly).

[EricCousineau-TRI]

See also: Benchmarking by @sloretz for usability, basic RAM analysis, etc., comparing a popular binding options (boost.python, cffi, cppyy, raw CPython, pybind11, SIP, SWIG) for the rclpy library: ros2/rclpy#665
https://docs.google.com/document/d/1BsTlxgzAkpX-5Q7FW6_Hyw1fszDEQef_OEvYmdnrBrE/edit#

Wim, not sure if this is what you meant by improvements in 3.8, but Shane also pointed to possible optimizations via PEP 590: Vectorcall (I dunno if pybind11 already leverages this directly but I see it crop up in this search: https://github.com/pybind/pybind11/issues?q=vectorcall+)

[wlav]

Yes, I was referring to vectorcall. The problem with its implementation is that it creates several distinctive paths, but then squeezes them through a single call interface. The upshot is very branchy code, more memory allocations to track, etc. Not the end of the world, but in the particular case of cppyy, certain code paths such as the handling of self were done rather differently from what the PEP 590 folks had in mind, so that bolting on vectorcalls as-is leads to unmaintainable code. Hence, I'm rewriting the calling code wholesale to fit vectorcalls better (simplifying memory tracking etc.) and this is proving quite a bit of work, but well worth it (already have a 20% reduction in call overhead, so am confident to hit that 30% when all is said and done).

Not sure about PyBind11, but with its greater call overhead, the benefits will be far less and there is the additional problem of requiring an extra pointer data member per bound object, so vectorcalls do come with a clear memory cost that can only be worth it if the CPU gains are big enough. But then, if the C++ side does substantial work in each call, then the call overhead matters little to nothing (see e.g. the benchmarking text you quote; myself, I benchmark mostly with empty functions), thus leaving that extra pointer as pure waste.

(Aside, the way I currently see it, storing the dispatch pointer as data member in the proxy is something that's only useful for C, not for C++.)

[EricCousineau-TRI]

From @wjakob - use perf to count instructions.