gh-116291: Tier 2 - True function inlining redux #116290

Fidget-Spinner · 2024-03-03T20:33:44Z

This PR adds true function inlining to CPython's tier 2 optimizer. It's a reduced form without frame reconstruction. See design discussion here faster-cpython/ideas#653.

This passes the entire CPython test suite.

All topmost frame call bodies consisting of purely pure ops are inlined. E.g.

def foo(a, b):
   return a + b

(only if type propagation works for the above properly)

If we really want to boost this, we should do loop peeling to remove as many guards as possible just so we can inline stuff.

I also forgot to add @gvanrossum as co-author, so I will add him before I merge

Issue: Tier 2: True function inlining without reconstruction #116291

…on into tier2_inliner_redux

Fidget-Spinner · 2024-03-03T23:04:00Z

Microbenchmarks (non-scientific) with JIT enabled:

# Main
./python -m timeit -s """
def cast(x):
 return x
""" "cast(1)"
20000000 loops, best of 5: 13.6 nsec per loop


./python -m timeit -s """
from _typing import _idfunc
""" "_idfunc(1)"
50000000 loops, best of 5: 8.21 nsec per loop


# This branch
./python -m timeit -s """
> def cast(x):
>  return x
> """ "cast(1)"
50000000 loops, best of 5: 9.18 nsec per loop

Inlined functions have almost as low overhead as C API function calls. If we do a bit more cleanup passes I'm quite confident we can surpass C API function call performance.

Include/internal/pycore_frame.h

pablogsal · 2024-03-04T22:01:26Z

As for debuggers, when the PEP 669 interface for debugging is used, Tier 2 will be disabled for code objects into which the debugger steps or where the debugger sets a breakpoint, so that should be manageable.

Note that these debuggers I am talking about won't be able to call anything on the Python process. These are debuggers that operate from outside the process so they cannot call anything on the process being debugged, not to set breakpoints, or to enable PEP 669 interface. These debuggers will normally do only do one thing: read the process memory to show the current call stack (with maybe the locals). The use case are generally to debug crashes and frozen programs. In this sense they have the same problem as profilers except that they may be interested also on reading something other than just calls.

But I don't disagree that there are cases where the loss of information is a problem, and I hope your proposed solution helps address this. (I think from that solution I infer that the tooling you're thinking of only counts hot functions, not hot lines?)

They would report both: function calls and hotlines. My solution is a best effort that indeed will only shows function calls because it's also designed to be able to be used in the presence of the JIT, where getting function lines can be much more challenging. To allow you to get a better feel of what's currently reported by these tools, here are some screenshots from these tools (some are profilers some are external debuggers):

austin:

py-spy:

pystack:

gvanrossum · 2024-03-04T22:19:48Z

These are debuggers that operate from outside

Ah, I see. Yeah, instead of the inlined function's frame they would see only the caller's frame (with a few extra unexplained values on the stack).

I'm not sure how much of a problem this is going to be in practice just yet. Certainly not with the current version of inlining (which is not very powerful -- only a handful of uops are considered "pure", and we don't inline loops).

In the future (3.14 timeframe) we may be inlining more things (though probably not loops yet), and eventually we may have to let the user choose between the fastest optimization and the most transparency towards external profilers and debuggers. (In 3.13 you could make that decision at build time by turing on no-GIL mode, which disables the specializing part of Tier 1 and all of Tier 2 :-).

My own vision for call inlining is that it's mostly going to make "trivial" functions disappear -- a good example would be typing.cast(), which returns its second argument unchanged. I currently sometimes hesitate about putting that in because of the perceived overhead of even the most trivial function call -- if the optimizer could be trusted to make that disappear, I'd be less reluctant to use casts properly. Similarly, "trivial" attribute getter and setter functions might be optimized away to the extent that we will no longer need to advise people that using def get_x(self): return self.x is unpythonic. But yeah, in all those cases the optimized code will not be as transparent to external profilers and debuggers, so some users might want to disable or limit call inlining. Even so I don't believe those tools will suddenly be no longer useful -- they will just provide slightly less accurate data for some cases. But maybe I am myopic -- my C debugger often gets confused by stuff that the optimizer inlined (even when I passed -O0 -g to the compiler) and I just live with it. Others may hold Python to a higher standard.

pablogsal · 2024-03-04T22:54:25Z

My own vision for call inlining is that it's mostly going to make "trivial" functions disappear -- a good example would be typing.cast(), which returns its second argument unchanged. I currently sometimes hesitate about putting that in because of the perceived overhead of even the most trivial function call -- if the optimizer could be trusted to make that disappear, I'd be less reluctant to use casts properly. Similarly, "trivial" attribute getter and setter functions might be optimized away to the extent that we will no longer need to advise people that using def get_x(self): return self.x is unpythonic. But yeah, in all those cases the optimized code will not be as transparent to external profilers and debuggers, so some users might want to disable or limit call inlining.

Makes sense. Thanks for taking the time to explain all of this. I think is a complex problem and some kind of compromise would need to be made here, as it would be unwise to stop all these optimisations just to keep the status quo but on the other hand this is an area that is very easy to degrade with optimizations and I think it's very important to have something working here, even if it has some limitations (such as only showing function names and not lines some times). Maybe something we can do is having something that can warn the tool that some inlining is going on, so if the tool shows something incorrect the user can know that some optimizations may be going on.

But maybe I am myopic -- my C debugger often gets confused by stuff that the optimizer inlined (even when I passed -O0 -g to the compiler) and I just live with it. Others may hold Python to a higher standard.

Precisely! I think this is an area that a lot of people are frustrated about in C/C++. Note also that C and C++ can fix this a bit with debugging information in the binaries, but most distros don't ship it because it weighs a lot. For Python I think the problem may be that this is not a pain point currently so any change will be a degradation of the experience. I assume that at the end we need to find some form of compromise or allow the user to choose.

Include/internal/pycore_frame.h

… be executed

gvanrossum · 2024-03-06T20:29:25Z

Thanks! We will have to have a debate including @pablogsal and @markshannon before we merge this. Also, I have to carefully read the analysis code (so far I've only focused on the runtime).

Here are the benchmark results: https://github.com/faster-cpython/benchmarking-public/blob/main/results/bm-20240305-3.13.0a4%2B-2f9539b-JIT/README.md

It looks like it is now neither slower nor faster. It not being slower suggests that the previous disastrous (4% slower) result was due to the frame being 8 bytes longer (which would require more chunks for the datastack allocator and more noise whenever two frames aren't adjacent). It not being faster is fine with me, there are probably not many opportunities for this kind of inlining.

Another open issue: when we're in the inlined code, target is invalid: it points into the inlined code object but that is not the code object on the frame. Since we're not inlining instructions that can deopt or side-exit, this shouldn't be a problem, but I feel that we should probably set target to something blatantly invalid so that nobody gets confused by this when debugging. (This only applies to the Tier 2 interpreter -- in the JIT we have no idea where we are anyway. :-)

gvanrossum

Unfortunately this is only half a review -- there will be more, but I'm going offline for a while so here is what I've got so far. (Note especially _PRE_INLINE.)

gvanrossum · 2024-03-06T21:27:54Z