Remove the _PyOptimizer APIs

[brandtbucher]

We should rip the _PyOptimizer API out, and just keep all of the code that does that actual optimizing.

The API itself is unstable, constantly changing, poorly-defined, and undocumented. There's a bunch of infrastructure required just to test the API (not the actual optimizations we perform), and it introduces indirection and artificial boundaries into some pretty performance-sensitive stuff. Nobody is using it that we're aware of, nobody we've talked to is planning on using it, and frankly we don't want anyone to start using it. So let's remove it.

Linked PRs

• gh-126599: Refactor the _PyOptimizer APIs #126853

[xuantengh]

Hi can I take and try this issue?

[rruuaanng]

Don't rush, please wait for the discussion result :)

[brandtbucher]

This has already been discussed quite a bit offline with the relevant parties. I think we can probably move ahead with it.

It's a pretty big project, but help is welcome if you really feel confident tackling it. It's not just deleting things, but also refactoring the code to remove the current API boundary between the interpreter and the optimizer. I'm happy to help out if you get stuck or need any advice, too.

[rruuaanng]

@xuantengh Would you like to challenge?

If you don't have the confidence to give it to me, I will finish it when I have time.

[xuantengh]

Would you like to challenge?

Hi I would like to give it a try.

[xuantengh]

Hi @brandtbucher, after some preliminary investigation, this issue aims to remove the following 3 internal C APIs:

• _PyOptimizer_NewCounter
• _PyOptimizer_NewUOpOptimizer
• _PyOptimizer_Optimize

where the first two are only used for tests (_testinternalcapi.c), and it's trival to rip them. For the last, it's used in instruction JUMP_BACKWARD as well as tier2 OPs _EXIT_TRACE and _DYNAMIC_EXIT. So we need to refactor the code to let these instruction/OPs directly conduct the optimizations?

Please correct me if something wrong or missing.

[rruuaanng]

@xuantengh Maybe you should submit a PR :)