bpo-41796: Make _ast module state per interpreter

[vstinner]

The ast module internal state is now per interpreter.

• Rename "astmodulestate" to "struct ast_state"
• Add pycore_ast.h internal header: the ast_state structure is now
  declared in pycore_ast.h.
• Add PyInterpreterState.ast (struct ast_state)
• Remove get_ast_state()
• Rename get_global_ast_state() to get_ast_state()
• PyAST_obj2mod() now handles get_ast_state() failures

https://bugs.python.org/issue41796

[vstinner]

In short, this change replaces global_ast_state of Python-ast.c with PyInterprreterState.ast. It removes:

// [bpo-41194](https://bugs.python.org/issue41194), [bpo-41261](https://bugs.python.org/issue41261), [bpo-41631](https://bugs.python.org/issue41631): The _ast module uses a global state.
static astmodulestate global_ast_state = {0};

cc @encukou @pablogsal @lysnikolaou

[vstinner]

Oh, I missed test_peg_generator. I should now be fixed with my second commit.

[lysnikolaou]

Should the Windows regen file (regen.vcxproj) be updated as well?

[vstinner]

@isidentical: Thanks for the review. I addressed your review. Please review the updated PR.

[vstinner]

@lysnikolaou: "Should the Windows regen file (regen.vcxproj) be updated as well?"

Oh, I didn't know this project. Thanks, I updated it.

[vstinner]

I merged my PR. Thanks for the useful review @lysnikolaou and @isidentical!

I dislike having to declare the struct ast_state once in pycore_ast.h and once in Python-ast.c, but Python-ast.c is a strange beast which is used outside Python without Py_BUILD_CORE. So declaring the structure twice is a practical solution for that.

@encukou used a different solution: use a pointer to an opaque structure in the PyInterpreterState, and allocate the structure on the heap memory when it's initialized. That would avoid to declare the structure twice. But I chose to follow the current trend of declaring all structures in pycore_interp.h and have a large unique and flat PyInterpreter structure (no pointer indirection).

Maybe tomorrow we should revisit this design to have something more flexible, to not have to declare everything in pycore_interp.h.

The best would be to be able to have a real module state, but I was biten multiple times by the C API which requires the AST module state, whereas the C API isn't directly related to a module. I tried a solution using an import and we got a crash with Mercurial lazy import, etc.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 LTO 3.x has failed when building commit 5cf4782.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/567/builds/926) and take a look at the build logs.
4. Check if the failure is related to this commit (5cf4782) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/567/builds/926

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

409 tests OK.

10 slowest tests:

• test_concurrent_futures: 3 min 3 sec
• test_peg_generator: 2 min 58 sec
• test_gdb: 1 min 46 sec
• test_multiprocessing_spawn: 1 min 12 sec
• test_nntplib: 1 min 6 sec
• test_multiprocessing_forkserver: 1 min 1 sec
• test_signal: 55.7 sec
• test_multiprocessing_fork: 54.4 sec
• test_asyncio: 47.1 sec
• test_io: 35.2 sec

1 test altered the execution environment:
test_asyncio

14 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_nis
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

Total duration: 4 min 49 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/sslproto.py", line 321, in __del__
    self.close()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/sslproto.py", line 316, in close
    self._ssl_protocol._start_shutdown()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/sslproto.py", line 590, in _start_shutdown
    self._abort()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/sslproto.py", line 731, in _abort
    self._transport.abort()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/selector_events.py", line 680, in abort
    self._force_close(None)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/selector_events.py", line 731, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/base_events.py", line 746, in call_soon
    self._check_closed()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.lto/build/Lib/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

[pablogsal]

I am relatively sure that this commit introduced reference leaks in the test suite:

https://buildbot.python.org/all/#/builders/205/builds/83

OK
......
test_ast leaked [23640, 23636, 23640] references, sum=70916
test_ast leaked [7932, 7930, 7932] memory blocks, sum=23794
1 test failed again:
test_ast

See also https://bugs.python.org/issue42250