[3.13] Ensurepip fails when built with --enable-experimental-jit and --with-pydebug (Linux)

[JeffersGlass]

Bug report

Bug description:

When building for Linux/X64, the 3.13 branch will not run successfully run python -m ensurepip if built with both --enable-experimental-jit and --with-pydebug:

git checkout 3.13
make clean
./configure --enable-experimental-jit --with-pydebug
make -j8
./python -m ensurepip

This fails with an error message like:

subprocess.CalledProcessError: Command '['/home/jglass/Documents/cpython/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpsu81mj6o/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpsu81mj6o\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' died with <Signals.SIGABRT: 6>.

Building with either config flag individually successfully builds.

Results of ./python -VV: Python 3.13.0b2+ (heads/3.13:ff358616ddb, Jun 12 2024, 21:17:57) [GCC 11.4.0]

Built on Ubuntu 22.04; same failure on two separate machines. These options build successfully on main (030b452e).

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-120437: Fix _CHECK_STACK_SPACE optimization problems introduced in gh-118322 #120712
• [3.13] gh-120437: Fix _CHECK_STACK_SPACE optimization problems introduced in gh-118322 (GH-120712) #120747

[Eclips4]

Thanks for the report!
Confirmed on 3.13 branch.

[Eclips4]

Here's the full trace:

Assertion failed: (tstate->datastack_top < tstate->datastack_limit), function _PyFrame_PushUnchecked, file pycore_frame.h, line 281.
Traceback (most recent call last):
  File "/Users/admin/Projects/cpython/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
                     "__main__", mod_spec)
  File "/Users/admin/Projects/cpython/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^
  File "/Users/admin/Projects/cpython/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ~~~~~~~~~~~~~~~^^
  File "/Users/admin/Projects/cpython/Lib/ensurepip/__init__.py", line 257, in _main
    return _bootstrap(
        root=args.root,
    ...<4 lines>...
        default_pip=args.default_pip,
    )
  File "/Users/admin/Projects/cpython/Lib/ensurepip/__init__.py", line 172, in _bootstrap
    return _run_pip([*args, "pip"], [os.fsdecode(tmp_wheel_path)])
  File "/Users/admin/Projects/cpython/Lib/ensurepip/__init__.py", line 87, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/admin/Projects/cpython/Lib/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/admin/Projects/cpython/python.exe', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/var/folders/02/ps_0fn7s0sx9kkngq6ykymdh0000gn/T/tmphjt45j3u/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/var/folders/02/ps_0fn7s0sx9kkngq6ykymdh0000gn/T/tmphjt45j3u\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' died with <Signals.SIGABRT: 6>.

I guess the main problem at the beginning of trace:

Assertion failed: (tstate->datastack_top < tstate->datastack_limit), function _PyFrame_PushUnchecked, file pycore_frame.h, line 281.

So it seems more related to JIT than to ensurepip.

[JeffersGlass]

Bisecting, this looks like it broke in 1ab6356.

[JeffersGlass]

This error does not appear to occur on Windows x64 builds on 3.13:

.\PCBuild\build.bat -e -d -p x64 --experimental-jit      
.\PCbuild\amd64\python_d.exe -m ensurepip      # this succeeds

.\PCbuild\amd64\python_d.exe
Python 3.13.0b2+ (heads/3.13:ffc8e21c80, Jun 13 2024, 09:01:50) [MSC v.1929 64 bit (AMD64)]   

So it seems more related to JIT than to ensurepip.

This makes sense - probably some JIT issue that ensurepip happens to be triggering.

[Zheaoli]

The bug confirm on 3.13 and has disappered in main branch

Let me find out it

[Zheaoli]

@Eclips4 would you mind assigning this to me? I may need a bit of time to find out the root cause

[Eclips4]

Sure!

[JeffersGlass]

Playing around with this and bisecting main, it looks like the commit where this was fixed is 9b8611e ... which doesn't make a lot of sense to me, that's part of a PR all about typing... Perhaps I messed up the bisect.

For what its worth, this is what I ran with git bisect run:

#!/bin/sh
make distclean -j8 || true
(./configure --enable-experimental-jit --with-pydebug && make -j8) || exit 125
! ./python -m venv venv # Negated, since the earlier commit is bad and the later one is good

[Zheaoli]

import runpy
import sys
sys.path = ['/tmp/tmp04bw2hi9/pip-23.3.2-py3-none-any.whl'] + sys.path
sys.argv[1:] = ['install', '--no-cache-dir', '--no-index', '--find-links', '/tmp/tmp04bw2hi9', 'pip']
runpy.run_module("pip", run_name="__main__", alter_sys=True)

This bug is confirmed that is introduced into the codebase in
1ab6356

I will dive more deeper, but I may need @markshannon help

[Zheaoli]

After more inspect, I think this is about Tier2, not JIT

./configure --with-pydebug --enable-pystats --enable-profiling --with-dtrace --enable-experimental-jit=interpreter

Continue to dive deeper, but I may need @Fidget-Spinner help...

BTW this issue should be labeled about Tier2 cc @Eclips4

[Eclips4]

BTW this issue should be labeled about Tier2 cc @Eclips4

Unfortunately, we don't have such label

[Zheaoli]

Unfortunately, we don't have such label

Got it

[Zheaoli]

Update, When I try to comment the following code, the issue fixed

cpython/Lib/mimetypes.py

Lines 31 to 39 in 1ab6356

	try:
	from _winapi import _mimetypes_read_windows_registry
	except ImportError:
	_mimetypes_read_windows_registry = None
	try:
	import winreg as _winreg
	except ImportError:
	_winreg = None

I have no idea about this issue now.. why it's happend and why it's fixed in 9b8611e

cc @markshannon I may need your help sir(

[Zheaoli]

#4  0x00007ffff7cce3df in __assert_fail_base (fmt=0x7ffff7e59b68 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x555555ad4848 "tstate->datastack_top < tstate->datastack_limit", 
    file=file@entry=0x555555ad4720 "./Include/internal/pycore_frame.h", line=line@entry=284, function=function@entry=0x555555ad86c0 <__PRETTY_FUNCTION__.35> "_PyFrame_PushUnchecked") at assert.c:94
#5  0x00007ffff7cdec67 in __assert_fail (assertion=0x555555ad4848 "tstate->datastack_top < tstate->datastack_limit", file=0x555555ad4720 "./Include/internal/pycore_frame.h", line=284, 
    function=0x555555ad86c0 <__PRETTY_FUNCTION__.35> "_PyFrame_PushUnchecked") at assert.c:103
#6  0x0000555555861966 in _PyFrame_PushUnchecked (tstate=0x555555f2e4a0 <_PyRuntime+293952>, func=0x7ffff6bf37d0, null_locals_from=3) at ./Include/internal/pycore_frame.h:284
#7  0x00005555558a1fac in _PyEval_EvalFrameDefault (tstate=0x555555f2e4a0 <_PyRuntime+293952>, frame=0x7ffff7f98e58, throwflag=0) at Python/executor_cases.c.h:3326
#8  0x00005555558620b9 in _PyEval_EvalFrame (tstate=0x555555f2e4a0 <_PyRuntime+293952>, frame=0x7ffff7f98b80, throwflag=0) at ./Include/internal/pycore_ceval.h:118
#9  0x00005555558aa929 in _PyEval_Vector (tstate=0x555555f2e4a0 <_PyRuntime+293952>, func=0x7ffff6de10d0, locals=0x0, args=0x7ffffff878c0, argcount=2, kwnames=0x0) at Python/ceval.c:1818
#10 0x00005555556b9ef8 in _PyFunction_Vectorcall (func=<function at remote 0x7ffff6de10d0>, stack=0x7ffffff878c0, nargsf=2, kwnames=0x0) at Objects/call.c:413
#11 0x00005555556b8b89 in _PyObject_VectorcallTstate (tstate=0x555555f2e4a0 <_PyRuntime+293952>, callable=<function at remote 0x7ffff6de10d0>, args=0x7ffffff878c0, nargsf=2, kwnames=0x0)
    at ./Include/internal/pycore_call.h:168
#12 0x00005555556bb2d2 in object_vacall (tstate=0x555555f2e4a0 <_PyRuntime+293952>, base=0x0, callable=<function at remote 0x7ffff6de10d0>, vargs=0x7ffffff87930) at Objects/call.c:819
#13 0x00005555556bb61b in PyObject_CallMethodObjArgs (obj=0x0, name='_find_and_load') at Objects/call.c:880
#14 0x000055555590eed7 in import_find_and_load (tstate=0x555555f2e4a0 <_PyRuntime+293952>, abs_name='_winapi') at Python/import.c:3080
#15 0x000055555590f1ce in PyImport_ImportModuleLevelObject (name='_winapi', 
    globals={'__name__': 'mimetypes', '__doc__': 'Guess the MIME type of a file.\n\nThis module defines two useful functions:\n\nguess_type(url, strict=True) -- guess the MIME type and encoding of a URL.\n\nguess_extension(type, strict=True) -- guess the extension for a given MIME type.\n\nIt also contains the following, for tuning the behavior:\n\nData:\n\nknownfiles -- list of files to parse\ninited -- flag set when init() has been called\nsuffix_map -- dictionary mapping suffixes to suffixes\nencodings_map -- dictionary mapping suffixes to encodings\ntypes_map -- dictionary mapping suffixes to types\n\nFunctions:\n\ninit([files]) -- parse a list of files, default knownfiles (on Windows, the\n  default values are taken from the registry)\nread_mime_types(file) -- parse one file, return a dictionary or None\n', '__package__': '', '__loader__': <SourceFileLoader(name='mimetypes', path='/home/manjusaka/Documents/projects/cpython/Lib/mimetypes.py') at remote 0x7ffff50d5100>, '__spec__': <ModuleSpec(name='mimetypes', loader...(truncated), 
    locals={'__name__': 'mimetypes', '__doc__': 'Guess the MIME type of a file.\n\nThis module defines two useful functions:\n\nguess_type(url, strict=True) -- guess the MIME type and encoding of a URL.\n\nguess_extension(type, strict=True) -- guess the extension for a given MIME type.\n\nIt also contains the following, for tuning the behavior:\n\nData:\n\nknownfiles -- list of files to parse\ninited -- flag set when init() has been called\nsuffix_map -- dictionary mapping suffixes to suffixes\nencodings_map -- dictionary mapping suffixes to encodings\ntypes_map -- dictionary mapping suffixes to types\n\nFunctions:\n\ninit([files]) -- parse a list of files, default knownfiles (on Windows, the\n  default values are taken from the registry)\nread_mime_types(file) -- parse one file, return a dictionary or None\n', '__package__': '', '__loader__': <SourceFileLoader(name='mimetypes', path='/home/manjusaka/Documents/projects/cpython/Lib/mimetypes.py') at remote 0x7ffff50d5100>, '__spec__': <ModuleSpec(name='mimetypes', loader...(truncated), fromlist=('_mimetypes_read_windows_registry',), level=0) at Python/import.c:3160
#16 0x00005555558ac882 in import_name (tstate=0x555555f2e4a0 <_PyRuntime+293952>, frame=0x7ffff7f98b18, name='_winapi', fromlist=('_mimetypes_read_windows_registry',), level=0) at Python/ceval.c:2629
#17 0x000055555587c9a2 in _PyEval_EvalFrameDefault (tstate=0x555555f2e4a0 <_PyRuntime+293952>, frame=0x7ffff7f98b18, throwflag=0) at Python/generated_cases.c.h:3196
#18 0x00005555558620b9 in _PyEval_EvalFrame (tstate=0x555555f2e4a0 <_PyRuntime+293952>, frame=0x7ffff7f98b18, throwflag=0) at ./Include/internal/pycore_ceval.h:118
#19 0x00005555558aa929 in _PyEval_Vector (tstate=0x555555f2e4a0 <_PyRuntime+293952>, func=0x7ffff5131190, 
    locals={'__name__': 'mimetypes', '__doc__': 'Guess the MIME type of a file.\n\nThis module defines two useful functions:\n\nguess_type(url, strict=True) -- guess the MIME type and encoding of a URL.\n\nguess_extension(type, strict=True) -- guess the extension for a given MIME type.\n\nIt also contains the following, for tuning the behavior:\n\nData:\n\nknownfiles -- list of files to parse\ninited -- flag set when init() has been called\nsuffix_map -- dictionary mapping suffixes to suffixes\nencodings_map -- dictionary mapping suffixes to encodings\ntypes_map -- dictionary mapping suffixes to types\n\nFunctions:\n\ninit([files]) -- parse a list of files, default knownfiles (on Windows, the\n  default values are taken from the registry)\nread_mime_types(file) -- parse one file, return a dictionary or None\n', '__package__': '', '__loader__': <SourceFileLoader(name='mimetypes', path='/home/manjusaka/Documents/projects/cpython/Lib/mimetypes.py') at remote 0x7ffff50d5100>, '__spec__': <ModuleSpec(name='mimetypes', loader...(truncated), args=0x0, argcount=0, kwnames=0x0) at Python/ceval.c:1818
#20 0x0000555555863db6 in PyEval_EvalCode (co=<code at remote 0x7ffff50ccc40>, 
    globals={'__name__': 'mimetypes', '__doc__': 'Guess the MIME type of a file.\n\nThis module defines two useful functions:\n\nguess_type(url, strict=True) -- guess the MIME type and encoding of a URL.\n\nguess_extension(type, strict=True) -- guess the extension for a given MIME type.\n\nIt also contains the following, for tuning the behavior:\n\nData:\n\nknownfiles -- list of files to parse\ninited -- flag set when init() has been called\nsuffix_map -- dictionary mapping suffixes to suffixes\nencodings_map -- dictionary mapping suffixes to encodings\ntypes_map -- dictionary mapping suffixes to types\n\nFunctions:\n\ninit([files]) -- parse a list of files, default knownfiles (on Windows, the\n  default values are taken from the registry)\nread_mime_types(file) -- parse one file, return a dictionary or None\n', '__package__': '', '__loader__': <SourceFileLoader(name='mimetypes', path='/home/manjusaka/Documents/projects/cpython/Lib/mimetypes.py') at remote 0x7ffff50d5100>, '__spec__': <ModuleSpec(name='mimetypes', loader...(truncated), 

[Fidget-Spinner]

@Zheaoli we have a pass that eliminates redundant _CHECK_STACK_SPACE. You might want to try disabling that pass (commenting it out) and seeing if it still fails. If it passes, that means that bug is in the pass.

[Zheaoli]

@Zheaoli we have a pass that eliminates redundant _CHECK_STACK_SPACE. You might want to try disabling that pass (commenting it out) and seeing if it still fails. If it passes, that means that bug is in the pass.

Thanks sir. The bug has been confirmed. It's not related with _CHECK_STACK_SPACE. We have extra stack size check in CALL specialization and it has been removed in 1ab6356#diff-4ef46fa654f95502e49a24f7dc8ee31a4cac9b3433fe9cd2b2d4dd78cfbad448L1819

I made a PR to fix it. But I'm not sure we need extra test for this case. I have we don't have enough test in 1ab6356

[Fidget-Spinner]

The bug is indeed in the pass that combines _CHECK_STACK_SPACE. If you comment out https://github.com/python/cpython/blob/main/Python/optimizer_bytecodes.c#L702 this works.

[Fidget-Spinner]

On second thought, for some reason it works for me even if I don't comment that out??