Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-98354: Add unicode check for 'name' attribute in _imp_create_builtin #98412

Merged

Conversation

chgnrdv
Copy link
Contributor

@chgnrdv chgnrdv commented Oct 18, 2022

_imp_create_builtin crashes if 'name' attribute of 'spec' argument is not a 'str' instance. This commit adds appropriate check.
Copy link
Member

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing a fix!

Agree that this needs a new test. Also, there should be a NEWS entry.

Your check currently still allows subclasses of str. That's probably fine, but worth an additional test.

@ericsnowcurrently
Copy link
Member

The fix looks good but I agree a test and NEWS entry are warranted.

We'll also want to backport the fix to 3.10 and 3.11.

Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Lib/test/test_imp.py Show resolved Hide resolved
@miss-islington miss-islington merged commit 1f369ad into python:main Oct 20, 2022
@chgnrdv chgnrdv deleted the _imp_create_builtin-check-name-for-unicode branch October 20, 2022 13:42
carljm added a commit to carljm/cpython that referenced this pull request Oct 20, 2022
* main: (40 commits)
  pythongh-98461: Fix source location in comprehensions bytecode (pythonGH-98464)
  pythongh-98421: Clean Up PyObject_Print (pythonGH-98422)
  pythongh-98360: multiprocessing now spawns children on Windows with correct argv[0] in virtual environments (pythonGH-98462)
  CODEOWNERS: Become a typing code owner (python#98480)
  [doc] Improve logging cookbook example. (pythonGH-98481)
  Add more tkinter.Canvas tests (pythonGH-98475)
  pythongh-95023: Added os.setns and os.unshare functions (python#95046)
  pythonGH-98363: Presize the list for batched() (pythonGH-98419)
  pythongh-98374: Suppress ImportError for invalid query for help() command. (pythongh-98450)
  typing tests: `_overload_dummy` raises `NotImplementedError`, not `RuntimeError` (python#98351)
  pythongh-98354: Add unicode check for 'name' attribute in _imp_create_builtin (pythonGH-98412)
  pythongh-98257: Make _PyEval_SetTrace() reentrant (python#98258)
  pythongh-98414: py.exe launcher does not use defaults for -V:company/ option (pythonGH-98460)
  pythongh-98417: Store int_max_str_digits on the Interpreter State (pythonGH-98418)
  Doc: Remove title text from internal links (python#98409)
  [doc] Refresh the venv introduction documentation, and correct the statement about VIRTUAL_ENV (pythonGH-98350)
  Docs: Bump sphinx-lint and fix unbalanced inline literal markup (python#98441)
  pythongh-92886: Replace assertion statements in `handlers.BaseHandler` to support running with optimizations (`-O`) (pythonGH-93231)
  pythongh-92886: Fix tests that fail when running with optimizations (`-O`) in `_test_multiprocessing.py` (pythonGH-93233)
  pythongh-92886: Fix tests that fail when running with optimizations (`-O`) in `test_py_compile.py` (pythonGH-93235)
  ...
@chgnrdv
Copy link
Contributor Author

chgnrdv commented Oct 21, 2022

@ericsnowcurrently @sweeneyde
Looks like import of builtins module in my test (both through import statement and import_helper.import_module function) leads interpreter to crash if test_imp is running as a part of full test package (i. e. launched through python -m test), at least on my machine. It can be prevented by not importing any module in my test at all and just checking bltin value's __name__ attribute, but does it deserve separate issue/PR? Documentation of test package says that excessive imports can lead to "anomalous behaviour", is this the case?
test_imp is running OK when launched separately.

@ericsnowcurrently
Copy link
Member

Thanks for bringing this up, @chgnrdv!

Looks like import of builtins module in my test (both through import statement and import_helper.import_module function) leads interpreter to crash if test_imp is running as a part of full test package (i. e. launched through python -m test), at least on my machine. [snip] test_imp is running OK when launched separately.

CI would have caught this if it's a consistent problem. Is this intermittent (depending on the order of tests?

It can be prevented by not importing any module in my test at all and just checking bltin value's __name__ attribute, but does it deserve separate issue/PR? Documentation of test package says that excessive imports can lead to "anomalous behaviour", is this the case?

It shouldn't be crashing though. Furthermore, if you are still seeing the behavior from gh-98354 in some cases then we should fix it. I've re-opened the issue.

@chgnrdv
Copy link
Contributor Author

chgnrdv commented Nov 2, 2022

Is this intermittent (depending on the order of tests?

It can depend on the order of tests, since it crashes when I run the test package in default order, but passes the CI (which, AFAIK, runs tests in random order).
If I run test_imp separately, it doesn't crash (and passes, as it should).

Furthermore, if you are still seeing the behavior from #98354 in some cases

I don't see the behaviour which I had to fix by this PR anymore and I'm sure that this issue or my fix itself can't cause any problem like this. I think that it is somehow related to how test package handles the import of builtins module in my test case.

Test package output with traceback and abort message:

0:33:04 load avg: 0.83 [190/437] test_imp
Modules/gcmodule.c:113: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f83fbfcb8f0
object refcount : 99
object type     : 0x559bf783d860
object type name: module
object repr     : <module 'builtins' (built-in)>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007f83fc323740 (most recent call first):
  Garbage-collecting
  File "/home/.../cpython/Lib/test/support/__init__.py", line 738 in gc_collect
  File "/home/.../cpython/Lib/test/libregrtest/save_env.py", line 314 in __exit__
  File "/home/.../cpython/Lib/test/libregrtest/runtest.py", line 312 in _runtest_inner2
  File "/home/.../cpython/Lib/test/libregrtest/runtest.py", line 360 in _runtest_inner
  File "/home/.../cpython/Lib/test/libregrtest/runtest.py", line 235 in _runtest
  File "/home/.../cpython/Lib/test/libregrtest/runtest.py", line 265 in runtest
  File "/home/.../cpython/Lib/test/libregrtest/main.py", line 450 in run_tests_sequential
  File "/home/.../cpython/Lib/test/libregrtest/main.py", line 564 in run_tests
  File "/home/.../cpython/Lib/test/libregrtest/main.py", line 742 in _main
  File "/home/.../cpython/Lib/test/libregrtest/main.py", line 701 in main
  File "/home/.../cpython/Lib/test/libregrtest/main.py", line 763 in main
  File "/home/.../cpython/Lib/test/__main__.py", line 2 in <module>
  File "/home/.../cpython/Lib/runpy.py", line 88 in _run_code
  File "/home/.../cpython/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi, _xxsubinterpreters, _testinternalcapi, _testbuffer, _testmultiphase, _ctypes_test, xxsubtype (total: 7)
Aborted (core dumped)

Stack trace:

#0  0x00007f5d903b134c in __pthread_kill_implementation () from /usr/lib/libc.so.6
#1  0x00007f5d903644b8 in raise () from /usr/lib/libc.so.6
#2  0x00007f5d9034e534 in abort () from /usr/lib/libc.so.6
#3  0x000055faf5565bf6 in fatal_error_exit (status=<optimized out>) at Python/pylifecycle.c:2629
#4  0x000055faf5567236 in fatal_error (fd=2, header=header@entry=1, prefix=prefix@entry=0x55faf56427e0 <__func__.2> "_PyObject_AssertFailed", 
    msg=msg@entry=0x55faf5642646 "_PyObject_AssertFailed", status=status@entry=-1) at Python/pylifecycle.c:2810
#5  0x000055faf55672a0 in _Py_FatalErrorFunc (func=func@entry=0x55faf56427e0 <__func__.2> "_PyObject_AssertFailed", 
    msg=msg@entry=0x55faf5642646 "_PyObject_AssertFailed") at Python/pylifecycle.c:2826
#6  0x000055faf547f0b7 in _PyObject_AssertFailed (obj=0x7f5d8ffc7890, expr=expr@entry=0x55faf56a3268 "gc_get_refs(g) > 0", 
    msg=msg@entry=0x55faf5642692 "refcount is too small", file=file@entry=0x55faf56a30d6 "Modules/gcmodule.c", line=line@entry=113, 
    function=function@entry=0x55faf56a36e8 <__func__.27> "gc_decref") at Objects/object.c:2361
#7  0x000055faf559f22d in gc_decref (g=<optimized out>) at Modules/gcmodule.c:113
#8  0x000055faf559f945 in visit_decref (op=0x7f5d8ffc7890, parent=0x7f5d890a3950) at Modules/gcmodule.c:459
#9  0x000055faf5467b25 in dict_traverse (op=<optimized out>, visit=0x55faf559f8e2 <visit_decref>, arg=0x7f5d890a3950) at Objects/dictobject.c:3525
#10 0x000055faf559df22 in subtract_refs (containers=containers@entry=0x55faf5883018 <_PyRuntime+69432>) at Modules/gcmodule.c:478
#11 0x000055faf559f329 in deduce_unreachable (base=base@entry=0x55faf5883018 <_PyRuntime+69432>, unreachable=unreachable@entry=0x7ffe797ee710)
    at Modules/gcmodule.c:1103
#12 0x000055faf559feb5 in gc_collect_main (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>, generation=generation@entry=2, 
    n_collected=n_collected@entry=0x7ffe797ee798, n_uncollectable=n_uncollectable@entry=0x7ffe797ee7a0, nofail=nofail@entry=0)
    at Modules/gcmodule.c:1229
#13 0x000055faf55a0431 in gc_collect_with_callback (tstate=0x55faf589e3e0 <_PyRuntime+180992>, generation=2) at Modules/gcmodule.c:1403
#14 0x000055faf55a0513 in gc_collect_impl (module=module@entry=0x7f5d8f7707d0, generation=generation@entry=2) at Modules/gcmodule.c:1541
#15 0x000055faf55a05d3 in gc_collect (module=0x7f5d8f7707d0, args=<optimized out>, args@entry=0x7f5d9062e980, nargs=nargs@entry=0, 
    kwnames=kwnames@entry=0x0) at Modules/clinic/gcmodule.c.h:139
#16 0x000055faf551d845 in _PyEval_EvalFrameDefault (tstate=0x55faf589e3e0 <_PyRuntime+180992>, frame=0x7f5d9062e920, throwflag=0)
    at Python/ceval.c:4502
#17 0x000055faf5520c0e in _PyEval_EvalFrame (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>, frame=frame@entry=0x7f5d9062e830, 
    throwflag=throwflag@entry=0) at ./Include/internal/pycore_ceval.h:88
#18 0x000055faf5520d26 in _PyEval_Vector (tstate=0x55faf589e3e0 <_PyRuntime+180992>, func=0x7f5d8f7b9ff0, locals=locals@entry=0x0, 
    args=0x7f5d9062e7f8, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:5826
#19 0x000055faf5431288 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at Objects/call.c:396
#20 0x000055faf5433c86 in _PyObject_VectorcallTstate (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>, 
    callable=callable@entry=0x7f5d8f7b9ff0, args=args@entry=0x7f5d9062e7f8, nargsf=nargsf@entry=4, kwnames=kwnames@entry=0x0)
    at ./Include/internal/pycore_call.h:92
#21 0x000055faf5433e4f in method_vectorcall (method=<optimized out>, args=0x7f5d9062e800, nargsf=<optimized out>, kwnames=0x0)
    at Objects/classobject.c:59
#22 0x000055faf54315f9 in _PyObject_VectorcallTstate (tstate=0x55faf589e3e0 <_PyRuntime+180992>, callable=callable@entry=0x7f5d88cbf770, 
    args=args@entry=0x7f5d9062e800, nargsf=9223372036854775811, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:92
#23 0x000055faf54316c5 in PyObject_Vectorcall (callable=callable@entry=0x7f5d88cbf770, args=args@entry=0x7f5d9062e800, nargsf=<optimized out>, 
    kwnames=kwnames@entry=0x0) at Objects/call.c:300
#24 0x000055faf551c178 in _PyEval_EvalFrameDefault (tstate=0x55faf589e3e0 <_PyRuntime+180992>, frame=0x7f5d9062e778, throwflag=0)
    at Python/ceval.c:4206
#25 0x000055faf5520c0e in _PyEval_EvalFrame (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>, frame=frame@entry=0x7f5d9062e290, 
    throwflag=throwflag@entry=0) at ./Include/internal/pycore_ceval.h:88
#26 0x000055faf5520d26 in _PyEval_Vector (tstate=0x55faf589e3e0 <_PyRuntime+180992>, func=0x7f5d8f7c8310, locals=locals@entry=0x0, 
    args=0x7f5d8ffbe200, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:5826
#27 0x000055faf5431288 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at Objects/call.c:396
#28 0x000055faf5433c86 in _PyObject_VectorcallTstate (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>, 
    callable=callable@entry=0x7f5d8f7c8310, args=args@entry=0x7f5d8ffbe200, nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x7f5d8fe68550)
    at ./Include/internal/pycore_call.h:92
#29 0x000055faf5433e4f in method_vectorcall (method=<optimized out>, args=0x7f5d8ffbe208, nargsf=<optimized out>, kwnames=0x7f5d8fe68550)
    at Objects/classobject.c:59
#30 0x000055faf5430ee1 in _PyVectorcall_Call (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>, func=0x55faf5433d0e <method_vectorcall>, 
    callable=callable@entry=0x7f5d8fe8d1f0, tuple=tuple@entry=0x55faf5882d20 <_PyRuntime+68672>, kwargs=kwargs@entry=0x7f5d8fdf9d30)
    at Objects/call.c:258
#31 0x000055faf54311d0 in _PyObject_Call (tstate=0x55faf589e3e0 <_PyRuntime+180992>, callable=callable@entry=0x7f5d8fe8d1f0, 
    args=args@entry=0x55faf5882d20 <_PyRuntime+68672>, kwargs=kwargs@entry=0x7f5d8fdf9d30) at Objects/call.c:329
#32 0x000055faf5431218 in PyObject_Call (callable=callable@entry=0x7f5d8fe8d1f0, args=args@entry=0x55faf5882d20 <_PyRuntime+68672>, 
    kwargs=kwargs@entry=0x7f5d8fdf9d30) at Objects/call.c:356
#33 0x000055faf550d9a3 in do_call_core (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>, func=func@entry=0x7f5d8fe8d1f0, 
    callargs=callargs@entry=0x55faf5882d20 <_PyRuntime+68672>, kwdict=kwdict@entry=0x7f5d8fdf9d30, use_tracing=0) at Python/ceval.c:6759
#34 0x000055faf551efa4 in _PyEval_EvalFrameDefault (tstate=0x55faf589e3e0 <_PyRuntime+180992>, frame=0x7f5d9062e210, throwflag=0)
    at Python/ceval.c:4776
#35 0x000055faf5520c0e in _PyEval_EvalFrame (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>,frame=frame@entry=0x7f5d9062e1b8,
    throwflag=throwflag@entry=0) at ./Include/internal/pycore_ceval.h:88
#36 0x000055faf5520d26 in _PyEval_Vector (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>,func=func@entry=0x7f5d8ff83c20,
    locals=locals@entry=0x7f5d8fdf94f0, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:5826
#37 0x000055faf5520e3c in PyEval_EvalCode (co=co@entry=0x7f5d8fe34640, globals=globals@entry=0x7f5d8fdf94f0, locals=locals@entry=0x7f5d8fdf94f0)
    at Python/ceval.c:583
#38 0x000055faf550721f in builtin_exec_impl (module=module@entry=0x7f5d8ffc7890, source=0x7f5d8fe34640, globals=0x7f5d8fdf94f0, 
    locals=0x7f5d8fdf94f0, closure=0x0) at Python/bltinmodule.c:1079
#39 0x000055faf550733a in builtin_exec (module=0x7f5d8ffc7890, args=<optimized out>, args@entry=0x7f5d9062e180, nargs=nargs@entry=2, 
    kwnames=kwnames@entry=0x0) at Python/clinic/bltinmodule.c.h:543
#40 0x000055faf547ab58 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7f5d8ffc7ef0, args=0x7f5d9062e180, nargsf=<optimized out>, kwnames=0x0)
    at Objects/methodobject.c:443
#41 0x000055faf54315f9 in _PyObject_VectorcallTstate (tstate=0x55faf589e3e0 <_PyRuntime+180992>, callable=callable@entry=0x7f5d8ffc7ef0, 
    args=args@entry=0x7f5d9062e180, nargsf=9223372036854775810, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:92
#42 0x000055faf54316c5 in PyObject_Vectorcall (callable=callable@entry=0x7f5d8ffc7ef0, args=args@entry=0x7f5d9062e180, nargsf=<optimized out>, 
    kwnames=kwnames@entry=0x0) at Objects/call.c:300
#43 0x000055faf551c178 in _PyEval_EvalFrameDefault (tstate=0x55faf589e3e0 <_PyRuntime+180992>, frame=0x7f5d9062e0d8, throwflag=0)
    at Python/ceval.c:4206
#44 0x000055faf5520c0e in _PyEval_EvalFrame (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>, frame=frame@entry=0x7f5d9062e020, 
    throwflag=throwflag@entry=0) at ./Include/internal/pycore_ceval.h:88
#45 0x000055faf5520d26 in _PyEval_Vector (tstate=0x55faf589e3e0 <_PyRuntime+180992>, func=0x7f5d8fe726d0, locals=locals@entry=0x0, 
    args=0x7f5d8fe686a8, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:5826
#46 0x000055faf5431288 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at Objects/call.c:396
#47 0x000055faf5430e33 in _PyVectorcall_Call (tstate=tstate@entry=0x55faf589e3e0 <_PyRuntime+180992>, func=0x55faf5431238 <_PyFunction_Vectorcall>, 
    callable=callable@entry=0x7f5d8fe726d0, tuple=tuple@entry=0x7f5d8fe68690, kwargs=kwargs@entry=0x0) at Objects/call.c:246
#48 0x000055faf54311d0 in _PyObject_Call (tstate=0x55faf589e3e0 <_PyRuntime+180992>, callable=callable@entry=0x7f5d8fe726d0, 
    args=args@entry=0x7f5d8fe68690, kwargs=kwargs@entry=0x0) at Objects/call.c:329
#49 0x000055faf5431218 in PyObject_Call (callable=callable@entry=0x7f5d8fe726d0, args=args@entry=0x7f5d8fe68690, kwargs=kwargs@entry=0x0)
    at Objects/call.c:356
#50 0x000055faf559cf4d in pymain_run_module (modname=<optimized out>, set_argv0=set_argv0@entry=1) at Modules/main.c:300
#51 0x000055faf559da6e in pymain_run_python (exitcode=exitcode@entry=0x7ffe797ef5d4) at Modules/main.c:604
#52 0x000055faf559dd56 in Py_RunMain () at Modules/main.c:689
#53 0x000055faf559ddcd in pymain_main (args=args@entry=0x7ffe797ef630) at Modules/main.c:719
#54 0x000055faf559de93 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#55 0x000055faf53a2762 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15

@chgnrdv
Copy link
Contributor Author

chgnrdv commented Nov 3, 2022

@ericsnowcurrently
I ran test package in random order on current main multiple times. I also tried to put test_imp on different positions in list of default and randomly ordered tests. Crash occurs between test_imp and the next one if test_imp is placed on position in test list that is large enough, with the lower bound somewhere between 75-100. Running with -v flag shows (along with python traceback) that the actual test passes, but crash happens during the process of saving some environment shared between tests.
When I run tests with command line arguments similar to which used in CI, i. e.
./python -u -W default -bb -E -E -m test -w -j 1 -u all -W --slowest --fail-env-changed --timeout=1200 -j4 -uall,-cpu
crash doesn't occur.

@ericsnowcurrently
Copy link
Member

@chgnrdv, sorry for not getting back to you. Is the problem solved by the fix for gh-99578?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

create_builtin() in _imp module trigger segfault if taking a builtin object as input
6 participants