diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 7f9d0f4da09be7..28ebc1643bd694 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -182,7 +182,7 @@ jobs: - name: Display build info run: .\python.bat -m test.pythoninfo - name: Tests - run: .\PCbuild\rt.bat -p Win32 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 + run: .\PCbuild\rt.bat -p Win32 -d -q --fast-ci build_win_amd64: name: 'Windows (x64)' @@ -201,7 +201,7 @@ jobs: - name: Display build info run: .\python.bat -m test.pythoninfo - name: Tests - run: .\PCbuild\rt.bat -p x64 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 + run: .\PCbuild\rt.bat -p x64 -d -q --fast-ci build_win_arm64: name: 'Windows (arm64)' @@ -252,7 +252,7 @@ jobs: - name: Display build info run: make pythoninfo - name: Tests - run: make buildbottest TESTOPTS="-j4 -uall,-cpu" + run: make test build_ubuntu: name: 'Ubuntu' @@ -319,7 +319,7 @@ jobs: run: sudo mount $CPYTHON_RO_SRCDIR -oremount,rw - name: Tests working-directory: ${{ env.CPYTHON_BUILDDIR }} - run: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu" + run: xvfb-run make test build_ubuntu_ssltests: name: 'Ubuntu SSL tests with OpenSSL' @@ -535,7 +535,7 @@ jobs: - name: Display build info run: make pythoninfo - name: Tests - run: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu" + run: xvfb-run make test all-required-green: # This job does nothing and is only used for the branch protection name: All required checks pass diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst index 8bd23daee73977..d261e4a4f338a5 100644 --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -22,42 +22,48 @@ can be inspected programmatically via importing :mod:`__future__` and examining its contents. -Each statement in :file:`__future__.py` is of the form:: +.. _future-classes: - FeatureName = _Feature(OptionalRelease, MandatoryRelease, - CompilerFlag) +.. class:: _Feature + Each statement in :file:`__future__.py` is of the form:: -where, normally, *OptionalRelease* is less than *MandatoryRelease*, and both are -5-tuples of the same form as :data:`sys.version_info`:: + FeatureName = _Feature(OptionalRelease, MandatoryRelease, + CompilerFlag) - (PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int - PY_MINOR_VERSION, # the 1; an int - PY_MICRO_VERSION, # the 0; an int - PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string - PY_RELEASE_SERIAL # the 3; an int - ) + where, normally, *OptionalRelease* is less than *MandatoryRelease*, and both are + 5-tuples of the same form as :data:`sys.version_info`:: -*OptionalRelease* records the first release in which the feature was accepted. + (PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int + PY_MINOR_VERSION, # the 1; an int + PY_MICRO_VERSION, # the 0; an int + PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string + PY_RELEASE_SERIAL # the 3; an int + ) -In the case of a *MandatoryRelease* that has not yet occurred, -*MandatoryRelease* predicts the release in which the feature will become part of -the language. +.. method:: _Feature.getOptionalRelease() -Else *MandatoryRelease* records when the feature became part of the language; in -releases at or after that, modules no longer need a future statement to use the -feature in question, but may continue to use such imports. + *OptionalRelease* records the first release in which the feature was accepted. -*MandatoryRelease* may also be ``None``, meaning that a planned feature got -dropped. +.. method:: _Feature.getMandatoryRelease() -Instances of class :class:`_Feature` have two corresponding methods, -:meth:`getOptionalRelease` and :meth:`getMandatoryRelease`. + In the case of a *MandatoryRelease* that has not yet occurred, + *MandatoryRelease* predicts the release in which the feature will become part of + the language. -*CompilerFlag* is the (bitfield) flag that should be passed in the fourth -argument to the built-in function :func:`compile` to enable the feature in -dynamically compiled code. This flag is stored in the :attr:`compiler_flag` -attribute on :class:`_Feature` instances. + Else *MandatoryRelease* records when the feature became part of the language; in + releases at or after that, modules no longer need a future statement to use the + feature in question, but may continue to use such imports. + + *MandatoryRelease* may also be ``None``, meaning that a planned feature got + dropped or that it is not yet decided. + +.. attribute:: _Feature.compiler_flag + + *CompilerFlag* is the (bitfield) flag that should be passed in the fourth + argument to the built-in function :func:`compile` to enable the feature in + dynamically compiled code. This flag is stored in the :attr:`_Feature.compiler_flag` + attribute on :class:`_Feature` instances. No feature description will ever be deleted from :mod:`__future__`. Since its introduction in Python 2.1 the following features have found their way into the diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 22360b22fd924b..48d6176d26bb8f 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -1381,15 +1381,19 @@ call fails (for example because the path doesn't exist). >>> p.resolve() PosixPath('/home/antoine/pathlib/setup.py') - If the path doesn't exist and *strict* is ``True``, :exc:`FileNotFoundError` - is raised. If *strict* is ``False``, the path is resolved as far as possible - and any remainder is appended without checking whether it exists. If an - infinite loop is encountered along the resolution path, :exc:`RuntimeError` - is raised. + If a path doesn't exist or a symlink loop is encountered, and *strict* is + ``True``, :exc:`OSError` is raised. If *strict* is ``False``, the path is + resolved as far as possible and any remainder is appended without checking + whether it exists. .. versionchanged:: 3.6 The *strict* parameter was added (pre-3.6 behavior is strict). + .. versionchanged:: 3.13 + Symlink loops are treated like other errors: :exc:`OSError` is raised in + strict mode, and no exception is raised in non-strict mode. In previous + versions, :exc:`RuntimeError` is raised no matter the value of *strict*. + .. method:: Path.rglob(pattern, *, case_sensitive=None, follow_symlinks=None) Glob the given relative *pattern* recursively. This is like calling diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 875916be1049a3..54c3907dec98cc 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -472,6 +472,12 @@ Standard names are defined for the following types: .. versionadded:: 3.12 +.. class:: CapsuleType + + The type of :ref:`capsule objects `. + + .. versionadded:: 3.13 + Additional Utility Classes and Functions ---------------------------------------- diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index 1406b663c6a8e2..d6e062df945c64 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -111,7 +111,7 @@ See :ref:`__slots__ documentation ` for details. Exceptions raised by the callback will be noted on the standard error output, but cannot be propagated; they are handled in exactly the same way as exceptions - raised from an object's :meth:`__del__` method. + raised from an object's :meth:`~object.__del__` method. Weak references are :term:`hashable` if the *object* is hashable. They will maintain their hash value even after the *object* was deleted. If @@ -221,8 +221,7 @@ than needed. Added support for ``|`` and ``|=`` operators, as specified in :pep:`584`. :class:`WeakValueDictionary` objects have an additional method that has the -same issues as the :meth:`keyrefs` method of :class:`WeakKeyDictionary` -objects. +same issues as the :meth:`WeakKeyDictionary.keyrefs` method. .. method:: WeakValueDictionary.valuerefs() @@ -281,7 +280,7 @@ objects. Exceptions raised by finalizer callbacks during garbage collection will be shown on the standard error output, but cannot be propagated. They are handled in the same way as exceptions raised - from an object's :meth:`__del__` method or a weak reference's + from an object's :meth:`~object.__del__` method or a weak reference's callback. When the program exits, each remaining live finalizer is called @@ -523,18 +522,18 @@ is still alive. For instance obj dead or exiting -Comparing finalizers with :meth:`__del__` methods -------------------------------------------------- +Comparing finalizers with :meth:`~object.__del__` methods +--------------------------------------------------------- Suppose we want to create a class whose instances represent temporary directories. The directories should be deleted with their contents when the first of the following events occurs: * the object is garbage collected, -* the object's :meth:`remove` method is called, or +* the object's :meth:`!remove` method is called, or * the program exits. -We might try to implement the class using a :meth:`__del__` method as +We might try to implement the class using a :meth:`~object.__del__` method as follows:: class TempDir: @@ -553,12 +552,12 @@ follows:: def __del__(self): self.remove() -Starting with Python 3.4, :meth:`__del__` methods no longer prevent +Starting with Python 3.4, :meth:`~object.__del__` methods no longer prevent reference cycles from being garbage collected, and module globals are no longer forced to :const:`None` during :term:`interpreter shutdown`. So this code should work without any issues on CPython. -However, handling of :meth:`__del__` methods is notoriously implementation +However, handling of :meth:`~object.__del__` methods is notoriously implementation specific, since it depends on internal details of the interpreter's garbage collector implementation. diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index f9290f528e5555..54c93008503c02 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -17,7 +17,7 @@ for parsing and creating XML data. This module will use a fast implementation whenever available. .. deprecated:: 3.3 - The :mod:`xml.etree.cElementTree` module is deprecated. + The :mod:`!xml.etree.cElementTree` module is deprecated. .. warning:: @@ -825,6 +825,8 @@ Reference Functions ^^^^^^^^^ +.. module:: xml.etree.ElementInclude + .. function:: xml.etree.ElementInclude.default_loader( href, parse, encoding=None) :module: @@ -862,6 +864,9 @@ Functions Element Objects ^^^^^^^^^^^^^^^ +.. module:: xml.etree.ElementTree + :noindex: + .. class:: Element(tag, attrib={}, **extra) Element class. This class defines the Element interface, and provides a diff --git a/Doc/tools/.nitignore b/Doc/tools/.nitignore index 39a10992193ffc..f260a571661b76 100644 --- a/Doc/tools/.nitignore +++ b/Doc/tools/.nitignore @@ -28,7 +28,6 @@ Doc/howto/enum.rst Doc/howto/isolating-extensions.rst Doc/howto/logging.rst Doc/howto/urllib2.rst -Doc/library/__future__.rst Doc/library/abc.rst Doc/library/ast.rst Doc/library/asyncio-dev.rst @@ -134,12 +133,10 @@ Doc/library/unittest.mock.rst Doc/library/unittest.rst Doc/library/urllib.parse.rst Doc/library/urllib.request.rst -Doc/library/weakref.rst Doc/library/wsgiref.rst Doc/library/xml.dom.minidom.rst Doc/library/xml.dom.pulldom.rst Doc/library/xml.dom.rst -Doc/library/xml.etree.elementtree.rst Doc/library/xml.rst Doc/library/xml.sax.handler.rst Doc/library/xml.sax.reader.rst diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index 763f9778776990..a9555199a2ac24 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -964,9 +964,18 @@ Main Makefile targets You can use the configure :option:`--enable-optimizations` option to make this the default target of the ``make`` command (``make all`` or just ``make``). -* ``make buildbottest``: Build Python and run the Python test suite, the same - way than buildbots test Python. Set ``TESTTIMEOUT`` variable (in seconds) - to change the test timeout (1200 by default: 20 minutes). + +* ``make test``: Build Python and run the Python test suite with ``--fast-ci`` + option. Variables: + + * ``TESTOPTS``: additional regrtest command line options. + * ``TESTPYTHONOPTS``: additional Python command line options. + * ``TESTTIMEOUT``: timeout in seconds (default: 20 minutes). + +* ``make buildbottest``: Similar to ``make test``, but use ``--slow-ci`` + option and default timeout of 20 minutes, instead of ``--fast-ci`` option + and a default timeout of 10 minutes. + * ``make install``: Build and install Python. * ``make regen-all``: Regenerate (almost) all generated files; ``make regen-stdlib-module-names`` and ``autoconf`` must be run separately diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst index 22538a476d31b4..dfe2ea886a9209 100644 --- a/Doc/whatsnew/3.12.rst +++ b/Doc/whatsnew/3.12.rst @@ -3,7 +3,7 @@ What's New In Python 3.12 **************************** -:Editor: TBD +:Editor: Adam Turner .. Rules for maintenance: @@ -46,15 +46,12 @@ researching a change. This article explains the new features in Python 3.12, compared to 3.11. - +Python 3.12 will be released on October 2, 2023. For full details, see the :ref:`changelog `. -.. note:: - - Prerelease users should be aware that this document is currently in draft - form. It will be updated substantially as Python 3.12 moves towards release, - so it's worth checking back even after reading earlier versions. +.. seealso:: + :pep:`693` -- Python 3.12 Release Schedule Summary -- Release highlights ============================= @@ -93,6 +90,13 @@ Important deprecations, removals or restrictions: `the migration guide `_ for advice on its replacement. +* :gh:`95299`: Do not pre-install ``setuptools`` in virtual environments + created with :mod:`venv`. + This means that ``distutils``, ``setuptools``, ``pkg_resources``, + and ``easy_install`` will no longer available by default; to access these + run ``pip install setuptools`` in the :ref:`activated ` + virtual environment. + Improved Error Messages ======================= @@ -461,45 +465,9 @@ and others in :gh:`103764`.) Other Language Changes ====================== -* Add :ref:`support for the perf profiler ` through the new - environment variable :envvar:`PYTHONPERFSUPPORT` - and command-line option :option:`-X perf <-X>`, - as well as the new :func:`sys.activate_stack_trampoline`, - :func:`sys.deactivate_stack_trampoline`, - and :func:`sys.is_stack_trampoline_active` functions. - (Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes - with contributions from Gregory P. Smith [Google] and Mark Shannon - in :gh:`96123`.) - -* The extraction methods in :mod:`tarfile`, and :func:`shutil.unpack_archive`, - have a new a *filter* argument that allows limiting tar features than may be - surprising or dangerous, such as creating files outside the destination - directory. - See :ref:`tarfile extraction filters ` for details. - In Python 3.14, the default will switch to ``'data'``. - (Contributed by Petr Viktorin in :pep:`706`.) - -* :class:`types.MappingProxyType` instances are now hashable if the underlying - mapping is hashable. - (Contributed by Serhiy Storchaka in :gh:`87995`.) - -* :class:`memoryview` now supports the half-float type (the "e" format code). - (Contributed by Donghee Na and Antoine Pitrou in :gh:`90751`.) - * The parser now raises :exc:`SyntaxError` when parsing source code containing null bytes. (Contributed by Pablo Galindo in :gh:`96670`.) -* :func:`ast.parse` now raises :exc:`SyntaxError` instead of :exc:`ValueError` - when parsing source code containing null bytes. (Contributed by Pablo Galindo - in :gh:`96670`.) - -* The Garbage Collector now runs only on the eval breaker mechanism of the - Python bytecode evaluation loop instead of object allocations. The GC can - also run when :c:func:`PyErr_CheckSignals` is called so C extensions that - need to run for a long time without executing any Python code also have a - chance to execute the GC periodically. (Contributed by Pablo Galindo in - :gh:`97922`.) - * A backslash-character pair that is not a valid escape sequence now generates a :exc:`SyntaxWarning`, instead of :exc:`DeprecationWarning`. For example, ``re.compile("\d+\.\d+")`` now emits a :exc:`SyntaxWarning` @@ -515,10 +483,6 @@ Other Language Changes In a future Python version they will be eventually a :exc:`SyntaxError`. (Contributed by Victor Stinner in :gh:`98401`.) -* All builtin and extension callables expecting boolean parameters now accept - arguments of any type instead of just :class:`bool` and :class:`int`. - (Contributed by Serhiy Storchaka in :gh:`60203`.) - * Variables used in the target part of comprehensions that are not stored to can now be used in assignment expressions (``:=``). For example, in ``[(b := 1) for a, b.prop in some_iter]``, the assignment to @@ -526,13 +490,6 @@ Other Language Changes part of comprehensions (like ``a``) is still disallowed, as per :pep:`572`. (Contributed by Nikita Sobolev in :gh:`100581`.) -* :class:`slice` objects are now hashable, allowing them to be used as dict keys and - set items. (Contributed by Will Bradshaw, Furkan Onder, and Raymond Hettinger in :gh:`101264`.) - -* :func:`sum` now uses Neumaier summation to improve accuracy and commutativity - when summing floats or mixed ints and floats. - (Contributed by Raymond Hettinger in :gh:`100425`.) - * Exceptions raised in a class or type's ``__set_name__`` method are no longer wrapped by a :exc:`RuntimeError`. Context information is added to the exception as a :pep:`678` note. (Contributed by Irit Katriel in :gh:`77757`.) @@ -542,6 +499,53 @@ Other Language Changes :exc:`ExceptionGroup`. Also changed in version 3.11.4. (Contributed by Irit Katriel in :gh:`103590`.) +* The Garbage Collector now runs only on the eval breaker mechanism of the + Python bytecode evaluation loop instead of object allocations. The GC can + also run when :c:func:`PyErr_CheckSignals` is called so C extensions that + need to run for a long time without executing any Python code also have a + chance to execute the GC periodically. (Contributed by Pablo Galindo in + :gh:`97922`.) + +* All builtin and extension callables expecting boolean parameters now accept + arguments of any type instead of just :class:`bool` and :class:`int`. + (Contributed by Serhiy Storchaka in :gh:`60203`.) + +* :class:`memoryview` now supports the half-float type (the "e" format code). + (Contributed by Donghee Na and Antoine Pitrou in :gh:`90751`.) + +* :class:`slice` objects are now hashable, allowing them to be used as dict keys and + set items. (Contributed by Will Bradshaw, Furkan Onder, and Raymond Hettinger in :gh:`101264`.) + +* :func:`sum` now uses Neumaier summation to improve accuracy and commutativity + when summing floats or mixed ints and floats. + (Contributed by Raymond Hettinger in :gh:`100425`.) + +* :func:`ast.parse` now raises :exc:`SyntaxError` instead of :exc:`ValueError` + when parsing source code containing null bytes. (Contributed by Pablo Galindo + in :gh:`96670`.) + +* The extraction methods in :mod:`tarfile`, and :func:`shutil.unpack_archive`, + have a new a *filter* argument that allows limiting tar features than may be + surprising or dangerous, such as creating files outside the destination + directory. + See :ref:`tarfile extraction filters ` for details. + In Python 3.14, the default will switch to ``'data'``. + (Contributed by Petr Viktorin in :pep:`706`.) + +* :class:`types.MappingProxyType` instances are now hashable if the underlying + mapping is hashable. + (Contributed by Serhiy Storchaka in :gh:`87995`.) + +* Add :ref:`support for the perf profiler ` through the new + environment variable :envvar:`PYTHONPERFSUPPORT` + and command-line option :option:`-X perf <-X>`, + as well as the new :func:`sys.activate_stack_trampoline`, + :func:`sys.deactivate_stack_trampoline`, + and :func:`sys.is_stack_trampoline_active` functions. + (Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes + with contributions from Gregory P. Smith [Google] and Mark Shannon + in :gh:`96123`.) + New Modules =========== @@ -566,7 +570,7 @@ asyncio writing to sockets and uses :meth:`~socket.socket.sendmsg` if the platform supports it. (Contributed by Kumar Aditya in :gh:`91166`.) -* Added :func:`asyncio.eager_task_factory` and :func:`asyncio.create_eager_task_factory` +* Add :func:`asyncio.eager_task_factory` and :func:`asyncio.create_eager_task_factory` functions to allow opting an event loop in to eager task execution, making some use-cases 2x to 5x faster. (Contributed by Jacob Bower & Itamar Oren in :gh:`102853`, :gh:`104140`, and :gh:`104138`) @@ -576,19 +580,10 @@ asyncio :class:`asyncio.ThreadedChildWatcher`. (Contributed by Kumar Aditya in :gh:`98024`.) -* The child watcher classes :class:`asyncio.MultiLoopChildWatcher`, - :class:`asyncio.FastChildWatcher`, :class:`asyncio.AbstractChildWatcher` - and :class:`asyncio.SafeChildWatcher` are deprecated and - will be removed in Python 3.14. It is recommended to not manually - configure a child watcher as the event loop now uses the best available - child watcher for each platform (:class:`asyncio.PidfdChildWatcher` - if supported and :class:`asyncio.ThreadedChildWatcher` otherwise). - (Contributed by Kumar Aditya in :gh:`94597`.) - -* :func:`asyncio.set_child_watcher`, :func:`asyncio.get_child_watcher`, - :meth:`asyncio.AbstractEventLoopPolicy.set_child_watcher` and - :meth:`asyncio.AbstractEventLoopPolicy.get_child_watcher` are deprecated - and will be removed in Python 3.14. +* The event loop now uses the best available child watcher for each platform + (:class:`asyncio.PidfdChildWatcher` if supported and + :class:`asyncio.ThreadedChildWatcher` otherwise), so manually + configuring a child watcher is not recommended. (Contributed by Kumar Aditya in :gh:`94597`.) * Add *loop_factory* parameter to :func:`asyncio.run` to allow specifying @@ -652,7 +647,7 @@ inspect * Add :func:`inspect.getasyncgenstate` and :func:`inspect.getasyncgenlocals` for determining the current state of asynchronous generators. - (Contributed by Thomas Krennwallner in :issue:`35759`.) + (Contributed by Thomas Krennwallner in :gh:`79940`.) * The performance of :func:`inspect.getattr_static` has been considerably improved. Most calls to the function should be at least 2x faster than they @@ -662,17 +657,17 @@ inspect itertools --------- -* Added :class:`itertools.batched()` for collecting into even-sized +* Add :class:`itertools.batched()` for collecting into even-sized tuples where the last batch may be shorter than the rest. (Contributed by Raymond Hettinger in :gh:`98363`.) math ---- -* Added :func:`math.sumprod` for computing a sum of products. +* Add :func:`math.sumprod` for computing a sum of products. (Contributed by Raymond Hettinger in :gh:`100485`.) -* Extended :func:`math.nextafter` to include a *steps* argument +* Extend :func:`math.nextafter` to include a *steps* argument for moving up or down multiple steps at a time. (By Matthias Goergens, Mark Dickinson, and Raymond Hettinger in :gh:`94906`.) @@ -725,7 +720,7 @@ pathlib * Add *walk_up* optional parameter to :meth:`pathlib.PurePath.relative_to` to allow the insertion of ``..`` entries in the result; this behavior is more consistent with :func:`os.path.relpath`. - (Contributed by Domenico Ragusa in :issue:`40358`.) + (Contributed by Domenico Ragusa in :gh:`84538`.) * Add :meth:`pathlib.Path.is_junction` as a proxy to :func:`os.path.isjunction`. (Contributed by Charles Machalow in :gh:`99547`.) @@ -745,10 +740,10 @@ pdb random ------ -* Added :func:`random.binomialvariate`. +* Add :func:`random.binomialvariate`. (Contributed by Raymond Hettinger in :gh:`81620`.) -* Added a default of ``lamb=1.0`` to :func:`random.expovariate`. +* Add a default of ``lambd=1.0`` to :func:`random.expovariate`. (Contributed by Raymond Hettinger in :gh:`100234`.) shutil @@ -807,7 +802,7 @@ sqlite3 statistics ---------- -* Extended :func:`statistics.correlation` to include as a ``ranked`` method +* Extend :func:`statistics.correlation` to include as a ``ranked`` method for computing the Spearman correlation of ranked data. (Contributed by Raymond Hettinger in :gh:`95861`.) @@ -945,7 +940,7 @@ unicodedata unittest -------- -Added ``--durations`` command line option, showing the N slowest test cases:: +Add a ``--durations`` command line option, showing the N slowest test cases:: python3 -m unittest --durations=3 lib.tests.test_threading ..... @@ -961,7 +956,7 @@ Added ``--durations`` command line option, showing the N slowest test cases:: OK (skipped=3) -(Contributed by Giampaolo Rodola in :issue:`4080`) +(Contributed by Giampaolo Rodola in :gh:`48330`) uuid ---- @@ -973,11 +968,11 @@ uuid Optimizations ============= -* Removed ``wstr`` and ``wstr_length`` members from Unicode objects. +* Remove ``wstr`` and ``wstr_length`` members from Unicode objects. It reduces object size by 8 or 16 bytes on 64bit platform. (:pep:`623`) (Contributed by Inada Naoki in :gh:`92536`.) -* Added experimental support for using the BOLT binary optimizer in the build +* Add experimental support for using the BOLT binary optimizer in the build process, which improves performance by 1-5%. (Contributed by Kevin Modzelewski in :gh:`90536` and tuned by Donghee Na in :gh:`101525`) @@ -1010,7 +1005,7 @@ CPython bytecode changes * Remove the :opcode:`!JUMP_IF_FALSE_OR_POP` and :opcode:`!JUMP_IF_TRUE_OR_POP` instructions. (Contributed by Irit Katriel in :gh:`102859`.) -* Removed the :opcode:`!PRECALL` instruction. (Contributed by Mark Shannon in +* Remove the :opcode:`!PRECALL` instruction. (Contributed by Mark Shannon in :gh:`92925`.) * Add the :opcode:`LOAD_FAST_AND_CLEAR` instruction as part of the @@ -1042,15 +1037,52 @@ Demos and Tools Deprecated ========== -* :mod:`asyncio`: The :meth:`~asyncio.get_event_loop` method of the - default event loop policy now emits a :exc:`DeprecationWarning` if there - is no current event loop set and it decides to create one. - (Contributed by Serhiy Storchaka and Guido van Rossum in :gh:`100160`.) +* :mod:`argparse`: The *type*, *choices*, and *metavar* parameters + of :class:`!argparse.BooleanOptionalAction` are deprecated + and will be removed in 3.14. + (Contributed by Nikita Sobolev in :gh:`92248`.) + +* :mod:`ast`: The following :mod:`ast` features have been deprecated in documentation since + Python 3.8, now cause a :exc:`DeprecationWarning` to be emitted at runtime + when they are accessed or used, and will be removed in Python 3.14: + + * :class:`!ast.Num` + * :class:`!ast.Str` + * :class:`!ast.Bytes` + * :class:`!ast.NameConstant` + * :class:`!ast.Ellipsis` + + Use :class:`ast.Constant` instead. + (Contributed by Serhiy Storchaka in :gh:`90953`.) + +* :mod:`asyncio`: + + * The child watcher classes :class:`asyncio.MultiLoopChildWatcher`, + :class:`asyncio.FastChildWatcher`, :class:`asyncio.AbstractChildWatcher` + and :class:`asyncio.SafeChildWatcher` are deprecated and + will be removed in Python 3.14. + (Contributed by Kumar Aditya in :gh:`94597`.) + + * :func:`asyncio.set_child_watcher`, :func:`asyncio.get_child_watcher`, + :meth:`asyncio.AbstractEventLoopPolicy.set_child_watcher` and + :meth:`asyncio.AbstractEventLoopPolicy.get_child_watcher` are deprecated + and will be removed in Python 3.14. + (Contributed by Kumar Aditya in :gh:`94597`.) + + * The :meth:`~asyncio.get_event_loop` method of the + default event loop policy now emits a :exc:`DeprecationWarning` if there + is no current event loop set and it decides to create one. + (Contributed by Serhiy Storchaka and Guido van Rossum in :gh:`100160`.) * :mod:`calendar`: ``calendar.January`` and ``calendar.February`` constants are deprecated and replaced by :data:`calendar.JANUARY` and :data:`calendar.FEBRUARY`. (Contributed by Prince Roshan in :gh:`103636`.) +* :mod:`collections.abc`: Deprecated :class:`collections.abc.ByteString`. + Prefer :class:`Sequence` or :class:`collections.abc.Buffer`. + For use in typing, prefer a union, like ``bytes | bytearray``, or :class:`collections.abc.Buffer`. + (Contributed by Shantanu Jain in :gh:`91896`.) + * :mod:`datetime`: :class:`datetime.datetime`'s :meth:`~datetime.datetime.utcnow` and :meth:`~datetime.datetime.utcfromtimestamp` are deprecated and will be removed in a future version. Instead, use timezone-aware objects to represent @@ -1059,40 +1091,86 @@ Deprecated :const:`datetime.UTC`. (Contributed by Paul Ganssle in :gh:`103857`.) +* :mod:`email`: Deprecate the *isdst* parameter in :func:`email.utils.localtime`. + (Contributed by Alan Williams in :gh:`72346`.) + +* :mod:`importlib.abc`: Deprecated the following classes, scheduled for removal in + Python 3.14: + + * :class:`!importlib.abc.ResourceReader` + * :class:`!importlib.abc.Traversable` + * :class:`!importlib.abc.TraversableResources` + + Use :mod:`importlib.resources.abc` classes instead: + + * :class:`importlib.resources.abc.Traversable` + * :class:`importlib.resources.abc.TraversableResources` + + (Contributed by Jason R. Coombs and Hugo van Kemenade in :gh:`93963`.) + +* :mod:`itertools`: Deprecate the support for copy, deepcopy, and pickle operations, + which is undocumented, inefficient, historically buggy, and inconsistent. + This will be removed in 3.14 for a significant reduction in code + volume and maintenance burden. + (Contributed by Raymond Hettinger in :gh:`101588`.) + * :mod:`os`: The ``st_ctime`` fields return by :func:`os.stat` and :func:`os.lstat` on Windows are deprecated. In a future release, they will contain the last metadata change time, consistent with other platforms. For now, they still contain the creation time, which is also available in the new ``st_birthtime`` field. (Contributed by Steve Dower in :gh:`99726`.) +* :mod:`multiprocessing`: In Python 3.14, the default :mod:`multiprocessing` + start method will change to a safer one on Linux, BSDs, + and other non-macOS POSIX platforms where ``'fork'`` is currently + the default (:gh:`84559`). Adding a runtime warning about this was deemed too + disruptive as the majority of code is not expected to care. Use the + :func:`~multiprocessing.get_context` or + :func:`~multiprocessing.set_start_method` APIs to explicitly specify when + your code *requires* ``'fork'``. See :ref:`contexts and start methods + `. + +* :mod:`pkgutil`: :func:`pkgutil.find_loader` and :func:`pkgutil.get_loader` + are deprecated and will be removed in Python 3.14; + use :func:`importlib.util.find_spec` instead. + (Contributed by Nikita Sobolev in :gh:`97850`.) + +* :mod:`pty`: The module has two undocumented ``master_open()`` and ``slave_open()`` + functions that have been deprecated since Python 2 but only gained a + proper :exc:`DeprecationWarning` in 3.12. Remove them in 3.14. + (Contributed by Soumendra Ganguly and Gregory P. Smith in :gh:`85984`.) + * :mod:`os`: On POSIX platforms, :func:`os.fork` can now raise a :exc:`DeprecationWarning` when it can detect being called from a multithreaded process. There has always been a fundamental incompatibility with the POSIX platform when doing so. Even if such code *appeared* to work. We added the warning to to raise awareness as issues encounted by code doing this are becoming more frequent. See the :func:`os.fork` documentation for - more details. + more details along with `this discussion on fork being incompatible with threads + `_ for *why* we're now surfacing this + longstanding platform compatibility problem to developers. When this warning appears due to usage of :mod:`multiprocessing` or :mod:`concurrent.futures` the fix is to use a different :mod:`multiprocessing` start method such as ``"spawn"`` or ``"forkserver"``. -* :mod:`shutil`: The *onerror* argument of :func:`shutil.rmtree` is deprecated as will be removed +* :mod:`shutil`: The *onerror* argument of :func:`shutil.rmtree` is deprecated and will be removed in Python 3.14. Use *onexc* instead. (Contributed by Irit Katriel in :gh:`102828`.) * :mod:`sqlite3`: - * :ref:`default adapters and converters - ` are now deprecated. - Instead, use the :ref:`sqlite3-adapter-converter-recipes` - and tailor them to your needs. - (Contributed by Erlend E. Aasland in :gh:`90016`.) - - * In :meth:`~sqlite3.Cursor.execute`, :exc:`DeprecationWarning` is now emitted - when :ref:`named placeholders ` are used together with - parameters supplied as a :term:`sequence` instead of as a :class:`dict`. - Starting from Python 3.14, using named placeholders with parameters supplied - as a sequence will raise a :exc:`~sqlite3.ProgrammingError`. - (Contributed by Erlend E. Aasland in :gh:`101698`.) + + * :ref:`default adapters and converters + ` are now deprecated. + Instead, use the :ref:`sqlite3-adapter-converter-recipes` + and tailor them to your needs. + (Contributed by Erlend E. Aasland in :gh:`90016`.) + + * In :meth:`~sqlite3.Cursor.execute`, :exc:`DeprecationWarning` is now emitted + when :ref:`named placeholders ` are used together with + parameters supplied as a :term:`sequence` instead of as a :class:`dict`. + Starting from Python 3.14, using named placeholders with parameters supplied + as a sequence will raise a :exc:`~sqlite3.ProgrammingError`. + (Contributed by Erlend E. Aasland in :gh:`101698`.) * :mod:`sys`: The :data:`sys.last_type`, :data:`sys.last_value` and :data:`sys.last_traceback` fields are deprecated. Use :data:`sys.last_exc` instead. @@ -1102,16 +1180,24 @@ Deprecated Python 3.14, when ``'data'`` filter will become the default. See :ref:`tarfile-extraction-filter` for details. -* :mod:`typing`: :class:`typing.Hashable` and :class:`typing.Sized` aliases for :class:`collections.abc.Hashable` - and :class:`collections.abc.Sized`. (:gh:`94309`.) +* :mod:`typing`: + + * :class:`typing.Hashable` and :class:`typing.Sized` aliases for :class:`collections.abc.Hashable` + and :class:`collections.abc.Sized`. (:gh:`94309`.) + + * :class:`typing.ByteString`, deprecated since Python 3.9, now causes a + :exc:`DeprecationWarning` to be emitted when it is used. + (Contributed by Alex Waygood in :gh:`91896`.) * :mod:`xml.etree.ElementTree`: The module now emits :exc:`DeprecationWarning` when testing the truth value of an :class:`xml.etree.ElementTree.Element`. Before, the Python implementation emitted :exc:`FutureWarning`, and the C implementation emitted nothing. + (Contributed by Jacob Walls in :gh:`83122`.) -* The 3-arg signatures (type, value, traceback) of :meth:`~coroutine.throw`, - :meth:`~generator.throw` and :meth:`~agen.athrow` are deprecated and +* The 3-arg signatures (type, value, traceback) of :meth:`coroutine throw() + `, :meth:`generator throw() ` and + :meth:`async generator throw() ` are deprecated and may be removed in a future version of Python. Use the single-arg versions of these functions instead. (Contributed by Ofey Chan in :gh:`89874`.) @@ -1120,17 +1206,21 @@ Deprecated :exc:`ImportWarning`). (Contributed by Brett Cannon in :gh:`65961`.) -* In accordance with :pep:`699`, the ``ma_version_tag`` field in :c:type:`PyDictObject` - is deprecated for extension modules. Accessing this field will generate a compiler - warning at compile time. This field will be removed in Python 3.14. - (Contributed by Ramvikrams and Kumar Aditya in :gh:`101193`. PEP by Ken Jin.) +* Setting ``__package__`` or ``__cached__`` on a module is deprecated, + and will cease to be set or taken into consideration by the import system in Python 3.14. + (Contributed by Brett Cannon in :gh:`65961`.) * The bitwise inversion operator (``~``) on bool is deprecated. It will throw an error in Python 3.14. Use ``not`` for logical negation of bools instead. In the rare case that you really need the bitwise inversion of the underlying - ``int``, convert to int explicitly with ``~int(x)``. (Contributed by Tim Hoffmann + ``int``, convert to int explicitly: ``~int(x)``. (Contributed by Tim Hoffmann in :gh:`103487`.) +* Accessing ``co_lnotab`` on code objects was deprecated in Python 3.10 via :pep:`626`, + but it only got a proper :exc:`DeprecationWarning` in 3.12, + therefore it will be removed in 3.14. + (Contributed by Nikita Sobolev in :gh:`101866`.) + Pending Removal in Python 3.13 ------------------------------ @@ -1179,14 +1269,13 @@ APIs: Pending Removal in Python 3.14 ------------------------------ +The following APIs have been deprecated +and will be removed in Python 3.14. + * :mod:`argparse`: The *type*, *choices*, and *metavar* parameters - of :class:`!argparse.BooleanOptionalAction` are deprecated - and will be removed in 3.14. - (Contributed by Nikita Sobolev in :gh:`92248`.) + of :class:`!argparse.BooleanOptionalAction` -* :mod:`ast`: The following :mod:`ast` features have been deprecated in documentation since - Python 3.8, now cause a :exc:`DeprecationWarning` to be emitted at runtime - when they are accessed or used, and will be removed in Python 3.14: +* :mod:`ast`: * :class:`!ast.Num` * :class:`!ast.Str` @@ -1194,81 +1283,48 @@ Pending Removal in Python 3.14 * :class:`!ast.NameConstant` * :class:`!ast.Ellipsis` - Use :class:`ast.Constant` instead. - (Contributed by Serhiy Storchaka in :gh:`90953`.) +* :mod:`asyncio`: -* :mod:`asyncio`: the *msg* parameter of both - :meth:`asyncio.Future.cancel` and - :meth:`asyncio.Task.cancel` (:gh:`90985`) + * :class:`!asyncio.MultiLoopChildWatcher` + * :class:`!asyncio.FastChildWatcher` + * :class:`!asyncio.AbstractChildWatcher` + * :class:`!asyncio.SafeChildWatcher` + * :func:`!asyncio.set_child_watcher` + * :func:`!asyncio.get_child_watcher`, + * :meth:`!asyncio.AbstractEventLoopPolicy.set_child_watcher` + * :meth:`!asyncio.AbstractEventLoopPolicy.get_child_watcher` -* :mod:`collections.abc`: Deprecated :class:`collections.abc.ByteString`. - Prefer :class:`Sequence` or :class:`collections.abc.Buffer`. - For use in typing, prefer a union, like ``bytes | bytearray``, or :class:`collections.abc.Buffer`. - (Contributed by Shantanu Jain in :gh:`91896`.) +* :mod:`collections.abc`: :class:`!collections.abc.ByteString`. -* :mod:`email`: Deprecated the *isdst* parameter in :func:`email.utils.localtime`. - (Contributed by Alan Williams in :gh:`72346`.) +* :mod:`email`: the *isdst* parameter in :func:`email.utils.localtime`. -* :mod:`importlib.abc`: Deprecated the following classes, scheduled for removal in - Python 3.14: +* :mod:`importlib.abc`: * :class:`!importlib.abc.ResourceReader` * :class:`!importlib.abc.Traversable` * :class:`!importlib.abc.TraversableResources` - Use :mod:`importlib.resources.abc` classes instead: - - * :class:`importlib.resources.abc.Traversable` - * :class:`importlib.resources.abc.TraversableResources` - - (Contributed by Jason R. Coombs and Hugo van Kemenade in :gh:`93963`.) - -* :mod:`itertools`: The module had undocumented, inefficient, historically buggy, - and inconsistent support for copy, deepcopy, and pickle operations. - This will be removed in 3.14 for a significant reduction in code - volume and maintenance burden. - (Contributed by Raymond Hettinger in :gh:`101588`.) - -* :mod:`multiprocessing`: The default :mod:`multiprocessing` start method will change to a safer one on - Linux, BSDs, and other non-macOS POSIX platforms where ``'fork'`` is currently - the default (:gh:`84559`). Adding a runtime warning about this was deemed too - disruptive as the majority of code is not expected to care. Use the - :func:`~multiprocessing.get_context` or - :func:`~multiprocessing.set_start_method` APIs to explicitly specify when - your code *requires* ``'fork'``. See :ref:`multiprocessing-start-methods`. +* :mod:`itertools`: Support for copy, deepcopy, and pickle operations. -* :mod:`pkgutil`: :func:`pkgutil.find_loader` and :func:`pkgutil.get_loader` - now raise :exc:`DeprecationWarning`; - use :func:`importlib.util.find_spec` instead. - (Contributed by Nikita Sobolev in :gh:`97850`.) +* :mod:`pkgutil`: -* :mod:`pty`: The module has two undocumented ``master_open()`` and ``slave_open()`` - functions that have been deprecated since Python 2 but only gained a - proper :exc:`DeprecationWarning` in 3.12. Remove them in 3.14. + * :func:`!pkgutil.find_loader` + * :func:`!pkgutil.get_loader`. -* :mod:`shutil`: The *onerror* argument of :func:`shutil.rmtree` is deprecated in 3.12, - and will be removed in 3.14. +* :mod:`pty`: -* :mod:`typing`: :class:`typing.ByteString`, deprecated since Python 3.9, now causes a - :exc:`DeprecationWarning` to be emitted when it is used. + * :func:`!pty.master_open` + * :func:`!pty.slave_open` -* :mod:`xml.etree.ElementTree`: Testing the truth value of an :class:`xml.etree.ElementTree.Element` - is deprecated and will raise an exception in Python 3.14. +* :mod:`shutil`: The *onerror* argument of :func:`shutil.rmtree` -* Creating immutable types (:c:macro:`Py_TPFLAGS_IMMUTABLETYPE`) with mutable - bases using the C API (:gh:`95388`). +* :mod:`typing`: :class:`!typing.ByteString` -* ``__package__`` and ``__cached__`` will cease to be set or taken - into consideration by the import system (:gh:`97879`). +* :mod:`xml.etree.ElementTree`: Testing the truth value of an :class:`xml.etree.ElementTree.Element`. -* Accessing ``co_lnotab`` was deprecated in :pep:`626` since 3.10 - and was planned to be removed in 3.12 - but it only got a proper :exc:`DeprecationWarning` in 3.12. - May be removed in 3.14. - (Contributed by Nikita Sobolev in :gh:`101866`.) +* The ``__package__`` and ``__cached__`` attributes on module objects. -* Creating :c:data:`immutable types ` with mutable - bases using the C API (:gh:`95388`) +* The ``co_lnotab`` attribute of code objects. Pending Removal in Future Versions ---------------------------------- @@ -1402,7 +1458,7 @@ imp * The :mod:`!imp` module has been removed. (Contributed by Barry Warsaw in :gh:`98040`.) -* Replace removed :mod:`!imp` functions with :mod:`importlib` functions: + To migrate, consult the following correspondence table: ================================= ======================================= imp importlib @@ -1417,9 +1473,10 @@ imp ``imp.new_module(name)`` ``types.ModuleType(name)`` ``imp.reload()`` :func:`importlib.reload` ``imp.source_from_cache()`` :func:`importlib.util.source_from_cache` + ``imp.load_source()`` *See below* ================================= ======================================= -* Replace ``imp.load_source()`` with:: + Replace ``imp.load_source()`` with:: import importlib.util import importlib.machinery @@ -1434,9 +1491,9 @@ imp loader.exec_module(module) return module -* Removed :mod:`!imp` functions and attributes with no replacements: +* Remove :mod:`!imp` functions and attributes with no replacements: - * undocumented functions: + * Undocumented functions: * ``imp.init_builtin()`` * ``imp.load_compiled()`` @@ -1517,7 +1574,7 @@ ssl unittest -------- -* Removed many old deprecated :mod:`unittest` features: +* Remove many long-deprecated :mod:`unittest` features: * A number of :class:`~unittest.TestCase` method aliases: @@ -1554,13 +1611,13 @@ unittest * An alias of the :class:`~unittest.TextTestResult` class: ``_TextTestResult`` (deprecated in Python 3.2). - (Contributed by Serhiy Storchaka in :issue:`45162`.) + (Contributed by Serhiy Storchaka in :gh:`89325`.) webbrowser ---------- * Remove support for obsolete browsers from :mod:`webbrowser`. - Removed browsers include: Grail, Mosaic, Netscape, Galeon, Skipstone, + The removed browsers include: Grail, Mosaic, Netscape, Galeon, Skipstone, Iceape, Firebird, and Firefox versions 35 and below (:gh:`102871`). xml.etree.ElementTree @@ -1583,8 +1640,8 @@ zipimport Others ------ -* Removed the ``suspicious`` rule from the documentation Makefile, and - removed ``Doc/tools/rstlint.py``, both in favor of `sphinx-lint +* Remove the ``suspicious`` rule from the documentation :file:`Makefile` and + :file:`Doc/tools/rstlint.py`, both in favor of `sphinx-lint `_. (Contributed by Julien Palard in :gh:`98179`.) @@ -1614,7 +1671,7 @@ Changes in the Python API contain ASCII letters and digits and underscore. (Contributed by Serhiy Storchaka in :gh:`91760`.) -* Removed ``randrange()`` functionality deprecated since Python 3.10. Formerly, +* Remove ``randrange()`` functionality deprecated since Python 3.10. Formerly, ``randrange(10.0)`` losslessly converted to ``randrange(10)``. Now, it raises a :exc:`TypeError`. Also, the exception raised for non-integer values such as ``randrange(10.5)`` or ``randrange('10')`` has been changed from :exc:`ValueError` to @@ -1628,7 +1685,7 @@ Changes in the Python API to :term:`filesystem encoding and error handler`. Argument files should be encoded in UTF-8 instead of ANSI Codepage on Windows. -* Removed the ``asyncore``-based ``smtpd`` module deprecated in Python 3.4.7 +* Remove the ``asyncore``-based ``smtpd`` module deprecated in Python 3.4.7 and 3.5.4. A recommended replacement is the :mod:`asyncio`-based aiosmtpd_ PyPI module. @@ -1710,9 +1767,9 @@ Changes in the Python API Build Changes ============= -* Python no longer uses ``setup.py`` to build shared C extension modules. +* Python no longer uses :file:`setup.py` to build shared C extension modules. Build parameters like headers and libraries are detected in ``configure`` - script. Extensions are built by ``Makefile``. Most extensions use + script. Extensions are built by :file:`Makefile`. Most extensions use ``pkg-config`` and fall back to manual detection. (Contributed by Christian Heimes in :gh:`93939`.) @@ -1725,7 +1782,7 @@ Build Changes if the Clang compiler accepts the flag. (Contributed by Donghee Na in :gh:`89536`.) -* Add ``COMPILEALL_OPTS`` variable in Makefile to override :mod:`compileall` +* Add ``COMPILEALL_OPTS`` variable in :file:`Makefile` to override :mod:`compileall` options (default: ``-j0``) in ``make install``. Also merged the 3 ``compileall`` commands into a single command to build .pyc files for all optimization levels (0, 1, 2) at once. @@ -1753,7 +1810,7 @@ New Features ------------ -* :pep:`697`: Introduced the :ref:`Unstable C API tier `, +* :pep:`697`: Introduce the :ref:`Unstable C API tier `, intended for low-level tools like debuggers and JIT compilers. This API may change in each minor release of CPython without deprecation warnings. @@ -1775,7 +1832,7 @@ New Features (Contributed by Petr Viktorin in :gh:`101101`.) -* :pep:`697`: Added API for extending types whose instance memory layout is +* :pep:`697`: Add an API for extending types whose instance memory layout is opaque: - :c:member:`PyType_Spec.basicsize` can be zero or negative to specify @@ -1790,7 +1847,7 @@ New Features (Contributed by Petr Viktorin in :gh:`103509`.) -* Added the new :ref:`limited C API ` function :c:func:`PyType_FromMetaclass`, +* Add the new :ref:`limited C API ` function :c:func:`PyType_FromMetaclass`, which generalizes the existing :c:func:`PyType_FromModuleAndSpec` using an additional metaclass argument. (Contributed by Wenzel Jakob in :gh:`93012`.) @@ -1829,13 +1886,13 @@ New Features protocol are now available in the :ref:`Limited API `. (Contributed by Wenzel Jakob in :gh:`98586`.) -* Added two new public functions, +* Add two new public functions, :c:func:`PyEval_SetProfileAllThreads` and :c:func:`PyEval_SetTraceAllThreads`, that allow to set tracing and profiling functions in all running threads in addition to the calling one. (Contributed by Pablo Galindo in :gh:`93503`.) -* Added new function :c:func:`PyFunction_SetVectorcall` to the C API +* Add new function :c:func:`PyFunction_SetVectorcall` to the C API which sets the vectorcall field of a given :c:type:`PyFunctionObject`. (Contributed by Andrew Frost in :gh:`92257`.) @@ -1845,11 +1902,11 @@ New Features compilers, or debuggers. (Contributed by Carl Meyer in :gh:`91052`.) -* Added :c:func:`PyType_AddWatcher` and :c:func:`PyType_Watch` API to register +* Add :c:func:`PyType_AddWatcher` and :c:func:`PyType_Watch` API to register callbacks to receive notification on changes to a type. (Contributed by Carl Meyer in :gh:`91051`.) -* Added :c:func:`PyCode_AddWatcher` and :c:func:`PyCode_ClearWatcher` +* Add :c:func:`PyCode_AddWatcher` and :c:func:`PyCode_ClearWatcher` APIs to register callbacks to receive notification on creation and destruction of code objects. (Contributed by Itamar Oren in :gh:`91054`.) @@ -1879,8 +1936,8 @@ New Features to replace the legacy-api :c:func:`!PyErr_Display`. (Contributed by Irit Katriel in :gh:`102755`). -* :pep:`683`: Introduced Immortal Objects to Python which allows objects - to bypass reference counts and introduced changes to the C-API: +* :pep:`683`: Introduce *Immortal Objects*, which allows objects + to bypass reference counts, and related changes to the C-API: - ``_Py_IMMORTAL_REFCNT``: The reference count that defines an object as immortal. @@ -1892,12 +1949,12 @@ New Features - ``SSTATE_INTERNED_IMMORTAL_STATIC`` An identifier for interned unicode objects that are immortal and static - ``sys.getunicodeinternedsize`` This returns the total number of unicode - objects that have been interned. This is now needed for refleak.py to + objects that have been interned. This is now needed for :file:`refleak.py` to correctly track reference counts and allocated blocks (Contributed by Eddie Elizondo in :gh:`84436`.) -* :pep:`684`: Added the new :c:func:`Py_NewInterpreterFromConfig` +* :pep:`684`: Add the new :c:func:`Py_NewInterpreterFromConfig` function and :c:type:`PyInterpreterConfig`, which may be used to create sub-interpreters with their own GILs. (See :ref:`whatsnew312-pep684` for more info.) @@ -1946,7 +2003,7 @@ Porting to Python 3.12 copied as-is to the result string, and any extra arguments discarded. (Contributed by Serhiy Storchaka in :gh:`95781`.) -* Fixed wrong sign placement in :c:func:`PyUnicode_FromFormat` and +* Fix wrong sign placement in :c:func:`PyUnicode_FromFormat` and :c:func:`PyUnicode_FromFormatV`. (Contributed by Philip Georgi in :gh:`95504`.) @@ -2051,6 +2108,11 @@ Porting to Python 3.12 Deprecated ---------- +* In accordance with :pep:`699`, the ``ma_version_tag`` field in :c:type:`PyDictObject` + is deprecated for extension modules. Accessing this field will generate a compiler + warning at compile time. This field will be removed in Python 3.14. + (Contributed by Ramvikrams and Kumar Aditya in :gh:`101193`. PEP by Ken Jin.) + * Deprecate global configuration variable: * :c:var:`Py_DebugFlag`: use :c:member:`PyConfig.parser_debug` @@ -2080,13 +2142,13 @@ Deprecated :c:type:`PyConfig` instead. (Contributed by Victor Stinner in :gh:`77782`.) -* Creating immutable types (:c:macro:`Py_TPFLAGS_IMMUTABLETYPE`) with mutable - bases is deprecated and will be disabled in Python 3.14. +* Creating :c:data:`immutable types ` with mutable + bases is deprecated and will be disabled in Python 3.14. (:gh:`95388`) -* The ``structmember.h`` header is deprecated, though it continues to be +* The :file:`structmember.h` header is deprecated, though it continues to be available and there are no plans to remove it. - Its contents are now available just by including ``Python.h``, + Its contents are now available just by including :file:`Python.h`, with a ``Py`` prefix added if it was missing: - :c:struct:`PyMemberDef`, :c:func:`PyMember_GetOne` and @@ -2096,14 +2158,14 @@ Deprecated - The flags :c:macro:`Py_READONLY` (previously ``READONLY``) and :c:macro:`Py_AUDIT_READ` (previously all uppercase) - Several items are not exposed from ``Python.h``: + Several items are not exposed from :file:`Python.h`: - :c:macro:`T_OBJECT` (use :c:macro:`Py_T_OBJECT_EX`) - :c:macro:`T_NONE` (previously undocumented, and pretty quirky) - The macro ``WRITE_RESTRICTED`` which does nothing. - The macros ``RESTRICTED`` and ``READ_RESTRICTED``, equivalents of :c:macro:`Py_AUDIT_READ`. - - In some configurations, ```` is not included from ``Python.h``. + - In some configurations, ```` is not included from :file:`Python.h`. It should be included manually when using ``offsetof()``. The deprecated header continues to provide its original @@ -2130,11 +2192,98 @@ Deprecated overrides :c:member:`~PyTypeObject.tp_new` is deprecated. Call the metaclass instead. +Pending Removal in Python 3.14 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* The ``ma_version_tag`` field in :c:type:`PyDictObject` for extension modules + (:pep:`699`; :gh:`101193`). + +* Global configuration variables: + + * :c:var:`Py_DebugFlag`: use :c:member:`PyConfig.parser_debug` + * :c:var:`Py_VerboseFlag`: use :c:member:`PyConfig.verbose` + * :c:var:`Py_QuietFlag`: use :c:member:`PyConfig.quiet` + * :c:var:`Py_InteractiveFlag`: use :c:member:`PyConfig.interactive` + * :c:var:`Py_InspectFlag`: use :c:member:`PyConfig.inspect` + * :c:var:`Py_OptimizeFlag`: use :c:member:`PyConfig.optimization_level` + * :c:var:`Py_NoSiteFlag`: use :c:member:`PyConfig.site_import` + * :c:var:`Py_BytesWarningFlag`: use :c:member:`PyConfig.bytes_warning` + * :c:var:`Py_FrozenFlag`: use :c:member:`PyConfig.pathconfig_warnings` + * :c:var:`Py_IgnoreEnvironmentFlag`: use :c:member:`PyConfig.use_environment` + * :c:var:`Py_DontWriteBytecodeFlag`: use :c:member:`PyConfig.write_bytecode` + * :c:var:`Py_NoUserSiteDirectory`: use :c:member:`PyConfig.user_site_directory` + * :c:var:`Py_UnbufferedStdioFlag`: use :c:member:`PyConfig.buffered_stdio` + * :c:var:`Py_HashRandomizationFlag`: use :c:member:`PyConfig.use_hash_seed` + and :c:member:`PyConfig.hash_seed` + * :c:var:`Py_IsolatedFlag`: use :c:member:`PyConfig.isolated` + * :c:var:`Py_LegacyWindowsFSEncodingFlag`: use :c:member:`PyPreConfig.legacy_windows_fs_encoding` + * :c:var:`Py_LegacyWindowsStdioFlag`: use :c:member:`PyConfig.legacy_windows_stdio` + * :c:var:`!Py_FileSystemDefaultEncoding`: use :c:member:`PyConfig.filesystem_encoding` + * :c:var:`!Py_HasFileSystemDefaultEncoding`: use :c:member:`PyConfig.filesystem_encoding` + * :c:var:`!Py_FileSystemDefaultEncodeErrors`: use :c:member:`PyConfig.filesystem_errors` + * :c:var:`!Py_UTF8Mode`: use :c:member:`PyPreConfig.utf8_mode` (see :c:func:`Py_PreInitialize`) + + The :c:func:`Py_InitializeFromConfig` API should be used with + :c:type:`PyConfig` instead. + +* Creating :c:data:`immutable types ` with mutable + bases (:gh:`95388`). + +Pending Removal in Python 3.15 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* :c:func:`PyImport_ImportModuleNoBlock`: use :c:func:`PyImport_ImportModule` +* :c:type:`!Py_UNICODE_WIDE` type: use :c:type:`wchar_t` +* :c:type:`Py_UNICODE` type: use :c:type:`wchar_t` +* Python initialization functions: + + * :c:func:`PySys_ResetWarnOptions`: clear :data:`sys.warnoptions` and + :data:`!warnings.filters` + * :c:func:`Py_GetExecPrefix`: get :data:`sys.exec_prefix` + * :c:func:`Py_GetPath`: get :data:`sys.path` + * :c:func:`Py_GetPrefix`: get :data:`sys.prefix` + * :c:func:`Py_GetProgramFullPath`: get :data:`sys.executable` + * :c:func:`Py_GetProgramName`: get :data:`sys.executable` + * :c:func:`Py_GetPythonHome`: get :c:member:`PyConfig.home` or + the :envvar:`PYTHONHOME` environment variable + +Pending Removal in Future Versions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following APIs are deprecated and will be removed, +although there is currently no date scheduled for their removal. + +* :c:macro:`Py_TPFLAGS_HAVE_FINALIZE`: unneeded since Python 3.8 +* :c:func:`PyErr_Fetch`: use :c:func:`PyErr_GetRaisedException` +* :c:func:`PyErr_NormalizeException`: use :c:func:`PyErr_GetRaisedException` +* :c:func:`PyErr_Restore`: use :c:func:`PyErr_SetRaisedException` +* :c:func:`PyModule_GetFilename`: use :c:func:`PyModule_GetFilenameObject` +* :c:func:`PyOS_AfterFork`: use :c:func:`PyOS_AfterFork_Child` +* :c:func:`PySlice_GetIndicesEx`: use :c:func:`PySlice_Unpack` and :c:func:`PySlice_AdjustIndices` +* :c:func:`!PyUnicode_AsDecodedObject`: use :c:func:`PyCodec_Decode` +* :c:func:`!PyUnicode_AsDecodedUnicode`: use :c:func:`PyCodec_Decode` +* :c:func:`!PyUnicode_AsEncodedObject`: use :c:func:`PyCodec_Encode` +* :c:func:`!PyUnicode_AsEncodedUnicode`: use :c:func:`PyCodec_Encode` +* :c:func:`PyUnicode_READY`: unneeded since Python 3.12 +* :c:func:`!PyErr_Display`: use :c:func:`PyErr_DisplayException` +* :c:func:`!_PyErr_ChainExceptions`: use ``_PyErr_ChainExceptions1`` +* :c:member:`!PyBytesObject.ob_shash` member: + call :c:func:`PyObject_Hash` instead +* :c:member:`!PyDictObject.ma_version_tag` member +* Thread Local Storage (TLS) API: + + * :c:func:`PyThread_create_key`: use :c:func:`PyThread_tss_alloc` + * :c:func:`PyThread_delete_key`: use :c:func:`PyThread_tss_free` + * :c:func:`PyThread_set_key_value`: use :c:func:`PyThread_tss_set` + * :c:func:`PyThread_get_key_value`: use :c:func:`PyThread_tss_get` + * :c:func:`PyThread_delete_key_value`: use :c:func:`PyThread_tss_delete` + * :c:func:`PyThread_ReInitTLS`: unneeded since Python 3.7 + Removed ------- -* Remove the ``token.h`` header file. There was never any public tokenizer C - API. The ``token.h`` header file was only designed to be used by Python +* Remove the :file:`token.h` header file. There was never any public tokenizer C + API. The :file:`token.h` header file was only designed to be used by Python internals. (Contributed by Victor Stinner in :gh:`92651`.) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index edc64fda2a6ad6..21a1b24194bcd8 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -17,7 +17,6 @@ import itertools import math import types -import warnings import weakref from types import GenericAlias diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py index 48d8db3ed423a5..73bdcbe8693991 100644 --- a/Lib/concurrent/futures/process.py +++ b/Lib/concurrent/futures/process.py @@ -444,24 +444,14 @@ def process_result_item(self, result_item): # Process the received a result_item. This can be either the PID of a # worker that exited gracefully or a _ResultItem - if isinstance(result_item, int): - # Clean shutdown of a worker using its PID - # (avoids marking the executor broken) - assert self.is_shutting_down() - p = self.processes.pop(result_item) - p.join() - if not self.processes: - self.join_executor_internals() - return - else: - # Received a _ResultItem so mark the future as completed. - work_item = self.pending_work_items.pop(result_item.work_id, None) - # work_item can be None if another process terminated (see above) - if work_item is not None: - if result_item.exception: - work_item.future.set_exception(result_item.exception) - else: - work_item.future.set_result(result_item.result) + # Received a _ResultItem so mark the future as completed. + work_item = self.pending_work_items.pop(result_item.work_id, None) + # work_item can be None if another process terminated (see above) + if work_item is not None: + if result_item.exception: + work_item.future.set_exception(result_item.exception) + else: + work_item.future.set_result(result_item.result) def is_shutting_down(self): # Check whether we should start shutting down the executor. @@ -521,7 +511,8 @@ def terminate_broken(self, cause): # gh-107219: Close the connection writer which can unblock # Queue._feed() if it was stuck in send_bytes(). - self.call_queue._writer.close() + if sys.platform == 'win32': + self.call_queue._writer.close() # clean up resources self.join_executor_internals() diff --git a/Lib/functools.py b/Lib/functools.py index 6cb532323b1d67..55990e742bf23f 100644 --- a/Lib/functools.py +++ b/Lib/functools.py @@ -19,8 +19,9 @@ # import types, weakref # Deferred to single_dispatch() from reprlib import recursive_repr from _thread import RLock -from types import GenericAlias +# Avoid importing types, so we can speedup import time +GenericAlias = type(list[int]) ################################################################################ ### update_wrapper() and wraps() decorator diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py index 7c425a2d8e7034..dbbf106f680964 100644 --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -42,7 +42,6 @@ BUFSIZE = 8192 # A very generous timeout when it comes to local connections... CONNECTION_TIMEOUT = 20. -WSA_OPERATION_ABORTED = 995 _mmap_counter = itertools.count() @@ -300,7 +299,7 @@ def _send_bytes(self, buf): finally: self._send_ov = None nwritten, err = ov.GetOverlappedResult(True) - if err == WSA_OPERATION_ABORTED: + if err == _winapi.ERROR_OPERATION_ABORTED: # close() was called by another thread while # WaitForMultipleObjects() was waiting for the overlapped # operation. diff --git a/Lib/multiprocessing/resource_tracker.py b/Lib/multiprocessing/resource_tracker.py index 3783c1ffc6e4a9..8e41f461cc934e 100644 --- a/Lib/multiprocessing/resource_tracker.py +++ b/Lib/multiprocessing/resource_tracker.py @@ -51,15 +51,31 @@ }) +class ReentrantCallError(RuntimeError): + pass + + class ResourceTracker(object): def __init__(self): - self._lock = threading.Lock() + self._lock = threading.RLock() self._fd = None self._pid = None + def _reentrant_call_error(self): + # gh-109629: this happens if an explicit call to the ResourceTracker + # gets interrupted by a garbage collection, invoking a finalizer (*) + # that itself calls back into ResourceTracker. + # (*) for example the SemLock finalizer + raise ReentrantCallError( + "Reentrant call into the multiprocessing resource tracker") + def _stop(self): with self._lock: + # This should not happen (_stop() isn't called by a finalizer) + # but we check for it anyway. + if self._lock._recursion_count() > 1: + return self._reentrant_call_error() if self._fd is None: # not running return @@ -81,6 +97,9 @@ def ensure_running(self): This can be run from any process. Usually a child process will use the resource created by its parent.''' with self._lock: + if self._lock._recursion_count() > 1: + # The code below is certainly not reentrant-safe, so bail out + return self._reentrant_call_error() if self._fd is not None: # resource tracker was launched before, is it still running? if self._check_alive(): @@ -159,7 +178,17 @@ def unregister(self, name, rtype): self._send('UNREGISTER', name, rtype) def _send(self, cmd, name, rtype): - self.ensure_running() + try: + self.ensure_running() + except ReentrantCallError: + # The code below might or might not work, depending on whether + # the resource tracker was already running and still alive. + # Better warn the user. + # (XXX is warnings.warn itself reentrant-safe? :-) + warnings.warn( + f"ResourceTracker called reentrantly for resource cleanup, " + f"which is unsupported. " + f"The {rtype} object {name!r} might leak.") msg = '{0}:{1}:{2}\n'.format(cmd, name, rtype).encode('ascii') if len(msg) > 512: # posix guarantees that writes to a pipe of less than PIPE_BUF @@ -176,6 +205,7 @@ def _send(self, cmd, name, rtype): unregister = _resource_tracker.unregister getfd = _resource_tracker.getfd + def main(fd): '''Run resource tracker.''' # protect the process from ^C and "killall python" etc diff --git a/Lib/pathlib.py b/Lib/pathlib.py index 8f4125f2cfc3ba..b8a619507150f7 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -1141,26 +1141,7 @@ def resolve(self, strict=False): normalizing it. """ - def check_eloop(e): - winerror = getattr(e, 'winerror', 0) - if e.errno == ELOOP or winerror == _WINERROR_CANT_RESOLVE_FILENAME: - raise RuntimeError("Symlink loop from %r" % e.filename) - - try: - s = os.path.realpath(self, strict=strict) - except OSError as e: - check_eloop(e) - raise - p = self.with_segments(s) - - # In non-strict mode, realpath() doesn't raise on symlink loops. - # Ensure we get an exception by calling stat() - if not strict: - try: - p.stat() - except OSError as e: - check_eloop(e) - return p + return self.with_segments(os.path.realpath(self, strict=strict)) def owner(self): """ diff --git a/Lib/random.py b/Lib/random.py index 84bbfc5df1bf23..1d789b107904fb 100644 --- a/Lib/random.py +++ b/Lib/random.py @@ -492,7 +492,14 @@ def choices(self, population, weights=None, *, cum_weights=None, k=1): ## -------------------- real-valued distributions ------------------- def uniform(self, a, b): - "Get a random number in the range [a, b) or [a, b] depending on rounding." + """Get a random number in the range [a, b) or [a, b] depending on rounding. + + The mean (expected value) and variance of the random variable are: + + E[X] = (a + b) / 2 + Var[X] = (b - a) ** 2 / 12 + + """ return a + (b - a) * self.random() def triangular(self, low=0.0, high=1.0, mode=None): @@ -503,6 +510,11 @@ def triangular(self, low=0.0, high=1.0, mode=None): http://en.wikipedia.org/wiki/Triangular_distribution + The mean (expected value) and variance of the random variable are: + + E[X] = (low + high + mode) / 3 + Var[X] = (low**2 + high**2 + mode**2 - low*high - low*mode - high*mode) / 18 + """ u = self.random() try: @@ -593,12 +605,15 @@ def expovariate(self, lambd=1.0): positive infinity if lambd is positive, and from negative infinity to 0 if lambd is negative. - """ - # lambd: rate lambd = 1/mean - # ('lambda' is a Python reserved word) + The mean (expected value) and variance of the random variable are: + + E[X] = 1 / lambd + Var[X] = 1 / lambd ** 2 + """ # we use 1-random() instead of random() to preclude the # possibility of taking the log of zero. + return -_log(1.0 - self.random()) / lambd def vonmisesvariate(self, mu, kappa): @@ -654,8 +669,12 @@ def gammavariate(self, alpha, beta): pdf(x) = -------------------------------------- math.gamma(alpha) * beta ** alpha + The mean (expected value) and variance of the random variable are: + + E[X] = alpha * beta + Var[X] = alpha * beta ** 2 + """ - # alpha > 0, beta > 0, mean is alpha*beta, variance is alpha*beta**2 # Warning: a few older sources define the gamma distribution in terms # of alpha > -1.0 @@ -714,6 +733,11 @@ def betavariate(self, alpha, beta): Conditions on the parameters are alpha > 0 and beta > 0. Returned values range between 0 and 1. + The mean (expected value) and variance of the random variable are: + + E[X] = alpha / (alpha + beta) + Var[X] = alpha * beta / ((alpha + beta)**2 * (alpha + beta + 1)) + """ ## See ## http://mail.python.org/pipermail/python-bugs-list/2001-January/003752.html @@ -766,6 +790,11 @@ def binomialvariate(self, n=1, p=0.5): Returns an integer in the range: 0 <= X <= n + The mean (expected value) and variance of the random variable are: + + E[X] = n * p + Var[x] = n * p * (1 - p) + """ # Error check inputs and handle edge cases if n < 0: diff --git a/Lib/test/__main__.py b/Lib/test/__main__.py index e5780b784b4b05..82b50ad2c6e777 100644 --- a/Lib/test/__main__.py +++ b/Lib/test/__main__.py @@ -1,2 +1,2 @@ from test.libregrtest.main import main -main() +main(_add_python_opts=True) diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 730b887dd4bcac..756d6808518fc4 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -79,7 +79,7 @@ if support.check_sanitizer(address=True): - # bpo-45200: Skip multiprocessing tests if Python is built with ASAN to + # gh-89363: Skip multiprocessing tests if Python is built with ASAN to # work around a libasan race condition: dead lock in pthread_create(). raise unittest.SkipTest("libasan has a pthread_create() dead lock") diff --git a/Lib/test/libregrtest/cmdline.py b/Lib/test/libregrtest/cmdline.py index 99f28152f1a1c7..c180bb76222a89 100644 --- a/Lib/test/libregrtest/cmdline.py +++ b/Lib/test/libregrtest/cmdline.py @@ -4,6 +4,8 @@ import sys from test.support import os_helper +from .utils import MS_WINDOWS + USAGE = """\ python -m test [options] [test_name1 [test_name2 ...]] @@ -145,6 +147,7 @@ class Namespace(argparse.Namespace): def __init__(self, **kwargs) -> None: + self.ci = False self.testdir = None self.verbose = 0 self.quiet = False @@ -181,6 +184,7 @@ def __init__(self, **kwargs) -> None: self.threshold = None self.fail_rerun = False self.tempdir = None + self._add_python_opts = True super().__init__(**kwargs) @@ -209,7 +213,13 @@ def _create_parser(): # We add help explicitly to control what argument group it renders under. group.add_argument('-h', '--help', action='help', help='show this help message and exit') - group.add_argument('--timeout', metavar='TIMEOUT', type=float, + group.add_argument('--fast-ci', action='store_true', + help='Fast Continuous Integration (CI) mode used by ' + 'GitHub Actions') + group.add_argument('--slow-ci', action='store_true', + help='Slow Continuous Integration (CI) mode used by ' + 'buildbot workers') + group.add_argument('--timeout', metavar='TIMEOUT', help='dump the traceback and exit if a test takes ' 'more than TIMEOUT seconds; disabled if TIMEOUT ' 'is negative or equals to zero') @@ -334,6 +344,9 @@ def _create_parser(): help='override the working directory for the test run') group.add_argument('--cleanup', action='store_true', help='remove old test_python_* directories') + group.add_argument('--dont-add-python-opts', dest='_add_python_opts', + action='store_false', + help="internal option, don't use it") return parser @@ -384,7 +397,51 @@ def _parse_args(args, **kwargs): for arg in ns.args: if arg.startswith('-'): parser.error("unrecognized arguments: %s" % arg) - sys.exit(1) + + if ns.timeout is not None: + # Support "--timeout=" (no value) so Makefile.pre.pre TESTTIMEOUT + # can be used by "make buildbottest" and "make test". + if ns.timeout != "": + try: + ns.timeout = float(ns.timeout) + except ValueError: + parser.error(f"invalid timeout value: {ns.timeout!r}") + else: + ns.timeout = None + + # Continuous Integration (CI): common options for fast/slow CI modes + if ns.slow_ci or ns.fast_ci: + # Similar to options: + # + # -j0 --randomize --fail-env-changed --fail-rerun --rerun + # --slowest --verbose3 --nowindows + if ns.use_mp is None: + ns.use_mp = 0 + ns.randomize = True + ns.fail_env_changed = True + ns.fail_rerun = True + ns.rerun = True + ns.print_slow = True + ns.verbose3 = True + if MS_WINDOWS: + ns.nowindows = True # Silence alerts under Windows + else: + ns._add_python_opts = False + + # When both --slow-ci and --fast-ci options are present, + # --slow-ci has the priority + if ns.slow_ci: + # Similar to: -u "all" --timeout=1200 + if not ns.use: + ns.use = [['all']] + if ns.timeout is None: + ns.timeout = 1200 # 20 minutes + elif ns.fast_ci: + # Similar to: -u "all,-cpu" --timeout=600 + if not ns.use: + ns.use = [['all', '-cpu']] + if ns.timeout is None: + ns.timeout = 600 # 10 minutes if ns.single and ns.fromfile: parser.error("-s and -f don't go together!") diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index a9dd08702deb59..c31d5ff187c56a 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -1,6 +1,7 @@ import os import random import re +import shlex import sys import time @@ -20,7 +21,8 @@ StrPath, StrJSON, TestName, TestList, TestTuple, FilterTuple, strip_py_suffix, count, format_duration, printlist, get_temp_dir, get_work_dir, exit_timeout, - display_header, cleanup_temp_dir) + display_header, cleanup_temp_dir, print_warning, + MS_WINDOWS) class Regrtest: @@ -46,7 +48,7 @@ class Regrtest: directly to set the values that would normally be set by flags on the command line. """ - def __init__(self, ns: Namespace): + def __init__(self, ns: Namespace, _add_python_opts: bool = False): # Log verbosity self.verbose: int = int(ns.verbose) self.quiet: bool = ns.quiet @@ -69,6 +71,11 @@ def __init__(self, ns: Namespace): self.want_rerun: bool = ns.rerun self.want_run_leaks: bool = ns.runleaks + ci_mode = (ns.fast_ci or ns.slow_ci) + self.want_add_python_opts: bool = (_add_python_opts + and ns._add_python_opts + and ci_mode) + # Select tests if ns.match_tests: self.match_tests: FilterTuple | None = tuple(ns.match_tests) @@ -424,7 +431,7 @@ def _run_tests(self, selected: TestTuple, tests: TestList | None) -> int: if (self.want_header or not(self.pgo or self.quiet or self.single_test_run or tests or self.cmdline_args)): - display_header() + display_header(self.use_resources) if self.randomize: print("Using random seed", self.random_seed) @@ -435,7 +442,15 @@ def _run_tests(self, selected: TestTuple, tests: TestList | None) -> int: setup_process() - self.logger.start_load_tracker() + if self.hunt_refleak and not self.num_workers: + # gh-109739: WindowsLoadTracker thread interfers with refleak check + use_load_tracker = False + else: + # WindowsLoadTracker is only needed on Windows + use_load_tracker = MS_WINDOWS + + if use_load_tracker: + self.logger.start_load_tracker() try: if self.num_workers: self._run_tests_mp(runtests, self.num_workers) @@ -448,7 +463,8 @@ def _run_tests(self, selected: TestTuple, tests: TestList | None) -> int: if self.want_rerun and self.results.need_rerun(): self.rerun_failed_tests(runtests) finally: - self.logger.stop_load_tracker() + if use_load_tracker: + self.logger.stop_load_tracker() self.display_summary() self.finalize_tests(tracer) @@ -473,7 +489,55 @@ def run_tests(self, selected: TestTuple, tests: TestList | None) -> int: # processes. return self._run_tests(selected, tests) - def main(self, tests: TestList | None = None): + def _add_python_opts(self): + python_opts = [] + + # Unbuffered stdout and stderr + if not sys.stdout.write_through: + python_opts.append('-u') + + # Add warnings filter 'default' + if 'default' not in sys.warnoptions: + python_opts.extend(('-W', 'default')) + + # Error on bytes/str comparison + if sys.flags.bytes_warning < 2: + python_opts.append('-bb') + + # Ignore PYTHON* environment variables + if not sys.flags.ignore_environment: + python_opts.append('-E') + + if not python_opts: + return + + cmd = [*sys.orig_argv, "--dont-add-python-opts"] + cmd[1:1] = python_opts + + # Make sure that messages before execv() are logged + sys.stdout.flush() + sys.stderr.flush() + + cmd_text = shlex.join(cmd) + try: + if hasattr(os, 'execv') and not MS_WINDOWS: + os.execv(cmd[0], cmd) + # execv() do no return and so we don't get to this line on success + else: + import subprocess + proc = subprocess.run(cmd) + sys.exit(proc.returncode) + except Exception as exc: + print_warning(f"Failed to change Python options: {exc!r}\n" + f"Command: {cmd_text}") + # continue executing main() + + def _init(self): + # Set sys.stdout encoder error handler to backslashreplace, + # similar to sys.stderr error handler, to avoid UnicodeEncodeError + # when printing a traceback or any other non-encodable character. + sys.stdout.reconfigure(errors="backslashreplace") + if self.junit_filename and not os.path.isabs(self.junit_filename): self.junit_filename = os.path.abspath(self.junit_filename) @@ -481,6 +545,12 @@ def main(self, tests: TestList | None = None): self.tmp_dir = get_temp_dir(self.tmp_dir) + def main(self, tests: TestList | None = None): + if self.want_add_python_opts: + self._add_python_opts() + + self._init() + if self.want_cleanup: cleanup_temp_dir(self.tmp_dir) sys.exit(0) @@ -505,7 +575,7 @@ def main(self, tests: TestList | None = None): sys.exit(exitcode) -def main(tests=None, **kwargs): +def main(tests=None, _add_python_opts=False, **kwargs): """Run the Python suite.""" ns = _parse_args(sys.argv[1:], **kwargs) - Regrtest(ns).main(tests=tests) + Regrtest(ns, _add_python_opts=_add_python_opts).main(tests=tests) diff --git a/Lib/test/libregrtest/results.py b/Lib/test/libregrtest/results.py index 6e7d65880f7347..35df50d581ff6a 100644 --- a/Lib/test/libregrtest/results.py +++ b/Lib/test/libregrtest/results.py @@ -8,11 +8,13 @@ printlist, count, format_duration) +# Python uses exit code 1 when an exception is not catched +# argparse.ArgumentParser.error() uses exit code 2 EXITCODE_BAD_TEST = 2 EXITCODE_ENV_CHANGED = 3 EXITCODE_NO_TESTS_RAN = 4 EXITCODE_RERUN_FAIL = 5 -EXITCODE_INTERRUPTED = 130 +EXITCODE_INTERRUPTED = 130 # 128 + signal.SIGINT=2 class TestResults: @@ -25,7 +27,7 @@ def __init__(self): self.env_changed: TestList = [] self.run_no_tests: TestList = [] self.rerun: TestList = [] - self.bad_results: list[TestResult] = [] + self.rerun_results: list[TestResult] = [] self.interrupted: bool = False self.test_times: list[tuple[float, TestName]] = [] @@ -33,6 +35,11 @@ def __init__(self): # used by --junit-xml self.testsuite_xml: list[str] = [] + def is_all_good(self): + return (not self.bad + and not self.skipped + and not self.interrupted) + def get_executed(self): return (set(self.good) | set(self.bad) | set(self.skipped) | set(self.resource_denied) | set(self.env_changed) @@ -82,6 +89,7 @@ def accumulate_result(self, result: TestResult, runtests: RunTests): self.good.append(test_name) case State.ENV_CHANGED: self.env_changed.append(test_name) + self.rerun_results.append(result) case State.SKIPPED: self.skipped.append(test_name) case State.RESOURCE_DENIED: @@ -93,7 +101,7 @@ def accumulate_result(self, result: TestResult, runtests: RunTests): case _: if result.is_failed(fail_env_changed): self.bad.append(test_name) - self.bad_results.append(result) + self.rerun_results.append(result) else: raise ValueError(f"invalid test state: {result.state!r}") @@ -109,12 +117,12 @@ def accumulate_result(self, result: TestResult, runtests: RunTests): self.add_junit(xml_data) def need_rerun(self): - return bool(self.bad_results) + return bool(self.rerun_results) def prepare_rerun(self) -> tuple[TestTuple, FilterDict]: tests: TestList = [] match_tests_dict = {} - for result in self.bad_results: + for result in self.rerun_results: tests.append(result.test_name) match_tests = result.get_rerun_match_tests() @@ -125,7 +133,8 @@ def prepare_rerun(self) -> tuple[TestTuple, FilterDict]: # Clear previously failed tests self.rerun_bad.extend(self.bad) self.bad.clear() - self.bad_results.clear() + self.env_changed.clear() + self.rerun_results.clear() return (tuple(tests), match_tests_dict) @@ -164,24 +173,12 @@ def write_junit(self, filename: StrPath): f.write(s) def display_result(self, tests: TestTuple, quiet: bool, print_slowest: bool): - if self.interrupted: - print("Test suite interrupted by signal SIGINT.") - omitted = set(tests) - self.get_executed() if omitted: print() print(count(len(omitted), "test"), "omitted:") printlist(omitted) - if self.good and not quiet: - print() - if (not self.bad - and not self.skipped - and not self.interrupted - and len(self.good) > 1): - print("All", end=' ') - print(count(len(self.good), "test"), "OK.") - if print_slowest: self.test_times.sort(reverse=True) print() @@ -189,36 +186,34 @@ def display_result(self, tests: TestTuple, quiet: bool, print_slowest: bool): for test_time, test in self.test_times[:10]: print("- %s: %s" % (test, format_duration(test_time))) - if self.bad: - print() - print(count(len(self.bad), "test"), "failed:") - printlist(self.bad) - - if self.env_changed: - print() - print("{} altered the execution environment:".format( - count(len(self.env_changed), "test"))) - printlist(self.env_changed) - - if self.skipped and not quiet: - print() - print(count(len(self.skipped), "test"), "skipped:") - printlist(self.skipped) - - if self.resource_denied and not quiet: - print() - print(count(len(self.resource_denied), "test"), "skipped (resource denied):") - printlist(self.resource_denied) + all_tests = [ + (self.bad, "test", "{} failed:"), + (self.env_changed, "test", "{} altered the execution environment (env changed):"), + ] + if not quiet: + all_tests.append((self.skipped, "test", "{} skipped:")) + all_tests.append((self.resource_denied, "test", "{} skipped (resource denied):")) + all_tests.append((self.rerun, "re-run test", "{}:")) + all_tests.append((self.run_no_tests, "test", "{} run no tests:")) + + for tests_list, count_text, title_format in all_tests: + if tests_list: + print() + count_text = count(len(tests_list), count_text) + print(title_format.format(count_text)) + printlist(tests_list) - if self.rerun: + if self.good and not quiet: print() - print("%s:" % count(len(self.rerun), "re-run test")) - printlist(self.rerun) + text = count(len(self.good), "test") + text = f"{text} OK." + if (self.is_all_good() and len(self.good) > 1): + text = f"All {text}" + print(text) - if self.run_no_tests: + if self.interrupted: print() - print(count(len(self.run_no_tests), "test"), "run no tests:") - printlist(self.run_no_tests) + print("Test suite interrupted by signal SIGINT.") def display_summary(self, first_runtests: RunTests, filtered: bool): # Total tests diff --git a/Lib/test/libregrtest/setup.py b/Lib/test/libregrtest/setup.py index 204f10fe839792..f0d8d7ebaa2fdb 100644 --- a/Lib/test/libregrtest/setup.py +++ b/Lib/test/libregrtest/setup.py @@ -11,7 +11,7 @@ from .runtests import RunTests from .utils import ( setup_unraisable_hook, setup_threading_excepthook, fix_umask, - replace_stdout, adjust_rlimit_nofile) + adjust_rlimit_nofile) UNICODE_GUARD_ENV = "PYTHONREGRTEST_UNICODE_GUARD" @@ -49,7 +49,7 @@ def setup_process(): faulthandler.register(signum, chain=True, file=stderr_fd) adjust_rlimit_nofile() - replace_stdout() + support.record_original_stdout(sys.stdout) # Some times __path__ and __file__ are not absolute (e.g. while running from diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py index 6af949cea9c926..acf35723a1abb0 100644 --- a/Lib/test/libregrtest/utils.py +++ b/Lib/test/libregrtest/utils.py @@ -1,4 +1,3 @@ -import atexit import contextlib import faulthandler import locale @@ -495,32 +494,6 @@ def normalize_test_name(test_full_name, *, is_error=False): return short_name -def replace_stdout(): - """Set stdout encoder error handler to backslashreplace (as stderr error - handler) to avoid UnicodeEncodeError when printing a traceback""" - stdout = sys.stdout - try: - fd = stdout.fileno() - except ValueError: - # On IDLE, sys.stdout has no file descriptor and is not a TextIOWrapper - # object. Leaving sys.stdout unchanged. - # - # Catch ValueError to catch io.UnsupportedOperation on TextIOBase - # and ValueError on a closed stream. - return - - sys.stdout = open(fd, 'w', - encoding=stdout.encoding, - errors="backslashreplace", - closefd=False, - newline='\n') - - def restore_stdout(): - sys.stdout.close() - sys.stdout = stdout - atexit.register(restore_stdout) - - def adjust_rlimit_nofile(): """ On macOS the default fd limit (RLIMIT_NOFILE) is sometimes too low (256) @@ -547,21 +520,13 @@ def adjust_rlimit_nofile(): f"{new_fd_limit}: {err}.") -def display_header(): - encoding = sys.stdout.encoding - +def display_header(use_resources: tuple[str, ...]): # Print basic platform information print("==", platform.python_implementation(), *sys.version.split()) print("==", platform.platform(aliased=True), "%s-endian" % sys.byteorder) print("== Python build:", ' '.join(get_build_info())) - - cwd = os.getcwd() - # gh-109508: support.os_helper.FS_NONASCII, used by get_work_dir(), cannot - # be encoded to the filesystem encoding on purpose, escape non-encodable - # characters with backslashreplace error handler. - formatted_cwd = cwd.encode(encoding, "backslashreplace").decode(encoding) - print("== cwd:", formatted_cwd) + print("== cwd:", os.getcwd()) cpu_count = os.cpu_count() if cpu_count: @@ -569,6 +534,13 @@ def display_header(): print("== encodings: locale=%s, FS=%s" % (locale.getencoding(), sys.getfilesystemencoding())) + + if use_resources: + print(f"== resources ({len(use_resources)}): " + f"{', '.join(sorted(use_resources))}") + else: + print(f"== resources: (all disabled, use -u option)") + # This makes it easier to remember what to set in your local # environment when trying to reproduce a sanitizer failure. asan = support.check_sanitizer(address=True) @@ -581,18 +553,18 @@ def display_header(): sanitizers.append("memory") if ubsan: sanitizers.append("undefined behavior") - if not sanitizers: - return - - print(f"== sanitizers: {', '.join(sanitizers)}") - for sanitizer, env_var in ( - (asan, "ASAN_OPTIONS"), - (msan, "MSAN_OPTIONS"), - (ubsan, "UBSAN_OPTIONS"), - ): - options= os.environ.get(env_var) - if sanitizer and options is not None: - print(f"== {env_var}={options!r}") + if sanitizers: + print(f"== sanitizers: {', '.join(sanitizers)}") + for sanitizer, env_var in ( + (asan, "ASAN_OPTIONS"), + (msan, "MSAN_OPTIONS"), + (ubsan, "UBSAN_OPTIONS"), + ): + options= os.environ.get(env_var) + if sanitizer and options is not None: + print(f"== {env_var}={options!r}") + + print(flush=True) def cleanup_temp_dir(tmp_dir: StrPath): diff --git a/Lib/test/lock_tests.py b/Lib/test/lock_tests.py index a4f52cb20ad301..e53e24b18f2760 100644 --- a/Lib/test/lock_tests.py +++ b/Lib/test/lock_tests.py @@ -330,6 +330,42 @@ def test_release_save_unacquired(self): lock.release() self.assertRaises(RuntimeError, lock._release_save) + def test_recursion_count(self): + lock = self.locktype() + self.assertEqual(0, lock._recursion_count()) + lock.acquire() + self.assertEqual(1, lock._recursion_count()) + lock.acquire() + lock.acquire() + self.assertEqual(3, lock._recursion_count()) + lock.release() + self.assertEqual(2, lock._recursion_count()) + lock.release() + lock.release() + self.assertEqual(0, lock._recursion_count()) + + phase = [] + + def f(): + lock.acquire() + phase.append(None) + while len(phase) == 1: + _wait() + lock.release() + phase.append(None) + + with threading_helper.wait_threads_exit(): + start_new_thread(f, ()) + while len(phase) == 0: + _wait() + self.assertEqual(len(phase), 1) + self.assertEqual(0, lock._recursion_count()) + phase.append(None) + while len(phase) == 2: + _wait() + self.assertEqual(len(phase), 3) + self.assertEqual(0, lock._recursion_count()) + def test_different_thread(self): # Cannot release from a different thread lock = self.locktype() @@ -1014,13 +1050,15 @@ def test_default_timeout(self): """ Test the barrier's default timeout """ - # create a barrier with a low default timeout - barrier = self.barriertype(self.N, timeout=0.3) + # gh-109401: Barrier timeout should be long enough + # to create 4 threads on a slow CI. + timeout = 1.0 + barrier = self.barriertype(self.N, timeout=timeout) def f(): i = barrier.wait() if i == self.N // 2: - # One thread is later than the default timeout of 0.3s. - time.sleep(1.0) + # One thread is later than the default timeout. + time.sleep(timeout * 2) self.assertRaises(threading.BrokenBarrierError, barrier.wait) self.run_threads(f) diff --git a/Lib/test/mapping_tests.py b/Lib/test/mapping_tests.py index b3e4192e65d957..b4cfce19a7174e 100644 --- a/Lib/test/mapping_tests.py +++ b/Lib/test/mapping_tests.py @@ -1,7 +1,6 @@ # tests common to dict and UserDict import unittest import collections -import sys from test.support import Py_C_RECURSION_LIMIT diff --git a/Lib/test/support/interpreters.py b/Lib/test/support/interpreters.py index 5c484d1170d1d9..eeff3abe0324e5 100644 --- a/Lib/test/support/interpreters.py +++ b/Lib/test/support/interpreters.py @@ -5,7 +5,7 @@ import _xxinterpchannels as _channels # aliases: -from _xxsubinterpreters import is_shareable, RunFailedError +from _xxsubinterpreters import is_shareable from _xxinterpchannels import ( ChannelError, ChannelNotFoundError, ChannelEmptyError, ) diff --git a/Lib/test/test_asyncio/test_eager_task_factory.py b/Lib/test/test_asyncio/test_eager_task_factory.py index fc9ad8eb43bb1b..0f8212dbec47be 100644 --- a/Lib/test/test_asyncio/test_eager_task_factory.py +++ b/Lib/test/test_asyncio/test_eager_task_factory.py @@ -7,7 +7,6 @@ from unittest import mock from asyncio import tasks from test.test_asyncio import utils as test_utils -import test.support from test.support.script_helper import assert_python_ok MOCK_ANY = mock.ANY diff --git a/Lib/test/test_asyncio/test_waitfor.py b/Lib/test/test_asyncio/test_waitfor.py index d5c02ba4a01df9..e714b154c5cadf 100644 --- a/Lib/test/test_asyncio/test_waitfor.py +++ b/Lib/test/test_asyncio/test_waitfor.py @@ -1,6 +1,7 @@ import asyncio import unittest import time +from test import support def tearDownModule(): @@ -130,7 +131,7 @@ async def foo(): nonlocal foo_running foo_running = True try: - await asyncio.sleep(10) + await asyncio.sleep(support.LONG_TIMEOUT) finally: foo_running = False return 'done' @@ -144,7 +145,7 @@ async def foo(): self.assertTrue(fut.done()) # it should have been cancelled due to the timeout self.assertTrue(fut.cancelled()) - self.assertLess(t1 - t0, 0.5) + self.assertLess(t1 - t0, support.SHORT_TIMEOUT) self.assertEqual(foo_running, False) async def test_wait_for_blocking(self): diff --git a/Lib/test/test_capi/test_abstract.py b/Lib/test/test_capi/test_abstract.py index 7fad853ff54fe3..eeaef60a8b47b5 100644 --- a/Lib/test/test_capi/test_abstract.py +++ b/Lib/test/test_capi/test_abstract.py @@ -1,8 +1,5 @@ import unittest -import sys from collections import OrderedDict -from test import support -from test.support import import_helper import _testcapi diff --git a/Lib/test/test_capi/test_dict.py b/Lib/test/test_capi/test_dict.py index b22fa20e14dfea..11b2ca910707df 100644 --- a/Lib/test/test_capi/test_dict.py +++ b/Lib/test/test_capi/test_dict.py @@ -1,9 +1,6 @@ import unittest -import sys from collections import OrderedDict, UserDict from types import MappingProxyType -from test import support -from test.support import import_helper import _testcapi diff --git a/Lib/test/test_capi/test_misc.py b/Lib/test/test_capi/test_misc.py index cf1cba3e693ef6..5ece213e7b2363 100644 --- a/Lib/test/test_capi/test_misc.py +++ b/Lib/test/test_capi/test_misc.py @@ -2,7 +2,7 @@ # these are all functions _testcapi exports whose name begins with 'test_'. import _thread -from collections import OrderedDict, deque +from collections import deque import contextlib import importlib.machinery import importlib.util diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py index d3a5517963c540..53e3e8f75aa766 100644 --- a/Lib/test/test_compile.py +++ b/Lib/test/test_compile.py @@ -1272,6 +1272,11 @@ def f(): else: 1 if 1 else 1 + def test_remove_empty_basic_block_with_jump_target_label(self): + # See gh-109823 + def f(x): + while x: + 0 if 1 else 0 @requires_debug_ranges() class TestSourcePositions(unittest.TestCase): diff --git a/Lib/test/test_concurrent_futures/test_deadlock.py b/Lib/test/test_concurrent_futures/test_deadlock.py index a76e075c3be180..af702542081ad9 100644 --- a/Lib/test/test_concurrent_futures/test_deadlock.py +++ b/Lib/test/test_concurrent_futures/test_deadlock.py @@ -145,6 +145,9 @@ def test_exit_at_task_unpickle(self): self._check_crash(BrokenProcessPool, id, ExitAtUnpickle()) def test_error_at_task_unpickle(self): + # gh-109832: Restore stderr overriden by _raise_error_ignore_stderr() + self.addCleanup(setattr, sys, 'stderr', sys.stderr) + # Check problem occurring while unpickling a task on workers self._check_crash(BrokenProcessPool, id, ErrorAtUnpickle()) @@ -180,6 +183,9 @@ def test_error_during_result_pickle_on_worker(self): self._check_crash(PicklingError, _return_instance, ErrorAtPickle) def test_error_during_result_unpickle_in_result_handler(self): + # gh-109832: Restore stderr overriden by _raise_error_ignore_stderr() + self.addCleanup(setattr, sys, 'stderr', sys.stderr) + # Check problem occurring while unpickling a task in # the result_handler thread self._check_crash(BrokenProcessPool, diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index 27f4978ca66a88..97b9bba24bcbca 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -10,7 +10,7 @@ import gc import pickle from test import support -from test.support import warnings_helper, import_helper, check_disallow_instantiation +from test.support import import_helper, check_disallow_instantiation from itertools import permutations from textwrap import dedent from collections import OrderedDict diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index d806eeac25fb2f..bd299483e7b0bd 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -41,7 +41,6 @@ darwin_malloc_err_warning, is_emscripten) from test.support.import_helper import import_fresh_module from test.support import threading_helper -from test.support import warnings_helper import random import inspect import threading diff --git a/Lib/test/test_dictviews.py b/Lib/test/test_dictviews.py index 34918585513846..cad568b6ac4c2d 100644 --- a/Lib/test/test_dictviews.py +++ b/Lib/test/test_dictviews.py @@ -1,7 +1,6 @@ import collections.abc import copy import pickle -import sys import unittest from test.support import Py_C_RECURSION_LIMIT diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py index bebd1bbb9e2703..2f191ea7a44c16 100644 --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -32,7 +32,7 @@ DEFAULT_ENCODING = 'utf-8' # the dummy data returned by server over the data channel when # RETR, LIST, NLST, MLSD commands are issued -RETR_DATA = 'abcde12345\r\n' * 1000 + 'non-ascii char \xAE\r\n' +RETR_DATA = 'abcde\xB9\xB2\xB3\xA4\xA6\r\n' * 1000 LIST_DATA = 'foo\r\nbar\r\n non-ascii char \xAE\r\n' NLST_DATA = 'foo\r\nbar\r\n non-ascii char \xAE\r\n' MLSD_DATA = ("type=cdir;perm=el;unique==keVO1+ZF4; test\r\n" @@ -67,11 +67,11 @@ class DummyDTPHandler(asynchat.async_chat): def __init__(self, conn, baseclass): asynchat.async_chat.__init__(self, conn) self.baseclass = baseclass - self.baseclass.last_received_data = '' + self.baseclass.last_received_data = bytearray() self.encoding = baseclass.encoding def handle_read(self): - new_data = self.recv(1024).decode(self.encoding, 'replace') + new_data = self.recv(1024) self.baseclass.last_received_data += new_data def handle_close(self): @@ -107,7 +107,7 @@ def __init__(self, conn, encoding=DEFAULT_ENCODING): self.in_buffer = [] self.dtp = None self.last_received_cmd = None - self.last_received_data = '' + self.last_received_data = bytearray() self.next_response = '' self.next_data = None self.rest = None @@ -590,19 +590,17 @@ def test_abort(self): self.client.abort() def test_retrbinary(self): - def callback(data): - received.append(data.decode(self.client.encoding)) received = [] - self.client.retrbinary('retr', callback) - self.check_data(''.join(received), RETR_DATA) + self.client.retrbinary('retr', received.append) + self.check_data(b''.join(received), + RETR_DATA.encode(self.client.encoding)) def test_retrbinary_rest(self): - def callback(data): - received.append(data.decode(self.client.encoding)) for rest in (0, 10, 20): received = [] - self.client.retrbinary('retr', callback, rest=rest) - self.check_data(''.join(received), RETR_DATA[rest:]) + self.client.retrbinary('retr', received.append, rest=rest) + self.check_data(b''.join(received), + RETR_DATA[rest:].encode(self.client.encoding)) def test_retrlines(self): received = [] @@ -612,7 +610,8 @@ def test_retrlines(self): def test_storbinary(self): f = io.BytesIO(RETR_DATA.encode(self.client.encoding)) self.client.storbinary('stor', f) - self.check_data(self.server.handler_instance.last_received_data, RETR_DATA) + self.check_data(self.server.handler_instance.last_received_data, + RETR_DATA.encode(self.server.encoding)) # test new callback arg flag = [] f.seek(0) @@ -631,7 +630,8 @@ def test_storlines(self): data = RETR_DATA.replace('\r\n', '\n').encode(self.client.encoding) f = io.BytesIO(data) self.client.storlines('stor', f) - self.check_data(self.server.handler_instance.last_received_data, RETR_DATA) + self.check_data(self.server.handler_instance.last_received_data, + RETR_DATA.encode(self.server.encoding)) # test new callback arg flag = [] f.seek(0) @@ -649,7 +649,7 @@ def test_nlst(self): def test_dir(self): l = [] - self.client.dir(lambda x: l.append(x)) + self.client.dir(l.append) self.assertEqual(''.join(l), LIST_DATA.replace('\r\n', '')) def test_mlsd(self): @@ -889,12 +889,10 @@ def test_makepasv(self): def test_transfer(self): def retr(): - def callback(data): - received.append(data.decode(self.client.encoding)) received = [] - self.client.retrbinary('retr', callback) - self.assertEqual(len(''.join(received)), len(RETR_DATA)) - self.assertEqual(''.join(received), RETR_DATA) + self.client.retrbinary('retr', received.append) + self.assertEqual(b''.join(received), + RETR_DATA.encode(self.client.encoding)) self.client.set_pasv(True) retr() self.client.set_pasv(False) diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py index 914ef942feab12..5a4394a0993c8d 100644 --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -4,7 +4,6 @@ # Lib/test/test_jit_gdb.py import os -import platform import re import subprocess import sys diff --git a/Lib/test/test_importlib/import_/test_packages.py b/Lib/test/test_importlib/import_/test_packages.py index eb0831f7d6d54b..0c29d6083265fa 100644 --- a/Lib/test/test_importlib/import_/test_packages.py +++ b/Lib/test/test_importlib/import_/test_packages.py @@ -1,7 +1,6 @@ from test.test_importlib import util import sys import unittest -from test import support from test.support import import_helper diff --git a/Lib/test/test_importlib/test_locks.py b/Lib/test/test_importlib/test_locks.py index ba9cf51c261d52..7091c36aaaf761 100644 --- a/Lib/test/test_importlib/test_locks.py +++ b/Lib/test/test_importlib/test_locks.py @@ -29,6 +29,8 @@ class ModuleLockAsRLockTests: test_timeout = None # _release_save() unsupported test_release_save_unacquired = None + # _recursion_count() unsupported + test_recursion_count = None # lock status in repr unsupported test_repr = None test_locked_repr = None diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py index bab868600895c1..92c99d645b25cc 100644 --- a/Lib/test/test_mmap.py +++ b/Lib/test/test_mmap.py @@ -255,10 +255,15 @@ def test_access_parameter(self): # Try writing with PROT_EXEC and without PROT_WRITE prot = mmap.PROT_READ | getattr(mmap, 'PROT_EXEC', 0) with open(TESTFN, "r+b") as f: - m = mmap.mmap(f.fileno(), mapsize, prot=prot) - self.assertRaises(TypeError, m.write, b"abcdef") - self.assertRaises(TypeError, m.write_byte, 0) - m.close() + try: + m = mmap.mmap(f.fileno(), mapsize, prot=prot) + except PermissionError: + # on macOS 14, PROT_READ | PROT_WRITE is not allowed + pass + else: + self.assertRaises(TypeError, m.write, b"abcdef") + self.assertRaises(TypeError, m.write_byte, 0) + m.close() def test_bad_file_desc(self): # Try opening a bad file descriptor... diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index 09df3fe471fc3e..484a5e6c3bd64d 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -3178,10 +3178,11 @@ def test_absolute(self): self.assertEqual(str(P('//a').absolute()), '//a') self.assertEqual(str(P('//a/b').absolute()), '//a/b') - def _check_symlink_loop(self, *args, strict=True): + def _check_symlink_loop(self, *args): path = self.cls(*args) - with self.assertRaises(RuntimeError): - print(path.resolve(strict)) + with self.assertRaises(OSError) as cm: + path.resolve(strict=True) + self.assertEqual(cm.exception.errno, errno.ELOOP) @unittest.skipIf( is_emscripten or is_wasi, @@ -3240,7 +3241,8 @@ def test_resolve_loop(self): os.symlink('linkZ/../linkZ', join('linkZ')) self._check_symlink_loop(BASE, 'linkZ') # Non-strict - self._check_symlink_loop(BASE, 'linkZ', 'foo', strict=False) + p = self.cls(BASE, 'linkZ', 'foo') + self.assertEqual(p.resolve(strict=False), p) # Loops with absolute symlinks. os.symlink(join('linkU/inside'), join('linkU')) self._check_symlink_loop(BASE, 'linkU') @@ -3249,7 +3251,8 @@ def test_resolve_loop(self): os.symlink(join('linkW/../linkW'), join('linkW')) self._check_symlink_loop(BASE, 'linkW') # Non-strict - self._check_symlink_loop(BASE, 'linkW', 'foo', strict=False) + q = self.cls(BASE, 'linkW', 'foo') + self.assertEqual(q.resolve(strict=False), q) def test_glob(self): P = self.cls diff --git a/Lib/test/test_peg_generator/__init__.py b/Lib/test/test_peg_generator/__init__.py index c23542e254c99f..b32db4426f251d 100644 --- a/Lib/test/test_peg_generator/__init__.py +++ b/Lib/test/test_peg_generator/__init__.py @@ -1,5 +1,4 @@ import os.path -import unittest from test import support from test.support import load_package_tests diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py index 41f4d172f8d057..a542abaf1f35aa 100644 --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -2,7 +2,6 @@ # handler, are obscure and unhelpful. import os -import platform import sys import sysconfig import unittest @@ -14,7 +13,7 @@ from xml.parsers import expat from xml.parsers.expat import errors -from test.support import sortdict, is_emscripten, is_wasi +from test.support import sortdict class SetAttributeTest(unittest.TestCase): diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index 408e667fffa0f0..3ece31be9af3c3 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -23,8 +23,9 @@ from test import support from test.support import os_helper, TestStats, without_optimizer from test.libregrtest import cmdline -from test.libregrtest import utils +from test.libregrtest import main from test.libregrtest import setup +from test.libregrtest import utils from test.libregrtest.utils import normalize_test_name if not support.has_subprocess_support: @@ -75,8 +76,15 @@ def test_help(self): def test_timeout(self): ns = self.parse_args(['--timeout', '4.2']) self.assertEqual(ns.timeout, 4.2) + + # negative, zero and empty string are treated as "no timeout" + for value in ('-1', '0', ''): + with self.subTest(value=value): + ns = self.parse_args([f'--timeout={value}']) + self.assertEqual(ns.timeout, None) + self.checkError(['--timeout'], 'expected one argument') - self.checkError(['--timeout', 'foo'], 'invalid float value') + self.checkError(['--timeout', 'foo'], 'invalid timeout value:') def test_wait(self): ns = self.parse_args(['--wait']) @@ -366,6 +374,49 @@ def test_unknown_option(self): self.checkError(['--unknown-option'], 'unrecognized arguments: --unknown-option') + def check_ci_mode(self, args, use_resources): + ns = cmdline._parse_args(args) + if utils.MS_WINDOWS: + self.assertTrue(ns.nowindows) + + # Check Regrtest attributes which are more reliable than Namespace + # which has an unclear API + regrtest = main.Regrtest(ns) + self.assertEqual(regrtest.num_workers, -1) + self.assertTrue(regrtest.want_rerun) + self.assertTrue(regrtest.randomize) + self.assertIsNone(regrtest.random_seed) + self.assertTrue(regrtest.fail_env_changed) + self.assertTrue(regrtest.fail_rerun) + self.assertTrue(regrtest.print_slowest) + self.assertTrue(regrtest.output_on_failure) + self.assertEqual(sorted(regrtest.use_resources), sorted(use_resources)) + return regrtest + + def test_fast_ci(self): + args = ['--fast-ci'] + use_resources = sorted(cmdline.ALL_RESOURCES) + use_resources.remove('cpu') + regrtest = self.check_ci_mode(args, use_resources) + self.assertEqual(regrtest.timeout, 10 * 60) + + def test_fast_ci_resource(self): + # it should be possible to override resources + args = ['--fast-ci', '-u', 'network'] + use_resources = ['network'] + self.check_ci_mode(args, use_resources) + + def test_slow_ci(self): + args = ['--slow-ci'] + use_resources = sorted(cmdline.ALL_RESOURCES) + regrtest = self.check_ci_mode(args, use_resources) + self.assertEqual(regrtest.timeout, 20 * 60) + + def test_dont_add_python_opts(self): + args = ['--dont-add-python-opts'] + ns = cmdline._parse_args(args) + self.assertFalse(ns._add_python_opts) + @dataclasses.dataclass(slots=True) class Rerun: @@ -463,7 +514,7 @@ def check_executed_tests(self, output, tests, *, stats, randomize = True rerun_failed = [] - if rerun is not None: + if rerun is not None and not env_changed: failed = [rerun.name] if not rerun.success: rerun_failed.append(rerun.name) @@ -500,7 +551,8 @@ def list_regex(line_format, tests): self.check_line(output, regex) if env_changed: - regex = list_regex('%s test%s altered the execution environment', + regex = list_regex(r'%s test%s altered the execution environment ' + r'\(env changed\)', env_changed) self.check_line(output, regex) @@ -590,7 +642,7 @@ def list_regex(line_format, tests): state = ', '.join(state) if rerun is not None: new_state = 'SUCCESS' if rerun.success else 'FAILURE' - state = 'FAILURE then ' + new_state + state = f'{state} then {new_state}' self.check_line(output, f'Result: {state}', full=True) def parse_random_seed(self, output): @@ -1228,6 +1280,15 @@ def test_env_changed(self): self.check_executed_tests(output, [testname], env_changed=testname, fail_env_changed=True, stats=1) + # rerun + output = self.run_tests("--rerun", testname) + self.check_executed_tests(output, [testname], + env_changed=testname, + rerun=Rerun(testname, + match=None, + success=True), + stats=2) + def test_rerun_fail(self): # FAILURE then FAILURE code = textwrap.dedent(""" @@ -1904,6 +1965,61 @@ def test_dev_mode(self): self.check_executed_tests(output, tests, stats=len(tests), parallel=True) + def check_reexec(self, option): + # --fast-ci and --slow-ci add "-u -W default -bb -E" options to Python + code = textwrap.dedent(r""" + import sys + import unittest + try: + from _testinternalcapi import get_config + except ImportError: + get_config = None + + class WorkerTests(unittest.TestCase): + @unittest.skipUnless(get_config is None, 'need get_config()') + def test_config(self): + config = get_config()['config'] + # -u option + self.assertEqual(config['buffered_stdio'], 0) + # -W default option + self.assertTrue(config['warnoptions'], ['default']) + # -bb option + self.assertTrue(config['bytes_warning'], 2) + # -E option + self.assertTrue(config['use_environment'], 0) + + def test_python_opts(self): + # -u option + self.assertTrue(sys.__stdout__.write_through) + self.assertTrue(sys.__stderr__.write_through) + + # -W default option + self.assertTrue(sys.warnoptions, ['default']) + + # -bb option + self.assertEqual(sys.flags.bytes_warning, 2) + + # -E option + self.assertTrue(sys.flags.ignore_environment) + """) + testname = self.create_test(code=code) + + # Use directly subprocess to control the exact command line + cmd = [sys.executable, + "-m", "test", option, + f'--testdir={self.tmptestdir}', + testname] + proc = subprocess.run(cmd, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True) + self.assertEqual(proc.returncode, 0, proc) + + def test_add_python_opts(self): + for opt in ("--fast-ci", "--slow-ci"): + with self.subTest(opt=opt): + self.check_reexec(opt) + class TestUtils(unittest.TestCase): def test_format_duration(self): diff --git a/Lib/test/test_sqlite3/test_dump.py b/Lib/test/test_sqlite3/test_dump.py index 3107e1b165d950..14a18c1ad37102 100644 --- a/Lib/test/test_sqlite3/test_dump.py +++ b/Lib/test/test_sqlite3/test_dump.py @@ -1,7 +1,6 @@ # Author: Paul Kippes import unittest -import sqlite3 as sqlite from .util import memory_database from .util import MemoryDatabaseMixin diff --git a/Lib/test/test_sqlite3/test_userfunctions.py b/Lib/test/test_sqlite3/test_userfunctions.py index 09019498fd5682..c6c3db159add64 100644 --- a/Lib/test/test_sqlite3/test_userfunctions.py +++ b/Lib/test/test_sqlite3/test_userfunctions.py @@ -29,7 +29,7 @@ from test.support import bigmemtest, gc_collect from .util import cx_limit, memory_database -from .util import with_tracebacks, check_tracebacks +from .util import with_tracebacks def func_returntext(): diff --git a/Lib/test/test_sqlite3/util.py b/Lib/test/test_sqlite3/util.py index 505406c437b632..5599823838beea 100644 --- a/Lib/test/test_sqlite3/util.py +++ b/Lib/test/test_sqlite3/util.py @@ -4,7 +4,6 @@ import re import sqlite3 import test.support -import unittest # Helper for temporary memory databases diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index f4948ceec66226..c616a27364b494 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1,5 +1,6 @@ import builtins import codecs +import _datetime import gc import locale import operator @@ -1581,7 +1582,7 @@ def delx(self): del self.__x x = property(getx, setx, delx, "") check(x, size('5Pi')) # PyCapsule - # XXX + check(_datetime.datetime_CAPI, size('6P')) # rangeiterator check(iter(range(1)), size('3l')) check(iter(range(2**65)), size('3P')) diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py index 6465a446565844..71fcad268b8036 100644 --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -572,6 +572,10 @@ def background_thread(evt): self.assertEqual(err, b'') @support.requires_fork() + # gh-89363: Skip multiprocessing tests if Python is built with ASAN to + # work around a libasan race condition: dead lock in pthread_create(). + @support.skip_if_sanitizer("libasan has a pthread_create() dead lock", + address=True) def test_is_alive_after_fork(self): # Try hard to trigger #18418: is_alive() could sometimes be True on # threads that vanished after a fork. @@ -1779,6 +1783,9 @@ class ConditionAsRLockTests(lock_tests.RLockTests): # Condition uses an RLock by default and exports its API. locktype = staticmethod(threading.Condition) + def test_recursion_count(self): + self.skipTest("Condition does not expose _recursion_count()") + class ConditionTests(lock_tests.ConditionTests): condtype = staticmethod(threading.Condition) diff --git a/Lib/test/test_tkinter/support.py b/Lib/test/test_tkinter/support.py index 10e64bf40a4afa..a37705f0ae6feb 100644 --- a/Lib/test/test_tkinter/support.py +++ b/Lib/test/test_tkinter/support.py @@ -1,6 +1,5 @@ import functools import tkinter -import unittest class AbstractTkTest: diff --git a/Lib/test/test_types.py b/Lib/test/test_types.py index c6bff79f903828..da32c4ea6477ce 100644 --- a/Lib/test/test_types.py +++ b/Lib/test/test_types.py @@ -4,6 +4,7 @@ import collections.abc from collections import namedtuple import copy +import _datetime import gc import inspect import pickle @@ -636,6 +637,9 @@ def test_traceback_and_frame_types(self): self.assertIsInstance(exc.__traceback__, types.TracebackType) self.assertIsInstance(exc.__traceback__.tb_frame, types.FrameType) + def test_capsule_type(self): + self.assertIsInstance(_datetime.datetime_CAPI, types.CapsuleType) + class UnionTests(unittest.TestCase): diff --git a/Lib/test/test_unittest/testmock/testthreadingmock.py b/Lib/test/test_unittest/testmock/testthreadingmock.py index 94e71921d9bc03..a02b532ed447cd 100644 --- a/Lib/test/test_unittest/testmock/testthreadingmock.py +++ b/Lib/test/test_unittest/testmock/testthreadingmock.py @@ -3,7 +3,7 @@ import concurrent.futures from test.support import threading_helper -from unittest.mock import patch, ThreadingMock, call +from unittest.mock import patch, ThreadingMock threading_helper.requires_working_threading(module=True) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py index a894bb10bd04da..eb83aa39425515 100644 --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -559,6 +559,13 @@ def test_zippath_from_non_installed_posix(self): platlibdir, stdlib_zip) additional_pythonpath_for_non_installed = [] + + # gh-109748: Don't copy __pycache__/ sub-directories, because they can + # be modified by other Python tests running in parallel. + ignored_names = {'__pycache__'} + def ignore_pycache(src, names): + return ignored_names + # Copy stdlib files to the non-installed python so venv can # correctly calculate the prefix. for eachpath in sys.path: @@ -575,7 +582,8 @@ def test_zippath_from_non_installed_posix(self): if os.path.isfile(fn): shutil.copy(fn, libdir) elif os.path.isdir(fn): - shutil.copytree(fn, os.path.join(libdir, name)) + shutil.copytree(fn, os.path.join(libdir, name), + ignore=ignore_pycache) else: additional_pythonpath_for_non_installed.append( eachpath) diff --git a/Lib/test/test_zipfile/test_core.py b/Lib/test/test_zipfile/test_core.py index 9960259c4cde0c..0f6c0f2107ce6b 100644 --- a/Lib/test/test_zipfile/test_core.py +++ b/Lib/test/test_zipfile/test_core.py @@ -3203,14 +3203,14 @@ def test_no_data(self): b = s.pack(2, 0) c = s.pack(3, 0) - self.assertEqual(b'', zipfile._strip_extra(a, (self.ZIP64_EXTRA,))) - self.assertEqual(b, zipfile._strip_extra(b, (self.ZIP64_EXTRA,))) + self.assertEqual(b'', zipfile._Extra.strip(a, (self.ZIP64_EXTRA,))) + self.assertEqual(b, zipfile._Extra.strip(b, (self.ZIP64_EXTRA,))) self.assertEqual( - b+b"z", zipfile._strip_extra(b+b"z", (self.ZIP64_EXTRA,))) + b+b"z", zipfile._Extra.strip(b+b"z", (self.ZIP64_EXTRA,))) - self.assertEqual(b+c, zipfile._strip_extra(a+b+c, (self.ZIP64_EXTRA,))) - self.assertEqual(b+c, zipfile._strip_extra(b+a+c, (self.ZIP64_EXTRA,))) - self.assertEqual(b+c, zipfile._strip_extra(b+c+a, (self.ZIP64_EXTRA,))) + self.assertEqual(b+c, zipfile._Extra.strip(a+b+c, (self.ZIP64_EXTRA,))) + self.assertEqual(b+c, zipfile._Extra.strip(b+a+c, (self.ZIP64_EXTRA,))) + self.assertEqual(b+c, zipfile._Extra.strip(b+c+a, (self.ZIP64_EXTRA,))) def test_with_data(self): s = struct.Struct("/dev/null 2>&1; then \ - pybuildbot.identify "CC='$(CC)'" "CXX='$(CXX)'"; \ - fi - $(TESTRUNNER) -j 1 -u all -W --slowest --fail-env-changed --fail-rerun --timeout=$(TESTTIMEOUT) $(TESTOPTS) + -@if which pybuildbot.identify >/dev/null 2>&1; then \ + pybuildbot.identify "CC='$(CC)'" "CXX='$(CXX)'"; \ + fi + $(TESTRUNNER) --slow-ci --timeout=$(TESTTIMEOUT) $(TESTOPTS) # Like testall, but run Python tests with HOSTRUNNER directly. .PHONY: hostrunnertest hostrunnertest: all - $(RUNSHARED) $(HOSTRUNNER) ./$(BUILDPYTHON) -m test -u all $(TESTOPTS) + $(RUNSHARED) $(HOSTRUNNER) ./$(BUILDPYTHON) -m test --slow-ci --timeout=$(TESTTIMEOUT) $(TESTOPTS) .PHONY: pythoninfo pythoninfo: all $(RUNSHARED) $(HOSTRUNNER) ./$(BUILDPYTHON) -m test.pythoninfo -QUICKTESTOPTS= $(TESTOPTS) -x test_subprocess test_io \ +QUICKTESTOPTS= -x test_subprocess test_io \ test_multibytecodec test_urllib2_localnet test_itertools \ test_multiprocessing_fork test_multiprocessing_spawn \ test_multiprocessing_forkserver \ @@ -1912,7 +1913,7 @@ QUICKTESTOPTS= $(TESTOPTS) -x test_subprocess test_io \ .PHONY: quicktest quicktest: all - $(TESTRUNNER) $(QUICKTESTOPTS) + $(TESTRUNNER) --fast-ci --timeout=$(TESTTIMEOUT) $(TESTOPTS) $(QUICKTESTOPTS) # SSL tests .PHONY: multisslcompile diff --git a/Misc/NEWS.d/next/Core and Builtins/2023-09-25-14-28-14.gh-issue-109823.kbVTKF.rst b/Misc/NEWS.d/next/Core and Builtins/2023-09-25-14-28-14.gh-issue-109823.kbVTKF.rst new file mode 100644 index 00000000000000..793c89f4445f54 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2023-09-25-14-28-14.gh-issue-109823.kbVTKF.rst @@ -0,0 +1,2 @@ +Fix bug where compiler does not adjust labels when removing an empty basic +block which is a jump target. diff --git a/Misc/NEWS.d/next/Library/2023-02-20-12-00-11.gh-issue-88233.o5Zb0t.rst b/Misc/NEWS.d/next/Library/2023-02-20-12-00-11.gh-issue-88233.o5Zb0t.rst new file mode 100644 index 00000000000000..945f92d3dfa93b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-02-20-12-00-11.gh-issue-88233.o5Zb0t.rst @@ -0,0 +1,2 @@ +Refactored ``zipfile._strip_extra`` to use higher level abstactions for +extras instead of a heavy-state loop. diff --git a/Misc/NEWS.d/next/Library/2023-09-09-17-09-54.gh-issue-109187.dIayNW.rst b/Misc/NEWS.d/next/Library/2023-09-09-17-09-54.gh-issue-109187.dIayNW.rst new file mode 100644 index 00000000000000..31b3ef77807cde --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-09-09-17-09-54.gh-issue-109187.dIayNW.rst @@ -0,0 +1,3 @@ +:meth:`pathlib.Path.resolve` now treats symlink loops like other errors: in +strict mode, :exc:`OSError` is raised, and in non-strict mode, no exception +is raised. diff --git a/Misc/NEWS.d/next/Library/2023-09-20-07-38-14.gh-issue-109599.IaSLJz.rst b/Misc/NEWS.d/next/Library/2023-09-20-07-38-14.gh-issue-109599.IaSLJz.rst new file mode 100644 index 00000000000000..8a15e765545f88 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-09-20-07-38-14.gh-issue-109599.IaSLJz.rst @@ -0,0 +1 @@ +Expose the type of PyCapsule objects as ``types.CapsuleType``. diff --git a/Misc/NEWS.d/next/Library/2023-09-22-20-16-44.gh-issue-109593.LboaNM.rst b/Misc/NEWS.d/next/Library/2023-09-22-20-16-44.gh-issue-109593.LboaNM.rst new file mode 100644 index 00000000000000..292aea0be24dfb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-09-22-20-16-44.gh-issue-109593.LboaNM.rst @@ -0,0 +1 @@ +Avoid deadlocking on a reentrant call to the multiprocessing resource tracker. Such a reentrant call, though unlikely, can happen if a GC pass invokes the finalizer for a multiprocessing object such as SemLock. diff --git a/Misc/NEWS.d/next/Library/2023-09-24-13-28-35.gh-issue-109653.9IFU0B.rst b/Misc/NEWS.d/next/Library/2023-09-24-13-28-35.gh-issue-109653.9IFU0B.rst new file mode 100644 index 00000000000000..c4f5a62433a2c1 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-09-24-13-28-35.gh-issue-109653.9IFU0B.rst @@ -0,0 +1,2 @@ +Improve import time of :mod:`functools` by around 13%. Patch by Alex +Waygood. diff --git a/Misc/NEWS.d/next/Library/2023-09-25-23-00-37.gh-issue-109631.eWSqpO.rst b/Misc/NEWS.d/next/Library/2023-09-25-23-00-37.gh-issue-109631.eWSqpO.rst new file mode 100644 index 00000000000000..58af2e57068267 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-09-25-23-00-37.gh-issue-109631.eWSqpO.rst @@ -0,0 +1,3 @@ +:mod:`re` functions such as :func:`re.findall`, :func:`re.split`, +:func:`re.search` and :func:`re.sub` which perform short repeated matches +can now be interrupted by user. diff --git a/Misc/NEWS.d/next/Tests/2023-09-19-13-33-20.gh-issue-109566.aX0g9o.rst b/Misc/NEWS.d/next/Tests/2023-09-19-13-33-20.gh-issue-109566.aX0g9o.rst new file mode 100644 index 00000000000000..10f90132c37ec9 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2023-09-19-13-33-20.gh-issue-109566.aX0g9o.rst @@ -0,0 +1,4 @@ +regrtest: Add ``--fast-ci`` and ``--slow-ci`` options. ``--fast-ci`` uses a +default timeout of 10 minutes and ``-u all,-cpu`` (skip slowest tests). +``--slow-ci`` uses a default timeout of 20 minues and ``-u all`` (run all +tests). Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2023-09-25-14-41-18.gh-issue-109276.uC_cWo.rst b/Misc/NEWS.d/next/Tests/2023-09-25-14-41-18.gh-issue-109276.uC_cWo.rst new file mode 100644 index 00000000000000..66651cf6f4b966 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2023-09-25-14-41-18.gh-issue-109276.uC_cWo.rst @@ -0,0 +1,3 @@ +regrtest: When a test fails with "env changed" and the --rerun option is +used, the test is now re-run in verbose mode in a fresh process. Patch by +Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2023-09-25-23-59-37.gh-issue-109739.MUn7K5.rst b/Misc/NEWS.d/next/Tests/2023-09-25-23-59-37.gh-issue-109739.MUn7K5.rst new file mode 100644 index 00000000000000..291524c758ca68 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2023-09-25-23-59-37.gh-issue-109739.MUn7K5.rst @@ -0,0 +1,3 @@ +regrtest: Fix reference leak check on Windows. Disable the load tracker on +Windows in the reference leak check mode (-R option). Patch by Victor +Stinner. diff --git a/Misc/NEWS.d/next/Tests/2023-09-26-00-49-18.gh-issue-109748.nxlT1i.rst b/Misc/NEWS.d/next/Tests/2023-09-26-00-49-18.gh-issue-109748.nxlT1i.rst new file mode 100644 index 00000000000000..840366ba8d1611 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2023-09-26-00-49-18.gh-issue-109748.nxlT1i.rst @@ -0,0 +1,3 @@ +Fix ``test_zippath_from_non_installed_posix()`` of test_venv: don't copy +``__pycache__/`` sub-directories, because they can be modified by other Python +tests running in parallel. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2023-09-26-18-12-01.gh-issue-109566.CP0Vhf.rst b/Misc/NEWS.d/next/Tests/2023-09-26-18-12-01.gh-issue-109566.CP0Vhf.rst new file mode 100644 index 00000000000000..d865f629fdb05b --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2023-09-26-18-12-01.gh-issue-109566.CP0Vhf.rst @@ -0,0 +1,3 @@ +regrtest: When ``--fast-ci`` or ``--slow-ci`` option is used, regrtest now +replaces the current process with a new process to add ``-u -W default -bb -E`` +options to Python. Patch by Victor Stinner. diff --git a/Modules/_sre/sre.h b/Modules/_sre/sre.h index f60078d6bb999b..83d89d57b11199 100644 --- a/Modules/_sre/sre.h +++ b/Modules/_sre/sre.h @@ -95,6 +95,7 @@ typedef struct { size_t data_stack_base; /* current repeat context */ SRE_REPEAT *repeat; + unsigned int sigcount; } SRE_STATE; typedef struct { diff --git a/Modules/_sre/sre_lib.h b/Modules/_sre/sre_lib.h index ae80009fd63bbe..3c805aeeca0974 100644 --- a/Modules/_sre/sre_lib.h +++ b/Modules/_sre/sre_lib.h @@ -564,7 +564,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) Py_ssize_t alloc_pos, ctx_pos = -1; Py_ssize_t ret = 0; int jump; - unsigned int sigcount=0; + unsigned int sigcount = state->sigcount; SRE(match_context)* ctx; SRE(match_context)* nextctx; @@ -1567,8 +1567,10 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) ctx_pos = ctx->last_ctx_pos; jump = ctx->jump; DATA_POP_DISCARD(ctx); - if (ctx_pos == -1) + if (ctx_pos == -1) { + state->sigcount = sigcount; return ret; + } DATA_LOOKUP_AT(SRE(match_context), ctx, ctx_pos); switch (jump) { diff --git a/Modules/_testcapi/clinic/vectorcall_limited.c.h b/Modules/_testcapi/clinic/vectorcall_limited.c.h new file mode 100644 index 00000000000000..a233aefec79ecd --- /dev/null +++ b/Modules/_testcapi/clinic/vectorcall_limited.c.h @@ -0,0 +1,20 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +PyDoc_STRVAR(_testcapi_call_vectorcall__doc__, +"call_vectorcall($module, callable, /)\n" +"--\n" +"\n"); + +#define _TESTCAPI_CALL_VECTORCALL_METHODDEF \ + {"call_vectorcall", (PyCFunction)_testcapi_call_vectorcall, METH_O, _testcapi_call_vectorcall__doc__}, + +PyDoc_STRVAR(_testcapi_call_vectorcall_method__doc__, +"call_vectorcall_method($module, callable, /)\n" +"--\n" +"\n"); + +#define _TESTCAPI_CALL_VECTORCALL_METHOD_METHODDEF \ + {"call_vectorcall_method", (PyCFunction)_testcapi_call_vectorcall_method, METH_O, _testcapi_call_vectorcall_method__doc__}, +/*[clinic end generated code: output=e980906a39602528 input=a9049054013a1b77]*/ diff --git a/Modules/_testcapi/vectorcall_limited.c b/Modules/_testcapi/vectorcall_limited.c index e981c8625c2fb2..3e81903098f954 100644 --- a/Modules/_testcapi/vectorcall_limited.c +++ b/Modules/_testcapi/vectorcall_limited.c @@ -1,7 +1,13 @@ +/* Test Vectorcall in the limited API */ + #define Py_LIMITED_API 0x030c0000 // 3.12 #include "parts.h" +#include "clinic/vectorcall_limited.c.h" -/* Test Vectorcall in the limited API */ +/*[clinic input] +module _testcapi +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=6361033e795369fc]*/ static PyObject * LimitedVectorCallClass_tpcall(PyObject *self, PyObject *args, PyObject *kwargs) { @@ -28,8 +34,16 @@ LimitedVectorCallClass_new(PyTypeObject *tp, PyTypeObject *a, PyTypeObject *kw) return self; } +/*[clinic input] +_testcapi.call_vectorcall + + callable: object + / +[clinic start generated code]*/ + static PyObject * -call_vectorcall(PyObject* self, PyObject *callable) +_testcapi_call_vectorcall(PyObject *module, PyObject *callable) +/*[clinic end generated code: output=bae81eec97fcaad7 input=55d88f92240957ee]*/ { PyObject *args[3] = { NULL, NULL, NULL }; PyObject *kwname = NULL, *kwnames = NULL, *result = NULL; @@ -73,8 +87,16 @@ call_vectorcall(PyObject* self, PyObject *callable) return result; } +/*[clinic input] +_testcapi.call_vectorcall_method + + callable: object + / +[clinic start generated code]*/ + static PyObject * -call_vectorcall_method(PyObject* self, PyObject *callable) +_testcapi_call_vectorcall_method(PyObject *module, PyObject *callable) +/*[clinic end generated code: output=e661f48dda08b6fb input=5ba81c27511395b6]*/ { PyObject *args[3] = { NULL, NULL, NULL }; PyObject *name = NULL, *kwname = NULL, @@ -149,8 +171,8 @@ static PyType_Spec LimitedVectorCallClass_spec = { }; static PyMethodDef TestMethods[] = { - {"call_vectorcall", call_vectorcall, METH_O}, - {"call_vectorcall_method", call_vectorcall_method, METH_O}, + _TESTCAPI_CALL_VECTORCALL_METHODDEF + _TESTCAPI_CALL_VECTORCALL_METHOD_METHODDEF {NULL}, }; diff --git a/Modules/_testclinic_limited.c b/Modules/_testclinic_limited.c index 0b606c9857fc40..4273383816a0dd 100644 --- a/Modules/_testclinic_limited.c +++ b/Modules/_testclinic_limited.c @@ -1,3 +1,9 @@ +// _testclinic_limited can built with the Py_BUILD_CORE_BUILTIN macro defined +// if one of the Modules/Setup files asks to build it as "static" (gh-109723). +#undef Py_BUILD_CORE +#undef Py_BUILD_CORE_MODULE +#undef Py_BUILD_CORE_BUILTIN + // For now, only limited C API 3.13 is supported #define Py_LIMITED_API 0x030d0000 diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index 9c915488f6e0de..e77e30dfe5e821 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -490,6 +490,18 @@ PyDoc_STRVAR(rlock_release_save_doc, \n\ For internal use by `threading.Condition`."); +static PyObject * +rlock_recursion_count(rlockobject *self, PyObject *Py_UNUSED(ignored)) +{ + unsigned long tid = PyThread_get_thread_ident(); + return PyLong_FromUnsignedLong( + self->rlock_owner == tid ? self->rlock_count : 0UL); +} + +PyDoc_STRVAR(rlock_recursion_count_doc, +"_recursion_count() -> int\n\ +\n\ +For internal use by reentrancy checks."); static PyObject * rlock_is_owned(rlockobject *self, PyObject *Py_UNUSED(ignored)) @@ -565,6 +577,8 @@ static PyMethodDef rlock_methods[] = { METH_VARARGS, rlock_acquire_restore_doc}, {"_release_save", (PyCFunction)rlock_release_save, METH_NOARGS, rlock_release_save_doc}, + {"_recursion_count", (PyCFunction)rlock_recursion_count, + METH_NOARGS, rlock_recursion_count_doc}, {"__enter__", _PyCFunction_CAST(rlock_acquire), METH_VARARGS | METH_KEYWORDS, rlock_acquire_doc}, {"__exit__", (PyCFunction)rlock_release, @@ -1066,7 +1080,7 @@ thread_bootstate_free(struct bootstate *boot, int decref) Py_DECREF(boot->args); Py_XDECREF(boot->kwargs); } - PyMem_Free(boot); + PyMem_RawFree(boot); } @@ -1184,13 +1198,16 @@ thread_PyThread_start_new_thread(PyObject *self, PyObject *fargs) return NULL; } - struct bootstate *boot = PyMem_NEW(struct bootstate, 1); + // gh-109795: Use PyMem_RawMalloc() instead of PyMem_Malloc(), + // because it should be possible to call thread_bootstate_free() + // without holding the GIL. + struct bootstate *boot = PyMem_RawMalloc(sizeof(struct bootstate)); if (boot == NULL) { return PyErr_NoMemory(); } boot->tstate = _PyThreadState_New(interp); if (boot->tstate == NULL) { - PyMem_Free(boot); + PyMem_RawFree(boot); if (!PyErr_Occurred()) { return PyErr_NoMemory(); } diff --git a/Modules/_xxtestfuzz/fuzzer.c b/Modules/_xxtestfuzz/fuzzer.c index 54f8a42273401f..816ba09c8fd7de 100644 --- a/Modules/_xxtestfuzz/fuzzer.c +++ b/Modules/_xxtestfuzz/fuzzer.c @@ -193,37 +193,33 @@ static int fuzz_json_loads(const char* data, size_t size) { #define MAX_RE_TEST_SIZE 0x10000 -PyObject* sre_compile_method = NULL; -PyObject* sre_error_exception = NULL; -int SRE_FLAG_DEBUG = 0; +PyObject* re_compile_method = NULL; +PyObject* re_error_exception = NULL; +int RE_FLAG_DEBUG = 0; /* Called by LLVMFuzzerTestOneInput for initialization */ static int init_sre_compile(void) { /* Import sre_compile.compile and sre.error */ - PyObject* sre_compile_module = PyImport_ImportModule("sre_compile"); - if (sre_compile_module == NULL) { + PyObject* re_module = PyImport_ImportModule("re"); + if (re_module == NULL) { return 0; } - sre_compile_method = PyObject_GetAttrString(sre_compile_module, "compile"); - if (sre_compile_method == NULL) { + re_compile_method = PyObject_GetAttrString(re_module, "compile"); + if (re_compile_method == NULL) { return 0; } - PyObject* sre_constants = PyImport_ImportModule("sre_constants"); - if (sre_constants == NULL) { + re_error_exception = PyObject_GetAttrString(re_module, "error"); + if (re_error_exception == NULL) { return 0; } - sre_error_exception = PyObject_GetAttrString(sre_constants, "error"); - if (sre_error_exception == NULL) { - return 0; - } - PyObject* debug_flag = PyObject_GetAttrString(sre_constants, "SRE_FLAG_DEBUG"); + PyObject* debug_flag = PyObject_GetAttrString(re_module, "DEBUG"); if (debug_flag == NULL) { return 0; } - SRE_FLAG_DEBUG = PyLong_AsLong(debug_flag); + RE_FLAG_DEBUG = PyLong_AsLong(debug_flag); return 1; } -/* Fuzz _sre.compile(x) */ +/* Fuzz re.compile(x) */ static int fuzz_sre_compile(const char* data, size_t size) { /* Ignore really long regex patterns that will timeout the fuzzer */ if (size > MAX_RE_TEST_SIZE) { @@ -236,7 +232,7 @@ static int fuzz_sre_compile(const char* data, size_t size) { uint16_t flags = ((uint16_t*) data)[0]; /* We remove the SRE_FLAG_DEBUG if present. This is because it prints to stdout which greatly decreases fuzzing speed */ - flags &= ~SRE_FLAG_DEBUG; + flags &= ~RE_FLAG_DEBUG; /* Pull the pattern from the remaining bytes */ PyObject* pattern_bytes = PyBytes_FromStringAndSize(data + 2, size - 2); @@ -249,9 +245,9 @@ static int fuzz_sre_compile(const char* data, size_t size) { return 0; } - /* compiled = _sre.compile(data[2:], data[0:2] */ + /* compiled = re.compile(data[2:], data[0:2] */ PyObject* compiled = PyObject_CallFunctionObjArgs( - sre_compile_method, pattern_bytes, flags_obj, NULL); + re_compile_method, pattern_bytes, flags_obj, NULL); /* Ignore ValueError as the fuzzer will more than likely generate some invalid combination of flags */ if (compiled == NULL && PyErr_ExceptionMatches(PyExc_ValueError)) { @@ -267,7 +263,7 @@ static int fuzz_sre_compile(const char* data, size_t size) { PyErr_Clear(); } /* Ignore re.error */ - if (compiled == NULL && PyErr_ExceptionMatches(sre_error_exception)) { + if (compiled == NULL && PyErr_ExceptionMatches(re_error_exception)) { PyErr_Clear(); } @@ -531,13 +527,8 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { #if !defined(_Py_FUZZ_ONE) || defined(_Py_FUZZ_fuzz_sre_compile) static int SRE_COMPILE_INITIALIZED = 0; if (!SRE_COMPILE_INITIALIZED && !init_sre_compile()) { - if (!PyErr_ExceptionMatches(PyExc_DeprecationWarning)) { - PyErr_Print(); - abort(); - } - else { - PyErr_Clear(); - } + PyErr_Print(); + abort(); } else { SRE_COMPILE_INITIALIZED = 1; } diff --git a/PCbuild/rt.bat b/PCbuild/rt.bat index 33f4212e14567d..7ae7141bfc4eaa 100644 --- a/PCbuild/rt.bat +++ b/PCbuild/rt.bat @@ -48,7 +48,7 @@ if NOT "%1"=="" (set regrtestargs=%regrtestargs% %1) & shift & goto CheckOpts if not defined prefix set prefix=%pcbuild%amd64 set exe=%prefix%\python%suffix%.exe -set cmd="%exe%" %dashO% -u -Wd -E -bb -m test %regrtestargs% +set cmd="%exe%" %dashO% -m test %regrtestargs% if defined qmode goto Qmode echo Deleting .pyc files ... diff --git a/Python/flowgraph.c b/Python/flowgraph.c index adfcef33895a53..9c24264cfbb459 100644 --- a/Python/flowgraph.c +++ b/Python/flowgraph.c @@ -960,6 +960,7 @@ eliminate_empty_basic_blocks(cfg_builder *g) { while(g->g_entryblock && g->g_entryblock->b_iused == 0) { g->g_entryblock = g->g_entryblock->b_next; } + int next_lbl = get_max_label(g->g_entryblock) + 1; for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { assert(b->b_iused > 0); for (int i = 0; i < b->b_iused; i++) { @@ -969,7 +970,13 @@ eliminate_empty_basic_blocks(cfg_builder *g) { while (target->b_iused == 0) { target = target->b_next; } - instr->i_target = target; + if (instr->i_target != target) { + if (!IS_LABEL(target->b_label)) { + target->b_label.id = next_lbl++; + } + instr->i_target = target; + instr->i_oparg = target->b_label.id; + } assert(instr->i_target && instr->i_target->b_iused > 0); } } diff --git a/Tools/buildbot/test.bat b/Tools/buildbot/test.bat index c1b2605a4b2c7e..781f9a4c8206c8 100644 --- a/Tools/buildbot/test.bat +++ b/Tools/buildbot/test.bat @@ -5,7 +5,7 @@ setlocal set PATH=%PATH%;%SystemRoot%\SysNative\OpenSSH;%SystemRoot%\System32\OpenSSH set here=%~dp0 set rt_opts=-q -d -set regrtest_args=-j1 +set regrtest_args= set arm32_ssh= :CheckOpts @@ -23,7 +23,7 @@ if "%PROCESSOR_ARCHITECTURE%"=="ARM" if "%arm32_ssh%"=="true" goto NativeExecuti if "%arm32_ssh%"=="true" goto :Arm32Ssh :NativeExecution -call "%here%..\..\PCbuild\rt.bat" %rt_opts% -uall -rwW --slowest --timeout=1200 %regrtest_args% +call "%here%..\..\PCbuild\rt.bat" %rt_opts% --slow-ci %regrtest_args% exit /b %ERRORLEVEL% :Arm32Ssh @@ -35,7 +35,7 @@ if NOT "%REMOTE_PYTHON_DIR:~-1,1%"=="\" (set REMOTE_PYTHON_DIR=%REMOTE_PYTHON_DI set TEMP_ARGS=--temp %REMOTE_PYTHON_DIR%temp -set rt_args=%rt_opts% %dashU% -rwW --slowest --timeout=1200 %regrtest_args% %TEMP_ARGS% +set rt_args=%rt_opts% --slow-ci %dashU% %regrtest_args% %TEMP_ARGS% ssh %SSH_SERVER% "set TEMP=%REMOTE_PYTHON_DIR%temp& cd %REMOTE_PYTHON_DIR% & %REMOTE_PYTHON_DIR%PCbuild\rt.bat" %rt_args% set ERR=%ERRORLEVEL% scp %SSH_SERVER%:"%REMOTE_PYTHON_DIR%test-results.xml" "%PYTHON_SOURCE%\test-results.xml" diff --git a/Tools/c-analyzer/cpython/ignored.tsv b/Tools/c-analyzer/cpython/ignored.tsv index 336b0281bda85d..1f398701a7a5b5 100644 --- a/Tools/c-analyzer/cpython/ignored.tsv +++ b/Tools/c-analyzer/cpython/ignored.tsv @@ -580,15 +580,15 @@ Modules/_testmultiphase.c - uninitialized_def - Modules/_testsinglephase.c - global_state - Modules/_xxtestfuzz/_xxtestfuzz.c - _fuzzmodule - Modules/_xxtestfuzz/_xxtestfuzz.c - module_methods - -Modules/_xxtestfuzz/fuzzer.c - SRE_FLAG_DEBUG - +Modules/_xxtestfuzz/fuzzer.c - RE_FLAG_DEBUG - Modules/_xxtestfuzz/fuzzer.c - ast_literal_eval_method - Modules/_xxtestfuzz/fuzzer.c - compiled_patterns - Modules/_xxtestfuzz/fuzzer.c - csv_error - Modules/_xxtestfuzz/fuzzer.c - csv_module - Modules/_xxtestfuzz/fuzzer.c - json_loads_method - Modules/_xxtestfuzz/fuzzer.c - regex_patterns - -Modules/_xxtestfuzz/fuzzer.c - sre_compile_method - -Modules/_xxtestfuzz/fuzzer.c - sre_error_exception - +Modules/_xxtestfuzz/fuzzer.c - re_compile_method - +Modules/_xxtestfuzz/fuzzer.c - re_error_exception - Modules/_xxtestfuzz/fuzzer.c - struct_error - Modules/_xxtestfuzz/fuzzer.c - struct_unpack_method - Modules/_xxtestfuzz/fuzzer.c LLVMFuzzerTestOneInput CSV_READER_INITIALIZED - diff --git a/Tools/scripts/run_tests.py b/Tools/scripts/run_tests.py index 445a34ae3e8eee..3e3d15d3b0da5c 100644 --- a/Tools/scripts/run_tests.py +++ b/Tools/scripts/run_tests.py @@ -18,19 +18,12 @@ def is_multiprocess_flag(arg): return arg.startswith('-j') or arg.startswith('--multiprocess') -def is_resource_use_flag(arg): - return arg.startswith('-u') or arg.startswith('--use') - def is_python_flag(arg): return arg.startswith('-p') or arg.startswith('--python') def main(regrtest_args): - args = [sys.executable, - '-u', # Unbuffered stdout and stderr - '-W', 'default', # Warnings set to 'default' - '-bb', # Warnings about bytes/bytearray - ] + args = [sys.executable] cross_compile = '_PYTHON_HOST_PLATFORM' in os.environ if (hostrunner := os.environ.get("_PYTHON_HOSTRUNNER")) is None: @@ -50,26 +43,18 @@ def main(regrtest_args): } else: environ = os.environ.copy() - args.append("-E") # Allow user-specified interpreter options to override our defaults. args.extend(test.support.args_from_interpreter_flags()) args.extend(['-m', 'test', # Run the test suite - '-r', # Randomize test order - '-w', # Re-run failed tests in verbose mode + '--fast-ci', # Fast Continuous Integration mode ]) - if sys.platform == 'win32': - args.append('-n') # Silence alerts under Windows if not any(is_multiprocess_flag(arg) for arg in regrtest_args): if cross_compile and hostrunner: # For now use only two cores for cross-compiled builds; # hostrunner can be expensive. args.extend(['-j', '2']) - else: - args.extend(['-j', '0']) # Use all CPU cores - if not any(is_resource_use_flag(arg) for arg in regrtest_args): - args.extend(['-u', 'all,-largefile,-audio,-gui']) if cross_compile and hostrunner: # If HOSTRUNNER is set and -p/--python option is not given, then @@ -80,7 +65,8 @@ def main(regrtest_args): args.extend(regrtest_args) - print(shlex.join(args)) + print(shlex.join(args), flush=True) + if sys.platform == 'win32': from subprocess import call sys.exit(call(args)) diff --git a/Tools/unicode/genmap_japanese.py b/Tools/unicode/genmap_japanese.py index 21de37b62bced0..838317fa54175e 100644 --- a/Tools/unicode/genmap_japanese.py +++ b/Tools/unicode/genmap_japanese.py @@ -2,7 +2,7 @@ # genmap_ja_codecs.py: Japanese Codecs Map Generator # # Original Author: Hye-Shik Chang -# Modified Author: Dong-hee Na +# Modified Author: Donghee Na # import os diff --git a/Tools/unicode/genmap_korean.py b/Tools/unicode/genmap_korean.py index 4b94a6c43e5382..4432a3601b7e3b 100644 --- a/Tools/unicode/genmap_korean.py +++ b/Tools/unicode/genmap_korean.py @@ -2,7 +2,7 @@ # genmap_korean.py: Korean Codecs Map Generator # # Original Author: Hye-Shik Chang -# Modified Author: Dong-hee Na +# Modified Author: Donghee Na # import os diff --git a/Tools/unicode/genmap_schinese.py b/Tools/unicode/genmap_schinese.py index 647c0333ed2728..862f1def71d122 100644 --- a/Tools/unicode/genmap_schinese.py +++ b/Tools/unicode/genmap_schinese.py @@ -2,7 +2,7 @@ # genmap_schinese.py: Simplified Chinese Codecs Map Generator # # Original Author: Hye-Shik Chang -# Modified Author: Dong-hee Na +# Modified Author: Donghee Na # import os import re diff --git a/Tools/unicode/genmap_support.py b/Tools/unicode/genmap_support.py index 5e1d9ee77b0002..4649bc3b7125fe 100644 --- a/Tools/unicode/genmap_support.py +++ b/Tools/unicode/genmap_support.py @@ -2,7 +2,7 @@ # genmap_support.py: Multibyte Codec Map Generator # # Original Author: Hye-Shik Chang -# Modified Author: Dong-hee Na +# Modified Author: Donghee Na # diff --git a/Tools/wasm/README.md b/Tools/wasm/README.md index e6dd4d5f00abde..8ef63c6dcd9ddc 100644 --- a/Tools/wasm/README.md +++ b/Tools/wasm/README.md @@ -79,7 +79,7 @@ PIC. To populate the build cache, run: ```shell . /opt/emsdk/emsdk_env.sh embuilder build zlib bzip2 MINIMAL_PIC -embuilder build --pic zlib bzip2 MINIMAL_PIC +embuilder --pic build zlib bzip2 MINIMAL_PIC ```