Merge branch 'main' into pythongh-116380-dir-fd

Author: barneygale

Doc/glossary.rst

@@ -547,12 +547,12 @@ Glossary
       tasks such as compression or hashing.  Also, the GIL is always released
       when doing I/O.
 
-      Past efforts to create a "free-threaded" interpreter (one which locks
-      shared data at a much finer granularity) have not been successful
-      because performance suffered in the common single-processor case. It
-      is believed that overcoming this performance issue would make the
-      implementation much more complicated and therefore costlier to maintain.
-
+      As of Python 3.13, the GIL can be disabled using the :option:`--disable-gil`
+      build configuration. After building Python with this option, code must be
+      run with :option:`-X gil 0 <-X>` or after setting the :envvar:`PYTHON_GIL=0 <PYTHON_GIL>`
+      environment variable. This feature enables improved performance for
+      multi-threaded applications and makes it easier to use multi-core CPUs
+      efficiently. For more details, see :pep:`703`.
 
    hash-based pyc
       A bytecode cache file that uses the hash rather than the last-modified

Doc/library/asyncio-stream.rst

@@ -260,7 +260,7 @@ StreamReader
       buffer is reset.  The :attr:`IncompleteReadError.partial` attribute
       may contain a portion of the separator.
 
-      The *separator* may also be an :term:`iterable` of separators. In this
+      The *separator* may also be a tuple of separators. In this
       case the return value will be the shortest possible that has any
       separator as the suffix. For the purposes of :exc:`LimitOverrunError`,
       the shortest possible separator is considered to be the one that
@@ -270,7 +270,7 @@ StreamReader
 
       .. versionchanged:: 3.13
 
-         The *separator* parameter may now be an :term:`iterable` of
+         The *separator* parameter may now be a :class:`tuple` of
          separators.
 
    .. method:: at_eof()

Doc/library/decimal.rst

@@ -1517,7 +1517,7 @@ are also included in the pure Python version for compatibility.
    the C version uses a thread-local rather than a coroutine-local context and the value
    is ``False``.  This is slightly faster in some nested context scenarios.
 
-.. versionadded:: 3.8.3
+   .. versionadded:: 3.8.3
 
 
 Rounding modes

Doc/library/glob.rst

@@ -75,6 +75,10 @@ The :mod:`glob` module defines the following functions:
       Using the "``**``" pattern in large directory trees may consume
       an inordinate amount of time.
 
+   .. note::
+      This function may return duplicate path names if *pathname*
+      contains multiple "``**``" patterns and *recursive* is true.
+
    .. versionchanged:: 3.5
       Support for recursive globs using "``**``".
 
@@ -94,6 +98,10 @@ The :mod:`glob` module defines the following functions:
    .. audit-event:: glob.glob pathname,recursive glob.iglob
    .. audit-event:: glob.glob/2 pathname,recursive,root_dir,dir_fd glob.iglob
 
+   .. note::
+      This function may return duplicate path names if *pathname*
+      contains multiple "``**``" patterns and *recursive* is true.
+
    .. versionchanged:: 3.5
       Support for recursive globs using "``**``".
 

Doc/library/ipaddress.rst

@@ -334,14 +334,14 @@ write code that handles both IP versions correctly.  Address objects are
    .. attribute:: is_multicast
    .. attribute:: is_private
    .. attribute:: is_global
+
+      .. versionadded:: 3.4
+
    .. attribute:: is_unspecified
    .. attribute:: is_reserved
    .. attribute:: is_loopback
    .. attribute:: is_link_local
 
-      .. versionadded:: 3.4
-         is_global
-
    .. attribute:: is_site_local
 
       ``True`` if the address is reserved for site-local usage.  Note that

Doc/library/platform.rst

@@ -219,8 +219,8 @@ Windows Platform
    default to an empty string).
 
    As a hint: *ptype* is ``'Uniprocessor Free'`` on single processor NT machines
-   and ``'Multiprocessor Free'`` on multi processor machines. The *'Free'* refers
-   to the OS version being free of debugging code. It could also state *'Checked'*
+   and ``'Multiprocessor Free'`` on multi processor machines. The ``'Free'`` refers
+   to the OS version being free of debugging code. It could also state ``'Checked'``
    which means the OS version uses debugging code, i.e. code that checks arguments,
    ranges, etc.
 

Doc/library/ssl.rst

@@ -1820,7 +1820,7 @@ to speed up repeated connections from the same clients.
    .. versionchanged:: 3.6
       *session* argument was added.
 
-    .. versionchanged:: 3.7
+   .. versionchanged:: 3.7
       The method returns an instance of :attr:`SSLContext.sslsocket_class`
       instead of hard-coded :class:`SSLSocket`.
 

Doc/library/urllib.request.rst

@@ -113,9 +113,9 @@ The :mod:`urllib.request` module defines the following functions:
       ``http/1.1`` when no *context* is given. Custom *context* should set
       ALPN protocols with :meth:`~ssl.SSLContext.set_alpn_protocols`.
 
-    .. versionchanged:: 3.13
-       Remove *cafile*, *capath* and *cadefault* parameters: use the *context*
-       parameter instead.
+   .. versionchanged:: 3.13
+      Remove *cafile*, *capath* and *cadefault* parameters: use the *context*
+      parameter instead.
 
 
 .. function:: install_opener(opener)

Doc/library/xml.etree.elementtree.rst

@@ -840,33 +840,28 @@ Functions
 
 .. module:: xml.etree.ElementInclude
 
-.. function:: xml.etree.ElementInclude.default_loader( href, parse, encoding=None)
-   :module:
+.. function:: default_loader(href, parse, encoding=None)
 
-   Default loader. This default loader reads an included resource from disk.  *href* is a URL.
-   *parse* is for parse mode either "xml" or "text".  *encoding*
-   is an optional text encoding.  If not given, encoding is ``utf-8``.  Returns the
-   expanded resource.  If the parse mode is ``"xml"``, this is an ElementTree
-   instance.  If the parse mode is "text", this is a Unicode string.  If the
-   loader fails, it can return None or raise an exception.
+   Default loader. This default loader reads an included resource from disk.
+   *href* is a URL.  *parse* is for parse mode either "xml" or "text".
+   *encoding* is an optional text encoding.  If not given, encoding is ``utf-8``.
+   Returns the expanded resource.
+   If the parse mode is ``"xml"``, this is an :class:`~xml.etree.ElementTree.Element` instance.
+   If the parse mode is ``"text"``, this is a string.
+   If the loader fails, it can return ``None`` or raise an exception.
 
 
-.. function:: xml.etree.ElementInclude.include( elem, loader=None, base_url=None, \
-                                                max_depth=6)
-   :module:
+.. function:: include(elem, loader=None, base_url=None, max_depth=6)
 
-   This function expands XInclude directives.  *elem* is the root element.  *loader* is
-   an optional resource loader.  If omitted, it defaults to :func:`default_loader`.
+   This function expands XInclude directives in-place in tree pointed by *elem*.
+   *elem* is either the root :class:`~xml.etree.ElementTree.Element` or an
+   :class:`~xml.etree.ElementTree.ElementTree` instance to find such element.
+   *loader* is an optional resource loader.  If omitted, it defaults to :func:`default_loader`.
    If given, it should be a callable that implements the same interface as
    :func:`default_loader`.  *base_url* is base URL of the original file, to resolve
    relative include file references.  *max_depth* is the maximum number of recursive
-   inclusions.  Limited to reduce the risk of malicious content explosion. Pass a
-   negative value to disable the limitation.
-
-   Returns the expanded resource.  If the parse mode is
-   ``"xml"``, this is an ElementTree instance.  If the parse mode is "text",
-   this is a Unicode string.  If the loader fails, it can return None or
-   raise an exception.
+   inclusions.  Limited to reduce the risk of malicious content explosion.
+   Pass ``None`` to disable the limitation.
 
    .. versionchanged:: 3.9
       Added the *base_url* and *max_depth* parameters.

Doc/whatsnew/3.13.rst

@@ -324,6 +324,10 @@ asyncio
   :exc:`asyncio.QueueShutDown`) for queue termination.
   (Contributed by Laurie Opperman and Yves Duprat in :gh:`104228`.)
 
+* Accept a tuple of separators in :meth:`asyncio.StreamReader.readuntil`,
+  stopping when one of them is encountered.
+  (Contributed by Bruce Merry in :gh:`81322`.)
+
 base64
 ------
 

Include/Python.h

@@ -45,6 +45,11 @@
 #  endif
 #endif
 
+// gh-111506: The free-threaded build is not compatible with the limited API
+// or the stable ABI.
+#if defined(Py_LIMITED_API) && defined(Py_GIL_DISABLED)
+#  error "The limited API is not currently supported in the free-threaded build"
+#endif
 
 // Include Python header files
 #include "pyport.h"

Include/cpython/pylifecycle.h

@@ -63,14 +63,23 @@ typedef struct {
         .gil = PyInterpreterConfig_OWN_GIL, \
     }
 
+// gh-117649: The free-threaded build does not currently support single-phase
+// init extensions in subinterpreters. For now, we ensure that
+// `check_multi_interp_extensions` is always `1`, even in the legacy config.
+#ifdef Py_GIL_DISABLED
+#  define _PyInterpreterConfig_LEGACY_CHECK_MULTI_INTERP_EXTENSIONS 1
+#else
+#  define _PyInterpreterConfig_LEGACY_CHECK_MULTI_INTERP_EXTENSIONS 0
+#endif
+
 #define _PyInterpreterConfig_LEGACY_INIT \
     { \
         .use_main_obmalloc = 1, \
         .allow_fork = 1, \
         .allow_exec = 1, \
         .allow_threads = 1, \
         .allow_daemon_threads = 1, \
-        .check_multi_interp_extensions = 0, \
+        .check_multi_interp_extensions = _PyInterpreterConfig_LEGACY_CHECK_MULTI_INTERP_EXTENSIONS, \
         .gil = PyInterpreterConfig_SHARED_GIL, \
     }
 

Include/internal/pycore_bytes_methods.h

@@ -26,11 +26,16 @@ extern void _Py_bytes_title(char *result, const char *s, Py_ssize_t len);
 extern void _Py_bytes_capitalize(char *result, const char *s, Py_ssize_t len);
 extern void _Py_bytes_swapcase(char *result, const char *s, Py_ssize_t len);
 
-extern PyObject *_Py_bytes_find(const char *str, Py_ssize_t len, PyObject *args);
-extern PyObject *_Py_bytes_index(const char *str, Py_ssize_t len, PyObject *args);
-extern PyObject *_Py_bytes_rfind(const char *str, Py_ssize_t len, PyObject *args);
-extern PyObject *_Py_bytes_rindex(const char *str, Py_ssize_t len, PyObject *args);
-extern PyObject *_Py_bytes_count(const char *str, Py_ssize_t len, PyObject *args);
+extern PyObject *_Py_bytes_find(const char *str, Py_ssize_t len, PyObject *sub,
+                                Py_ssize_t start, Py_ssize_t end);
+extern PyObject *_Py_bytes_index(const char *str, Py_ssize_t len, PyObject *sub,
+                                 Py_ssize_t start, Py_ssize_t end);
+extern PyObject *_Py_bytes_rfind(const char *str, Py_ssize_t len, PyObject *sub,
+                                 Py_ssize_t start, Py_ssize_t end);
+extern PyObject *_Py_bytes_rindex(const char *str, Py_ssize_t len, PyObject *sub,
+                                 Py_ssize_t start, Py_ssize_t end);
+extern PyObject *_Py_bytes_count(const char *str, Py_ssize_t len, PyObject *sub,
+                                 Py_ssize_t start, Py_ssize_t end);
 extern int _Py_bytes_contains(const char *str, Py_ssize_t len, PyObject *arg);
 extern PyObject *_Py_bytes_startswith(const char *str, Py_ssize_t len,
                                       PyObject *subobj, Py_ssize_t start,

Include/internal/pycore_crossinterp.h

@@ -217,6 +217,11 @@ typedef struct _excinfo {
     const char *errdisplay;
 } _PyXI_excinfo;
 
+PyAPI_FUNC(int) _PyXI_InitExcInfo(_PyXI_excinfo *info, PyObject *exc);
+PyAPI_FUNC(PyObject *) _PyXI_FormatExcInfo(_PyXI_excinfo *info);
+PyAPI_FUNC(PyObject *) _PyXI_ExcInfoAsObject(_PyXI_excinfo *info);
+PyAPI_FUNC(void) _PyXI_ClearExcInfo(_PyXI_excinfo *info);
+
 
 typedef enum error_code {
     _PyXI_ERR_NO_ERROR = 0,
@@ -313,6 +318,22 @@ PyAPI_FUNC(PyObject *) _PyXI_ApplyCapturedException(_PyXI_session *session);
 PyAPI_FUNC(int) _PyXI_HasCapturedException(_PyXI_session *session);
 
 
+/*************/
+/* other API */
+/*************/
+
+// Export for _testinternalcapi shared extension
+PyAPI_FUNC(PyInterpreterState *) _PyXI_NewInterpreter(
+    PyInterpreterConfig *config,
+    long *maybe_whence,
+    PyThreadState **p_tstate,
+    PyThreadState **p_save_tstate);
+PyAPI_FUNC(void) _PyXI_EndInterpreter(
+    PyInterpreterState *interp,
+    PyThreadState *tstate,
+    PyThreadState **p_save_tstate);
+
+
 #ifdef __cplusplus
 }
 #endif

Include/internal/pycore_gc.h

@@ -39,12 +39,13 @@ static inline PyObject* _Py_FROM_GC(PyGC_Head *gc) {
 
 /* Bit flags for ob_gc_bits (in Py_GIL_DISABLED builds) */
 #ifdef Py_GIL_DISABLED
-#  define _PyGC_BITS_TRACKED        (1)
-#  define _PyGC_BITS_FINALIZED      (2)
+#  define _PyGC_BITS_TRACKED        (1)     // Tracked by the GC
+#  define _PyGC_BITS_FINALIZED      (2)     // tp_finalize was called
 #  define _PyGC_BITS_UNREACHABLE    (4)
 #  define _PyGC_BITS_FROZEN         (8)
 #  define _PyGC_BITS_SHARED         (16)
 #  define _PyGC_BITS_SHARED_INLINE  (32)
+#  define _PyGC_BITS_DEFERRED       (64)    // Use deferred reference counting
 #endif
 
 /* True if the object is currently tracked by the GC. */

Include/internal/pycore_global_objects_fini_generated.h

@@ -764,6 +764,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(version)
         STRUCT_FOR_ID(volume)
         STRUCT_FOR_ID(wait_all)
+        STRUCT_FOR_ID(warn_on_full_buffer)
         STRUCT_FOR_ID(warnings)
         STRUCT_FOR_ID(warnoptions)
         STRUCT_FOR_ID(wbits)

Include/internal/pycore_global_strings.h

@@ -103,11 +103,23 @@ struct _is {
     int requires_idref;
     PyThread_type_lock id_mutex;
 
+#define _PyInterpreterState_WHENCE_NOTSET -1
+#define _PyInterpreterState_WHENCE_UNKNOWN 0
+#define _PyInterpreterState_WHENCE_RUNTIME 1
+#define _PyInterpreterState_WHENCE_LEGACY_CAPI 2
+#define _PyInterpreterState_WHENCE_CAPI 3
+#define _PyInterpreterState_WHENCE_XI 4
+#define _PyInterpreterState_WHENCE_STDLIB 5
+#define _PyInterpreterState_WHENCE_MAX 5
+    long _whence;
+
     /* Has been initialized to a safe state.
 
        In order to be effective, this must be set to 0 during or right
        after allocation. */
     int _initialized;
+    /* Has been fully initialized via pylifecycle.c. */
+    int _ready;
     int finalizing;
 
     uintptr_t last_restart_version;
@@ -305,6 +317,13 @@ PyAPI_FUNC(int) _PyInterpreterState_IDInitref(PyInterpreterState *);
 PyAPI_FUNC(int) _PyInterpreterState_IDIncref(PyInterpreterState *);
 PyAPI_FUNC(void) _PyInterpreterState_IDDecref(PyInterpreterState *);
 
+PyAPI_FUNC(int) _PyInterpreterState_IsReady(PyInterpreterState *interp);
+
+PyAPI_FUNC(long) _PyInterpreterState_GetWhence(PyInterpreterState *interp);
+extern void _PyInterpreterState_SetWhence(
+    PyInterpreterState *interp,
+    long whence);
+
 extern const PyConfig* _PyInterpreterState_GetConfig(PyInterpreterState *interp);
 
 // Get a copy of the current interpreter configuration.

Include/internal/pycore_interp.h

@@ -158,6 +158,21 @@ static inline void _Py_ClearImmortal(PyObject *op)
         op = NULL; \
     } while (0)
 
+// Mark an object as supporting deferred reference counting. This is a no-op
+// in the default (with GIL) build. Objects that use deferred reference
+// counting should be tracked by the GC so that they are eventually collected.
+extern void _PyObject_SetDeferredRefcount(PyObject *op);
+
+static inline int
+_PyObject_HasDeferredRefcount(PyObject *op)
+{
+#ifdef Py_GIL_DISABLED
+    return (op->ob_gc_bits & _PyGC_BITS_DEFERRED) != 0;
+#else
+    return 0;
+#endif
+}
+
 #if !defined(Py_GIL_DISABLED)
 static inline void
 _Py_DECREF_SPECIALIZED(PyObject *op, const destructor destruct)

Include/internal/pycore_object.h

@@ -77,6 +77,9 @@ _Py_IsMainInterpreterFinalizing(PyInterpreterState *interp)
             interp == &_PyRuntime._main_interpreter);
 }
 
+// Export for _xxsubinterpreters module.
+PyAPI_FUNC(PyObject *) _PyInterpreterState_GetIDObject(PyInterpreterState *);
+
 // Export for _xxsubinterpreters module.
 PyAPI_FUNC(int) _PyInterpreterState_SetRunningMain(PyInterpreterState *);
 PyAPI_FUNC(void) _PyInterpreterState_SetNotRunningMain(PyInterpreterState *);

Include/internal/pycore_pystate.h

@@ -162,6 +162,7 @@ extern PyTypeObject _PyExc_MemoryError;
 #define _PyInterpreterState_INIT(INTERP) \
     { \
         .id_refcount = -1, \
+        ._whence = _PyInterpreterState_WHENCE_NOTSET, \
         .imports = IMPORTS_INIT, \
         .ceval = { \
             .recursion_limit = Py_DEFAULT_RECURSION_LIMIT, \