From b1aab58f9586c83b90c4611c351f07bd1aad49a9 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Mon, 27 Feb 2023 18:06:12 +0100 Subject: [PATCH 1/5] Add docs for unstable C API (PEP 689) --- developer-workflow/c-api.rst | 130 +++++++++++++++++++++++++++++++---- 1 file changed, 118 insertions(+), 12 deletions(-) diff --git a/developer-workflow/c-api.rst b/developer-workflow/c-api.rst index 3e6199f8eb..e6e318924a 100644 --- a/developer-workflow/c-api.rst +++ b/developer-workflow/c-api.rst @@ -4,18 +4,20 @@ Changing Python's C API ======================= -The C API is divided into three sections: +The C API is divided into these tiers: 1. The internal, private API, available with ``Py_BUILD_CORE`` defined. Ideally declared in ``Include/internal/``. Any API named with a leading underscore is also considered private. -2. The public C API, available when ``Python.h`` is included normally. +2. The “general” public C API, available when ``Python.h`` is included normally. Ideally declared in ``Include/cpython/``. -3. The Limited API, available with ``Py_LIMITED_API`` defined. +3. The Unstable C API, identified by the ``PyUnstable_`` name prefix. + Ideally declared in ``Include/cpython/`` along with the general public API. +4. The Limited C API, available with ``Py_LIMITED_API`` defined. Ideally declared directly under ``Include/``. -Each section has higher stability & maintenance requirements, and you will -need to think about more issues when you add or change definitions in it. +Each tier has different stability & maintenance requirements to think about +when you add or change definitions in it. The compatibility guarantees for public C API are explained in the user documentation, ``Doc/c-api/stable.rst`` (:ref:`python:stable`). @@ -29,6 +31,10 @@ for building CPython itself, as indicated by a macro like ``Py_BUILD_CORE``. While internal API can be changed at any time, it's still good to keep it stable: other API or other CPython developers may depend on it. +For users, internal API is sometimes the best a workaround for a thorny problem +-- though those use cases should be discussed on `Discourse `_ +or an issue so we can try to find a supported way to serve them. + With PyAPI_FUNC or PyAPI_DATA ----------------------------- @@ -36,6 +42,7 @@ With PyAPI_FUNC or PyAPI_DATA Functions or structures in ``Include/internal/`` defined with ``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are exposed only for specific use cases like debuggers and profilers. +Ideally, these should be migrated to the :ref:`unstable-capi`. With the extern keyword @@ -53,14 +60,18 @@ Private names -------------- Any API named with a leading underscore is also considered internal. -There are two main use cases for using such names rather than putting the -definition in ``Include/internal/`` (or directly in a ``.c`` file): +There is currently only one main use case for using such names rather than +putting the definition in ``Include/internal/`` (or directly in a ``.c`` file): + +* Internal helpers for other public API, which users should not call directly. + +Note that historically, underscores were used for API that is better served by +the :ref:`unstable-capi`: + +* “provisional” API, included in a Python release to test real-world + usage of new API; +* API for very specialized uses like JIT compilers. -* Internal helpers for other public API; users should not use these directly; -* “Provisional” API, included in a Python release to test real-world usage - of new API. Such names should be renamed when stabilized; preferably with - a macro aliasing the old name to the new one. - See :pep:`"Finalizing the API" in PEP 590 <590#finalizing-the-api>` for an example. Internal API Tests ------------------ @@ -134,6 +145,101 @@ When moving existing tests, feel free to replace ``TestError`` with ``PyExc_AssertionError`` unless actually testing custom exceptions. +.. _unstable-capi: + +Unstable C API +============== + +The unstable C API tier is meant for extensions that need tight integration +with the interpreter, like debuggers and JIT compilers. +Users of this tier may need to change their code with every minor release. + +In many ways, this tier is like the general C API: + +- it's available when ``Python.h`` is included normally, +- it should be defined in ``Include/cpython/``, +- it requires tests, so we don't break it unintentionally +- it requires docs, so both we and the users + can agree on the expected behavior, +- it is tested and documented in the same way. + +The differences are: + +- Names of functions structs, macros, etc. start with the ``PyUnstable_`` + prefix. This defines what's in the unstable tier. +- The unstable API can change in minor versions, without any deprecation + period. +- A stability note is appears in the docs. + This happens automatically, based on the name + (via :cpy-file:`Doc/tools/extensions/c_annotations.py`). + +Despite being “unstable”, there are rules to make sure third-party code can +use this API reliably: + +* Changes and removals can be done in minor releases (3.x.0, including + Alphas and Betas for 3.x.0). +* Adding new unstable API *for an existing feature* is allowed even after + Beta feature freeze, up until the first Release Candidate. + Consensus on Core Development Discourse or is needed in the Beta period. +* Backwards-incompatible changes should make existing C callers fail to compile. + For example, arguments should be added/removed, or a function should be + renamed. +* When moving API into or out of the Unstable tier, the old name + should continue to be available (but deprecated) until an incompatible + change is made. In other words, while we're allowed to break calling code, + we shouldn't break it *unnecessarily*. + + +Moving API from the public tier to unstable +------------------------------------------- + +* Expose the API under its new name, with the ``PyUnstable_`` prefix. +* Make the old name an alias (e.g. a ``static inline`` function calling the + new function). +* Deprecate the old name, typically using ``Py_DEPRECATED``. +* Announce the change in What's New. + +The old name should continue to be available until an incompatible change is +made. Per Python’s backwards compatibility policy (:pep:`387`), +this deprecation needs to last at least two releases (modulo SC exceptions). + +The rules are relaxed for API that was introduced before the official +Unstable tier. +For the following, you can make an incompatible change (and remove the old name) +as if the function was already part of the Unstable tier: + +* API introduced before Python 3.12 that is documented to be less stable than default. +* API introduced before Python 3.12 that was named with a leading underscore. + +Moving API from the private tier to unstable +-------------------------------------------- + +* Expose the API under its new name, with the ``PyUnstable_`` prefix. +* If the old name is documented, or widely used externally, + make it an alias and deprecate it (typically with ``Py_DEPRECATED``). + It should continue to be available until an incompatible change is made, + as if it was previously public. + + This applies even to underscored names. Python wasn't always strict with + the leading underscore. + +* Announce the change in What's New. + +Moving API from unstable to public +---------------------------------- + +* Expose the API under its new name, without the ``PyUnstable_`` prefix. +* Make the old ``PyUnstable_*`` name be an alias (e.g. a ``static inline`` + function calling the new function). +* Announce the change in What's New. + +The old name should continue to be available until an incompatible change is +made. +There's no need to deprecate it (it was unstable to begin with), +but there's also no need to break working code just because some function +is now ready for a wider audience. + + Limited API =========== From 73e26115cab37e7e927827371b6c0383cb986008 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 7 Mar 2023 14:15:30 +0100 Subject: [PATCH 2/5] Apply suggestions from code review Co-authored-by: C.A.M. Gerlach Co-authored-by: Ezio Melotti --- developer-workflow/c-api.rst | 69 +++++++++++++++++++----------------- 1 file changed, 36 insertions(+), 33 deletions(-) diff --git a/developer-workflow/c-api.rst b/developer-workflow/c-api.rst index e6e318924a..d37c70159f 100644 --- a/developer-workflow/c-api.rst +++ b/developer-workflow/c-api.rst @@ -12,8 +12,8 @@ The C API is divided into these tiers: 2. The “general” public C API, available when ``Python.h`` is included normally. Ideally declared in ``Include/cpython/``. 3. The Unstable C API, identified by the ``PyUnstable_`` name prefix. - Ideally declared in ``Include/cpython/`` along with the general public API. -4. The Limited C API, available with ``Py_LIMITED_API`` defined. + Ideally declared in :cpy-file:`Include/cpython/` along with the general public API. +4. The Limited C API, available with :c:macro:`Py_LIMITED_API` defined. Ideally declared directly under ``Include/``. Each tier has different stability & maintenance requirements to think about @@ -31,8 +31,9 @@ for building CPython itself, as indicated by a macro like ``Py_BUILD_CORE``. While internal API can be changed at any time, it's still good to keep it stable: other API or other CPython developers may depend on it. -For users, internal API is sometimes the best a workaround for a thorny problem --- though those use cases should be discussed on `Discourse `_ +For users, internal API is sometimes the best workaround for a thorny problem +--- though those use cases should be discussed on the +`C API Discourse category `_ or an issue so we can try to find a supported way to serve them. @@ -61,16 +62,16 @@ Private names Any API named with a leading underscore is also considered internal. There is currently only one main use case for using such names rather than -putting the definition in ``Include/internal/`` (or directly in a ``.c`` file): +putting the definition in :cpy-file:`Include/internal/` (or directly in a ``.c`` file): -* Internal helpers for other public API, which users should not call directly. +* Internal helpers for other public APIs, which users should not call directly. -Note that historically, underscores were used for API that is better served by +Note that historically, underscores were used for APIs that are better served by the :ref:`unstable-capi`: -* “provisional” API, included in a Python release to test real-world - usage of new API; -* API for very specialized uses like JIT compilers. +* “provisional” APIs, included in a Python release to test real-world + usage of new APIs; +* APIs for very specialized uses like JIT compilers. Internal API Tests @@ -157,9 +158,9 @@ Users of this tier may need to change their code with every minor release. In many ways, this tier is like the general C API: - it's available when ``Python.h`` is included normally, -- it should be defined in ``Include/cpython/``, +- it should be defined in :cpy-file:`Include/cpython/`, - it requires tests, so we don't break it unintentionally -- it requires docs, so both we and the users +- it requires docs, so both we and the users, can agree on the expected behavior, - it is tested and documented in the same way. @@ -169,7 +170,7 @@ The differences are: prefix. This defines what's in the unstable tier. - The unstable API can change in minor versions, without any deprecation period. -- A stability note is appears in the docs. +- A stability note appears in the docs. This happens automatically, based on the name (via :cpy-file:`Doc/tools/extensions/c_annotations.py`). @@ -178,55 +179,57 @@ use this API reliably: * Changes and removals can be done in minor releases (3.x.0, including Alphas and Betas for 3.x.0). -* Adding new unstable API *for an existing feature* is allowed even after +* Adding a new unstable API *for an existing feature* is allowed even after Beta feature freeze, up until the first Release Candidate. - Consensus on Core Development Discourse or is needed in the Beta period. + Consensus on the `Core Development Discourse `_ + is needed in the Beta period. * Backwards-incompatible changes should make existing C callers fail to compile. For example, arguments should be added/removed, or a function should be renamed. -* When moving API into or out of the Unstable tier, the old name +* When moving an API into or out of the Unstable tier, the old name should continue to be available (but deprecated) until an incompatible change is made. In other words, while we're allowed to break calling code, we shouldn't break it *unnecessarily*. -Moving API from the public tier to unstable -------------------------------------------- +Moving an API from the public tier to Unstable +---------------------------------------------- * Expose the API under its new name, with the ``PyUnstable_`` prefix. * Make the old name an alias (e.g. a ``static inline`` function calling the new function). -* Deprecate the old name, typically using ``Py_DEPRECATED``. -* Announce the change in What's New. +* Deprecate the old name, typically using :c:macro:`Py_DEPRECATED`. +* Announce the change in the "What's New". The old name should continue to be available until an incompatible change is made. Per Python’s backwards compatibility policy (:pep:`387`), -this deprecation needs to last at least two releases (modulo SC exceptions). +this deprecation needs to last at least two releases +(modulo Steering Council exceptions). -The rules are relaxed for API that was introduced before the official -Unstable tier. -For the following, you can make an incompatible change (and remove the old name) -as if the function was already part of the Unstable tier: +The rules are relaxed for APIs that were introduced in Python versions +before 3.12, when the official Unstable tier was added. +You can make an incompatible change (and remove the old name) +as if the function was already part of the Unstable tier +for APIs introduced before Python 3.12 that are either: -* API introduced before Python 3.12 that is documented to be less stable than default. -* API introduced before Python 3.12 that was named with a leading underscore. +* Documented to be less stable than default. +* Named with a leading underscore. -Moving API from the private tier to unstable --------------------------------------------- +Moving an API from the private tier to unstable +----------------------------------------------- * Expose the API under its new name, with the ``PyUnstable_`` prefix. * If the old name is documented, or widely used externally, - make it an alias and deprecate it (typically with ``Py_DEPRECATED``). + make it an alias and deprecate it (typically with :c:macro:`Py_DEPRECATED`). It should continue to be available until an incompatible change is made, as if it was previously public. This applies even to underscored names. Python wasn't always strict with the leading underscore. - * Announce the change in What's New. -Moving API from unstable to public ----------------------------------- +Moving an API from unstable to public +------------------------------------- * Expose the API under its new name, without the ``PyUnstable_`` prefix. * Make the old ``PyUnstable_*`` name be an alias (e.g. a ``static inline`` From b210a1025b1b89bfe2d76f7a5c007011d4fc1aad Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Mon, 17 Apr 2023 17:26:17 +0200 Subject: [PATCH 3/5] Apply suggestions from code review Co-authored-by: C.A.M. Gerlach --- developer-workflow/c-api.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/developer-workflow/c-api.rst b/developer-workflow/c-api.rst index d37c70159f..25a7309111 100644 --- a/developer-workflow/c-api.rst +++ b/developer-workflow/c-api.rst @@ -9,14 +9,14 @@ The C API is divided into these tiers: 1. The internal, private API, available with ``Py_BUILD_CORE`` defined. Ideally declared in ``Include/internal/``. Any API named with a leading underscore is also considered private. -2. The “general” public C API, available when ``Python.h`` is included normally. +2. The “general” public C API, available when :cpy-file:`Include/Python.h` is included normally. Ideally declared in ``Include/cpython/``. 3. The Unstable C API, identified by the ``PyUnstable_`` name prefix. Ideally declared in :cpy-file:`Include/cpython/` along with the general public API. 4. The Limited C API, available with :c:macro:`Py_LIMITED_API` defined. Ideally declared directly under ``Include/``. -Each tier has different stability & maintenance requirements to think about +Each tier has different stability and maintenance requirements to consider when you add or change definitions in it. The compatibility guarantees for public C API are explained in the @@ -177,8 +177,8 @@ The differences are: Despite being “unstable”, there are rules to make sure third-party code can use this API reliably: -* Changes and removals can be done in minor releases (3.x.0, including - Alphas and Betas for 3.x.0). +* Changes and removals can be done in minor releases + (:samp:`3.{x}.0`, including Alphas and Betas for :samp:`3.{x}.0`). * Adding a new unstable API *for an existing feature* is allowed even after Beta feature freeze, up until the first Release Candidate. Consensus on the `Core Development Discourse `_ From b3b0a9bba694c8b55ce29794a33b2ba4a16b551f Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Mon, 17 Apr 2023 17:42:49 +0200 Subject: [PATCH 4/5] Reorder the tier list at the beginning --- developer-workflow/c-api.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/developer-workflow/c-api.rst b/developer-workflow/c-api.rst index 25a7309111..4635699c56 100644 --- a/developer-workflow/c-api.rst +++ b/developer-workflow/c-api.rst @@ -9,10 +9,10 @@ The C API is divided into these tiers: 1. The internal, private API, available with ``Py_BUILD_CORE`` defined. Ideally declared in ``Include/internal/``. Any API named with a leading underscore is also considered private. -2. The “general” public C API, available when :cpy-file:`Include/Python.h` is included normally. - Ideally declared in ``Include/cpython/``. -3. The Unstable C API, identified by the ``PyUnstable_`` name prefix. +2. The Unstable C API, identified by the ``PyUnstable_`` name prefix. Ideally declared in :cpy-file:`Include/cpython/` along with the general public API. +3. The “general” public C API, available when :cpy-file:`Include/Python.h` is included normally. + Ideally declared in ``Include/cpython/``. 4. The Limited C API, available with :c:macro:`Py_LIMITED_API` defined. Ideally declared directly under ``Include/``. From d02a9e9d8bfadfb5d5d1ede17d1110ac5c6596bd Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Mon, 17 Apr 2023 17:54:21 +0200 Subject: [PATCH 5/5] Old names for API that was made public should stay until deprecation --- developer-workflow/c-api.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/developer-workflow/c-api.rst b/developer-workflow/c-api.rst index 4635699c56..daa694814b 100644 --- a/developer-workflow/c-api.rst +++ b/developer-workflow/c-api.rst @@ -236,9 +236,9 @@ Moving an API from unstable to public function calling the new function). * Announce the change in What's New. -The old name should continue to be available until an incompatible change is -made. -There's no need to deprecate it (it was unstable to begin with), +The old name should remain available until the +new public name is deprecated or removed. +There's no need to deprecate the old name (it was unstable to begin with), but there's also no need to break working code just because some function is now ready for a wider audience.