From 7f260c8c11a8c4a022357ac7746508e1977f051e Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Fri, 28 Jun 2024 12:30:07 -0700 Subject: [PATCH 01/41] Type Stubs: move intro and syntax to spec. * Moves some content into spec, deletes duplicate spec content. * Adds placeholders for where remaining content will be moved. It will be split between the spec and the writing_stubs doc. --- docs/guides/writing_stubs.rst | 8 +- docs/reference/stubs.rst | 93 ----------------------- docs/spec/distributing.rst | 139 ++++++++++++++++++++++------------ 3 files changed, 96 insertions(+), 144 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index d463a48b7..304d8cbfe 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -9,7 +9,7 @@ For a full reference, refer to :ref:`stubs`. Maintaining stubs can be a little cumbersome because they are separated from the implementation. This page lists some tools that make writing and maintaining -stubs less painful. +stubs less painful, as well as some best practices on stub contents and style. Tools for generating stubs ========================== @@ -106,3 +106,9 @@ particularly with false positives. If your package has some particularly complex aspects, you could even consider writing dedicated typing tests for tricky definitions. For more details, see :ref:`testing`. + +Stub Content +============ + +Style Guide +=========== diff --git a/docs/reference/stubs.rst b/docs/reference/stubs.rst index 9f207847e..b70214224 100644 --- a/docs/reference/stubs.rst +++ b/docs/reference/stubs.rst @@ -4,97 +4,9 @@ Type Stubs ********** -Introduction -============ - -*type stubs*, also called *stub files*, provide type information for untyped -Python packages and modules. Type stubs serve multiple purposes: - -* They are the only way to add type information to extension modules. -* They can provide type information for packages that do not wish to - add them inline. -* They can be distributed separately from the implementation. - This allows stubs to be developed at a different pace or by different - authors, which is especially useful when adding type annotations to - existing packages. -* They can act as documentation, succinctly explaining the external - API of a package, without including the implementation or private - members. - -This document aims to give guidance to both authors of type stubs and developers -of type checkers and other tools. It describes the constructs that can be used safely in type stubs, -suggests a style guide for them, and lists constructs that type -checkers are expected to support. - -Type stubs that only use constructs described in this document should work with -all type checkers that also follow this document. -Type stub authors can elect to use additional constructs, but -must be prepared that some type checkers will not parse them as expected. - -A type checker that conforms to this document will parse a type stub that only uses -constructs described here without error and will not interpret any -construct in a contradictory manner. However, type checkers are not -required to implement checks for all these constructs, and -can elect to ignore unsupported ones. Additionally type checkers -can support constructs not described in this document and tool authors are -encouraged to experiment with additional features. - -Syntax -====== - -Type stubs are syntactically valid Python 3.8 files with a ``.pyi`` suffix. -The Python syntax used for type stubs is independent from the Python -versions supported by the implementation, and from the Python version the type -checker runs under (if any). Therefore, type stub authors should use the -latest available syntax features in stubs (up to Python 3.8), even if the -implementation supports older, pre-3.8 Python versions. -Type checker authors are encouraged to support syntax features from -post-3.8 Python versions, although type stub authors should not use such -features if they wish to maintain compatibility with all type checkers. - -For example, Python 3.7 added the ``async`` keyword (see :pep:`492`). -Stub authors should use it to mark coroutines, even if the implementation -still uses the ``@coroutine`` decorator. On the other hand, type stubs should -not use the ``type`` soft keyword from :pep:`695`, introduced in -Python 3.12, although type checker authors are encouraged to support it. - -Stubs are treated as if ``from __future__ import annotations`` is enabled. -In particular, built-in generics, pipe union syntax (``X | Y``), and forward -references can be used. - -The :py:mod:`ast` module from the standard library supports -all syntax features required by this document. - -Distribution -============ - -Type stubs can be distributed with or separately from the implementation; -see :ref:`distributing-type` and :ref:`providing-type-annotations` -for more information. - Supported Constructs ==================== -This sections lists constructs that type checkers will accept in type stubs. -Type stub authors can safely use these constructs. If a -construct is marked as "unspecified", type checkers may handle it -as they best see fit or report an error. Linters should usually -flag those constructs. Type stub authors should avoid using them to -ensure compatibility across type checkers. - -Unless otherwise mentioned, type stubs support all features from the -``typing`` module of the latest released Python version. If a stub uses -typing features from a later Python version than what the implementation -supports, these features can be imported from ``typing_extensions`` instead -of ``typing``. - -For example, a stub could use ``Literal``, introduced in Python 3.8, -for a library supporting Python 3.7+:: - - from typing_extensions import Literal - - def foo(x: Literal[""]) -> int: ... - Comments -------- @@ -964,8 +876,3 @@ No:: from typing import NamedTuple, TypedDict Point = NamedTuple("Point", [('x', float), ('y', float)]) Thing = TypedDict("Thing", {'stuff': str, 'index': int}) - -Copyright -========= - -This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 9bcd3a6f8..317621eef 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -10,72 +10,111 @@ Stub files (Originally specified in :pep:`484`.) -Stub files are files containing type hints that are only for use by -the type checker, not at runtime. There are several use cases for -stub files: - -* Extension modules - -* Third-party modules whose authors have not yet added type hints +*Stub files*, also called *type stubs*, provide type information for untyped +Python packages and modules. Stub files serve multiple purposes: + +* They are the only way to add type information to extension modules. +* They can provide type information for packages that do not wish to + add them inline. +* They can be distributed separately from the implementation. + This allows stubs to be developed at a different pace or by different + authors, which is especially useful when adding type annotations to + existing packages. +* They can act as documentation, succinctly explaining the external + API of a package, without including the implementation or private + members. + +Stub files that only use the constructs described in :ref:`Supported Constructs` +below should work with all type checkers that conform to this specification. A +conformant type checker will parse a stub that only uses such constructs without +error and will not interpret any construct in a contradictory manner. However, +type checkers are not required to implement checks for all these constructs and +can elect to ignore unsupported ones. Additionally, type checkers can support +constructs not decribed here, and tool authors are encouraged to experiment with +additional features. + + +Syntax +^^^^^^ + +Stub files are syntactically valid Python files in the earliest Python version +that is not yet end-of-life. They use a ``.pyi`` suffix. The Python syntax used +by stub files is independent from the Python versions supported by the +implementation, and from the Python version the type checker runs under (if +any). Therefore, stub authors should use the latest syntax features available in +the earliest supported version, even if the implementation supports older +versions. Type checker authors are encouraged to support syntax features from +newer versions, although stub authors should not use such features if they wish +to maintain compatibility with all type checkers. + +For example, Python 3.7 added the ``async`` keyword (see :pep:`492`). +Stub authors should use it to mark coroutines, even if the implementation +still uses the ``@coroutine`` decorator. On the other hand, stubs should not use +the ``type`` soft keyword from :pep:`695`, introduced in Python 3.12, util +Python 3.11 reaches end-of-life in October 2027. + +Stubs are treated as if ``from __future__ import annotations`` is enabled. In +particular, built-in generics, pipe union syntax (``X | Y``), and forward +references can be used. + +Type checkers should have a configurable search path for stub files. If a stub +file is found, the type checker should not read the corresponding "real" module. +See :ref:`Import resolution ordering` for more information. + + +Supported Constructs +^^^^^^^^^^^^^^^^^^^^ -* Standard library modules for which type hints have not yet been - written +This section lists constructs that type checkers will accept in stub files. Stub +authors can safely use these constructs. If a construct is marked as +"unspecified", type checkers may handle it as they best see fit or report an +error. Linters should usually flag those constructs. Stub authors should avoid +using them to ensure compatibility across type checkers. -* Modules that must be compatible with Python 2 and 3 +Unless otherwise mentioned, stub files support all features from the ``typing`` +module of the latest released Python version. If a stub uses typing features +from a later Python version than what the implementation supports, these +features can be imported from ``typing_extensions`` instead of ``typing``. -* Modules that use annotations for other purposes +For example, a stub could use ``Literal``, introduced in Python 3.8, +for a library supporting Python 3.7+:: -Stub files have the same syntax as regular Python modules. There is one -feature of the ``typing`` module that is different in stub files: -the ``@overload`` decorator described below. + from typing_extensions import Literal -The type checker should only check function signatures in stub files; -It is recommended that function bodies in stub files just be a single -ellipsis (``...``). + def foo(x: Literal[""]) -> int: ... -The type checker should have a configurable search path for stub files. -If a stub file is found the type checker should not read the -corresponding "real" module. +Comments +"""""""" -While stub files are syntactically valid Python modules, they use the -``.pyi`` extension to make it possible to maintain stub files in the -same directory as the corresponding real module. This also reinforces -the notion that no runtime behavior should be expected of stub files. +Imports +""""""" -Additional notes on stub files: +Built-in Generics +""""""""""""""""" -* Modules and variables imported into the stub are not considered - exported from the stub unless the import uses the ``import ... as - ...`` form or the equivalent ``from ... import ... as ...`` form. - (*UPDATE:* To clarify, the intention here is that only names - imported using the form ``X as X`` will be exported, i.e. the name - before and after ``as`` must be the same.) +Unions +"""""" -* However, as an exception to the previous bullet, all objects - imported into a stub using ``from ... import *`` are considered - exported. (This makes it easier to re-export all objects from a - given module that may vary by Python version.) +Module Level Attributes +""""""""""""""""""""""" -* Just like in `normal Python files `_, submodules - automatically become exported attributes of their parent module - when imported. For example, if the ``spam`` package has the - following directory structure:: +Classes +""""""" - spam/ - __init__.pyi - ham.pyi +Functions and Methods +""""""""""""""""""""" - where ``__init__.pyi`` contains a line such as ``from . import ham`` - or ``from .ham import Ham``, then ``ham`` is an exported attribute - of ``spam``. +Aliases and NewType +""""""""""""""""""" -* Stub files may be incomplete. To make type checkers aware of this, the file - can contain the following code:: +Decorators +"""""""""" - def __getattr__(name) -> Any: ... +Version and Platform Checks +""""""""""""""""""""""""""" - Any identifier not defined in the stub is therefore assumed to be of type - ``Any``. +Enums +""""" The Typeshed Project ^^^^^^^^^^^^^^^^^^^^ From 1d29334941088294987b74fe58cf2b8084c56191 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Fri, 28 Jun 2024 12:54:19 -0700 Subject: [PATCH 02/41] Move "Stub Contents" to writing_stubs. Moves the section on what to include in stubs with editorial changes only - e.g., changing "type stub" to "stub file" to match surrounding terminology. --- docs/guides/writing_stubs.rst | 230 +++++++++++++++++++++++++++++ docs/reference/best_practices.rst | 2 + docs/reference/stubs.rst | 232 ------------------------------ 3 files changed, 232 insertions(+), 232 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 304d8cbfe..baaf19095 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -110,5 +110,235 @@ writing dedicated typing tests for tricky definitions. For more details, see Stub Content ============ +This section documents best practices on what elements to include or +leave out of stub files. + +Modules excluded fom stubs +-------------------------- + +Not all modules should be included into stubs. + +It is recommended to exclude: + +1. Implementation details, with `multiprocessing/popen_spawn_win32.py `_ as a notable example +2. Modules that are not supposed to be imported, such as ``__main__.py`` +3. Protected modules that start with a single ``_`` char. However, when needed protected modules can still be added (see :ref:`undocumented-objects` section below) + +Public Interface +---------------- + +Stubs should include the complete public interface (classes, functions, +constants, etc.) of the module they cover, but it is not always +clear exactly what is part of the interface. + +The following should always be included: + +* All objects listed in the module's documentation. +* All objects included in ``__all__`` (if present). + +Other objects may be included if they are not prefixed with an underscore +or if they are being used in practice. (See the next section.) + +.. _undocumented-objects: + +Undocumented Objects +-------------------- + +Undocumented objects may be included as long as they are marked with a comment +of the form ``# undocumented``. + +Example:: + + def list2cmdline(seq: Sequence[str]) -> str: ... # undocumented + +Such undocumented objects are allowed because omitting objects can confuse +users. Users who see an error like "module X has no attribute Y" will +not know whether the error appeared because their code had a bug or +because the stub is wrong. Although it may also be helpful for a type +checker to point out usage of private objects, false negatives (no errors for +wrong code) are preferable over false positives (type errors +for correct code). In addition, even for private objects a type checker +can be helpful in pointing out that an incorrect type was used. + +``__all__`` +------------ + +A stub file should contain an ``__all__`` variable if and only if it also +present at runtime. In that case, the contents of ``__all__`` should be +identical in the stub and at runtime. If the runtime dynamically adds +or removes elements (for example if certain functions are only available on +some platforms), include all possible elements in the stubs. + +Stub-Only Objects +----------------- + +Definitions that do not exist at runtime may be included in stubs to aid in +expressing types. Sometimes, it is desirable to make a stub-only class available +to a stub's users - for example, to allow them to type the return value of a +public method for which a library does not provided a usable runtime type:: + + from typing import Protocol + + class _Readable(Protocol): + def read(self) -> str: ... + + def get_reader() -> _Readable: ... + +Structural Types +---------------- + +As seen in the example with ``_Readable`` in the previous section, a common use +of stub-only objects is to model types that are best described by their +structure. These objects are called protocols (:pep:`544`), and it is encouraged +to use them freely to describe simple structural types. + +It is `recommended <#private-definitions>`_ to prefix stub-only object names with ``_``. + +Incomplete Stubs +---------------- + +Partial stubs can be useful, especially for larger packages, but they should +follow the following guidelines: + +* Included functions and methods should list all arguments, but the arguments + can be left unannotated. +* Do not use ``Any`` to mark unannotated arguments or return values. +* Partial classes should include a ``__getattr__()`` method marked with an + ``# incomplete`` comment (see example below). +* Partial modules (i.e. modules that are missing some or all classes, + functions, or attributes) should include a top-level ``__getattr__()`` + function marked with an ``# incomplete`` comment (see example below). +* Partial packages (i.e. packages that are missing one or more sub-modules) + should have a ``__init__.pyi`` stub that is marked as incomplete (see above). + A better alternative is to create empty stubs for all sub-modules and + mark them as incomplete individually. + +Example of a partial module with a partial class ``Foo`` and a partially +annotated function ``bar()``:: + + def __getattr__(name: str) -> Any: ... # incomplete + + class Foo: + def __getattr__(self, name: str) -> Any: ... # incomplete + x: int + y: str + + def bar(x: str, y, *, z=...): ... + +The ``# incomplete`` comment is mainly intended as a reminder for stub +authors, but can be used by tools to flag such items. + +Attribute Access +---------------- + +Python has several methods for customizing attribute access: ``__getattr__``, +``__getattribute__``, ``__setattr__``, and ``__delattr__``. Of these, +``__getattr__`` and ``__setattr___`` should sometimes be included in stubs. + +In addition to marking incomplete definitions, ``__getattr__`` should be +included when a class or module allows any name to be accessed. For example, consider +the following class:: + + class Foo: + def __getattribute__(self, name): + return self.__dict__.setdefault(name) + +An appropriate stub definition is:: + + from typing import Any + + class Foo: + def __getattr__(self, name: str) -> Any | None: ... + +Note that only ``__getattr__``, not ``__getattribute__``, is guaranteed to be +supported in stubs. + +On the other hand, ``__getattr__`` should be omitted even if the source code +includes it, if only limited names are allowed. For example, consider this class:: + + class ComplexNumber: + def __init__(self, n): + self._n = n + def __getattr__(self, name): + if name in ("real", "imag"): + return getattr(self._n, name) + raise AttributeError(name) + +In this case, the stub should list the attributes individually:: + + class ComplexNumber: + @property + def real(self) -> float: ... + @property + def imag(self) -> float: ... + def __init__(self, n: complex) -> None: ... + +``__setattr___`` should be included when a class allows any name to be set and +restricts the type. For example:: + + class IntHolder: + def __setattr__(self, name, value): + if isinstance(value, int): + return super().__setattr__(name, value) + raise ValueError(value) + +A good stub definition would be:: + + class IntHolder: + def __setattr__(self, name: str, value: int) -> None: ... + +``__delattr__`` should not be included in stubs. + +Finally, even in the presence of ``__getattr__`` and ``__setattr__``, it is +still recommended to separately define known attributes. + +Constants +--------- + +When the value of a constant is important, annotate it using ``Literal`` +instead of its type. + +Yes:: + + TEL_LANDLINE: Literal["landline"] + TEL_MOBILE: Literal["mobile"] + DAY_FLAG: Literal[0x01] + NIGHT_FLAG: Literal[0x02] + +No:: + + TEL_LANDLINE: str + TEL_MOBILE: str + DAY_FLAG: int + NIGHT_FLAG: int + +Documentation or Implementation +------------------------------- + +Sometimes a library's documented types will differ from the actual types in the +code. In such cases, stub authors should use their best judgment. Consider these +two examples:: + + def print_elements(x): + """Print every element of list x.""" + for y in x: + print(y) + + def maybe_raise(x): + """Raise an error if x (a boolean) is true.""" + if x: + raise ValueError() + +The implementation of ``print_elements`` takes any iterable, despite the +documented type of ``list``. In this case, annotate the argument as +``Iterable[Any]``, to follow the :ref:`best practice` +of preferring abstract types for arguments. + +For ``maybe_raise``, on the other hand, it is better to annotate the argument as +``bool`` even though the implementation accepts any object. This guards against +common mistakes like unintentionally passing in ``None``. + +If in doubt, consider asking the library maintainers about their intent. + Style Guide =========== diff --git a/docs/reference/best_practices.rst b/docs/reference/best_practices.rst index 24a6ff880..a6585c07e 100644 --- a/docs/reference/best_practices.rst +++ b/docs/reference/best_practices.rst @@ -59,6 +59,8 @@ annotate it with ``object``:: if isinstance(o, int): cb(o) +.. _argument-return-practices: + Arguments and Return Types -------------------------- diff --git a/docs/reference/stubs.rst b/docs/reference/stubs.rst index b70214224..43e96a745 100644 --- a/docs/reference/stubs.rst +++ b/docs/reference/stubs.rst @@ -416,238 +416,6 @@ No:: BLUE: int rgb_value: int # no way for type checkers to know that this is not an enum member -Type Stub Content -================= - -This section documents best practices on what elements to include or -leave out of type stubs. - -Modules excluded fom stubs --------------------------- - -Not all modules should be included into stubs. - -It is recommended to exclude: - -1. Implementation details, with `multiprocessing/popen_spawn_win32.py `_ as a notable example -2. Modules that are not supposed to be imported, such as ``__main__.py`` -3. Protected modules that start with a single ``_`` char. However, when needed protected modules can still be added (see :ref:`undocumented-objects` section below) - -Public Interface ----------------- - -Stubs should include the complete public interface (classes, functions, -constants, etc.) of the module they cover, but it is not always -clear exactly what is part of the interface. - -The following should always be included: - -* All objects listed in the module's documentation. -* All objects included in ``__all__`` (if present). - -Other objects may be included if they are not prefixed with an underscore -or if they are being used in practice. (See the next section.) - -.. _undocumented-objects: - -Undocumented Objects --------------------- - -Undocumented objects may be included as long as they are marked with a comment -of the form ``# undocumented``. - -Example:: - - def list2cmdline(seq: Sequence[str]) -> str: ... # undocumented - -Such undocumented objects are allowed because omitting objects can confuse -users. Users who see an error like "module X has no attribute Y" will -not know whether the error appeared because their code had a bug or -because the stub is wrong. Although it may also be helpful for a type -checker to point out usage of private objects, false negatives (no errors for -wrong code) are preferable over false positives (type errors -for correct code). In addition, even for private objects a type checker -can be helpful in pointing out that an incorrect type was used. - -``__all__`` ------------- - -A type stub should contain an ``__all__`` variable if and only if it also -present at runtime. In that case, the contents of ``__all__`` should be -identical in the stub and at runtime. If the runtime dynamically adds -or removes elements (for example if certain functions are only available on -some platforms), include all possible elements in the stubs. - -Stub-Only Objects ------------------ - -Definitions that do not exist at runtime may be included in stubs to aid in -expressing types. Sometimes, it is desirable to make a stub-only class available -to a stub's users - for example, to allow them to type the return value of a -public method for which a library does not provided a usable runtime type:: - - from typing import Protocol - - class _Readable(Protocol): - def read(self) -> str: ... - - def get_reader() -> _Readable: ... - -Structural Types ----------------- - -As seen in the example with ``_Readable`` in the previous section, a common use -of stub-only objects is to model types that are best described by their -structure. These objects are called protocols (:pep:`544`), and it is encouraged -to use them freely to describe simple structural types. - -It is `recommended <#private-definitions>`_ to prefix stubs-only object names with ``_``. - -Incomplete Stubs ----------------- - -Partial stubs can be useful, especially for larger packages, but they should -follow the following guidelines: - -* Included functions and methods should list all arguments, but the arguments - can be left unannotated. -* Do not use ``Any`` to mark unannotated arguments or return values. -* Partial classes should include a ``__getattr__()`` method marked with an - ``# incomplete`` comment (see example below). -* Partial modules (i.e. modules that are missing some or all classes, - functions, or attributes) should include a top-level ``__getattr__()`` - function marked with an ``# incomplete`` comment (see example below). -* Partial packages (i.e. packages that are missing one or more sub-modules) - should have a ``__init__.pyi`` stub that is marked as incomplete (see above). - A better alternative is to create empty stubs for all sub-modules and - mark them as incomplete individually. - -Example of a partial module with a partial class ``Foo`` and a partially -annotated function ``bar()``:: - - def __getattr__(name: str) -> Any: ... # incomplete - - class Foo: - def __getattr__(self, name: str) -> Any: ... # incomplete - x: int - y: str - - def bar(x: str, y, *, z=...): ... - -The ``# incomplete`` comment is mainly intended as a reminder for stub -authors, but can be used by tools to flag such items. - -Attribute Access ----------------- - -Python has several methods for customizing attribute access: ``__getattr__``, -``__getattribute__``, ``__setattr__``, and ``__delattr__``. Of these, -``__getattr__`` and ``__setattr___`` should sometimes be included in stubs. - -In addition to marking incomplete definitions, ``__getattr__`` should be -included when a class or module allows any name to be accessed. For example, consider -the following class:: - - class Foo: - def __getattribute__(self, name): - return self.__dict__.setdefault(name) - -An appropriate stub definition is:: - - from typing import Any - class Foo: - def __getattr__(self, name: str) -> Any | None: ... - -Note that only ``__getattr__``, not ``__getattribute__``, is guaranteed to be -supported in stubs. - -On the other hand, ``__getattr__`` should be omitted even if the source code -includes it, if only limited names are allowed. For example, consider this class:: - - class ComplexNumber: - def __init__(self, n): - self._n = n - def __getattr__(self, name): - if name in ("real", "imag"): - return getattr(self._n, name) - raise AttributeError(name) - -In this case, the stub should list the attributes individually:: - - class ComplexNumber: - @property - def real(self) -> float: ... - @property - def imag(self) -> float: ... - def __init__(self, n: complex) -> None: ... - -``__setattr___`` should be included when a class allows any name to be set and -restricts the type. For example:: - - class IntHolder: - def __setattr__(self, name, value): - if isinstance(value, int): - return super().__setattr__(name, value) - raise ValueError(value) - -A good stub definition would be:: - - class IntHolder: - def __setattr__(self, name: str, value: int) -> None: ... - -``__delattr__`` should not be included in stubs. - -Finally, even in the presence of ``__getattr__`` and ``__setattr__``, it is -still recommended to separately define known attributes. - -Constants ---------- - -When the value of a constant is important, annotate it using ``Literal`` -instead of its type. - -Yes:: - - TEL_LANDLINE: Literal["landline"] - TEL_MOBILE: Literal["mobile"] - DAY_FLAG: Literal[0x01] - NIGHT_FLAG: Literal[0x02] - -No:: - - TEL_LANDLINE: str - TEL_MOBILE: str - DAY_FLAG: int - NIGHT_FLAG: int - -Documentation or Implementation -------------------------------- - -Sometimes a library's documented types will differ from the actual types in the -code. In such cases, type stub authors should use their best judgment. Consider -these two examples:: - - def print_elements(x): - """Print every element of list x.""" - for y in x: - print(y) - - def maybe_raise(x): - """Raise an error if x (a boolean) is true.""" - if x: - raise ValueError() - -The implementation of ``print_elements`` takes any iterable, despite the -documented type of ``list``. In this case, annotate the argument as -``Iterable[Any]``, to follow this PEP's style recommendation of preferring -abstract types. - -For ``maybe_raise``, on the other hand, it is better to annotate the argument as -``bool`` even though the implementation accepts any object. This guards against -common mistakes like unintentionally passing in ``None``. - -If in doubt, consider asking the library maintainers about their intent. - Style Guide =========== From 065c3d3a6b81c01a828840b9fa6c7edacc17f389 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Fri, 28 Jun 2024 13:14:02 -0700 Subject: [PATCH 03/41] Moves "Style Guide" to writing_stubs doc. Moves the style guide with a few updates: * Editorial changes like tweaking code examples to use recommended style. * Attribute declarations can use assignments when needed to convey extra information - e.g., final attributes and enum members. * Tweaks a reference to the double-underscore convention for positional-only arguments to note that it's historical. --- docs/guides/writing_stubs.rst | 234 ++++++++++++++++++++++++++++++++++ docs/reference/stubs.rst | 229 --------------------------------- docs/spec/distributing.rst | 1 + 3 files changed, 235 insertions(+), 229 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index baaf19095..6016a926a 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -342,3 +342,237 @@ If in doubt, consider asking the library maintainers about their intent. Style Guide =========== + +The recommendations in this section are aimed at stub authors who wish to +provide a consistent style for stubs. Type checkers should not reject stubs that +do not follow these recommendations, but linters can warn about them. + +Stub files should generally follow the Style Guide for Python Code (:pep:`8`) +and the :ref:`best-practices`. There are a few exceptions, outlined below, that take the +different structure of stub files into account and aim to create +more concise files. + +Maximum Line Length +------------------- + +Stub files should be limited to 130 characters per line. + +Blank Lines +----------- + +Do not use empty lines between functions, methods, and fields, except to +group them with one empty line. Use one empty line around classes with non-empty +bodies. Do not use empty lines between body-less classes, except for grouping. + +Yes:: + + def time_func() -> None: ... + def date_func() -> None: ... + + def ip_func() -> None: ... + + class Foo: + x: int + y: int + def __init__(self) -> None: ... + + class MyError(Exception): ... + class AnotherError(Exception): ... + +No:: + + def time_func() -> None: ... + + def date_func() -> None: ... # do no leave unnecessary empty lines + + def ip_func() -> None: ... + + + class Foo: # leave only one empty line above + x: int + class MyError(Exception): ... # leave an empty line between the classes + +Module Level Attributes +----------------------- + +Do not unnecessarily use an assignment for module-level attributes. + +Yes:: + + CONST: Literal["const"] + x: int + y: Final = 0 # this assignment conveys additional type information + +No:: + + CONST = "const" + x: int = 0 + y: float = ... + z = 0 # type: int + a = ... # type: int + +.. _stub-style-classes: + +Classes +------- + +Classes without bodies should use the ellipsis literal ``...`` in place +of the body on the same line as the class definition. + +Yes:: + + class MyError(Exception): ... + +No:: + + class MyError(Exception): + ... + class AnotherError(Exception): pass + +Instance attributes and class variables follow the same recommendations as +module level attributes: + +Yes:: + + class Foo: + c: ClassVar[str] + x: int + + class Color(Enum): + # An assignment with no type annotation is a convention used to indicate + # an enum member. + RED = 1 + +No:: + + class Foo: + c: ClassVar[str] = "" + d: ClassVar[int] = ... + x = 4 + y: int = ... + +Functions and Methods +--------------------- + +Use the same argument names as in the implementation, because +otherwise using keyword arguments will fail. Of course, this +does not apply to positional-only arguments marked with the historical double +underscore convention. + +Use the ellipsis literal ``...`` in place of actual default argument +values. Use an explicit ``X | None`` annotation instead of +a ``None`` default. + +Yes:: + + def foo(x: int = ...) -> None: ... + def bar(y: str | None = ...) -> None: ... + +No:: + + def foo(x: int = 0) -> None: ... + def bar(y: str = None) -> None: ... + def baz(z: str | None = None) -> None: ... + +Do not annotate ``self`` and ``cls`` in method definitions, except when +referencing a type variable. + +Yes:: + + _T = TypeVar("_T") + + class Foo: + def bar(self) -> None: ... + @classmethod + def create(cls: type[_T]) -> _T: ... + +No:: + + class Foo: + def bar(self: Foo) -> None: ... + @classmethod + def baz(cls: type[Foo]) -> int: ... + +The bodies of functions and methods should consist of only the ellipsis +literal ``...`` on the same line as the closing parenthesis and colon. + +Yes:: + + def to_int1(x: str) -> int: ... + def to_int2( + x: str, + ) -> int: ... + +No:: + + def to_int1(x: str) -> int: + return int(x) + def to_int2(x: str) -> int: + ... + def to_int3(x: str) -> int: pass + +.. _private-definitions: + +Private Definitions +------------------- + +Type variables, type aliases, and other definitions that should not +be used outside the stub should be marked as private by prefixing them +with an underscore. + +Yes:: + + _T = TypeVar("_T") + _DictList = Dict[str, List[Optional[int]] + +No:: + + T = TypeVar("T") + DictList = Dict[str, List[Optional[int]]] + +Language Features +----------------- + +Use the latest language features available, even for stubs targeting older +Python versions. Do not use quotes around forward references and do not use +``__future__`` imports. See :ref:`stub-file-syntax` for more information. + +Yes:: + + class Py35Class: + x: int + forward_reference: OtherClass + + class OtherClass: ... + +No:: + + class Py35Class: + x = 0 # type: int + forward_reference: 'OtherClass' + + class OtherClass: ... + +NamedTuple and TypedDict +------------------------ + +Use the class-based syntax for ``typing.NamedTuple`` and +``typing.TypedDict``, following the :ref:`stub-style-classes` section of this style guide. + +Yes:: + + from typing import NamedTuple, TypedDict + + class Point(NamedTuple): + x: float + y: float + + class Thing(TypedDict): + stuff: str + index: int + +No:: + + from typing import NamedTuple, TypedDict + Point = NamedTuple("Point", [('x', float), ('y', float)]) + Thing = TypedDict("Thing", {'stuff': str, 'index': int}) diff --git a/docs/reference/stubs.rst b/docs/reference/stubs.rst index 43e96a745..21d9f15aa 100644 --- a/docs/reference/stubs.rst +++ b/docs/reference/stubs.rst @@ -415,232 +415,3 @@ No:: RED: int BLUE: int rgb_value: int # no way for type checkers to know that this is not an enum member - -Style Guide -=========== - -The recommendations in this section are aimed at type stub authors -who wish to provide a consistent style for type stubs. Type checkers -should not reject stubs that do not follow these recommendations, but -linters can warn about them. - -Stub files should generally follow the Style Guide for Python Code (:pep:`8`) -and the :ref:`best-practices`. There are a few exceptions, outlined below, that take the -different structure of stub files into account and are aimed to create -more concise files. - -Maximum Line Length -------------------- - -Type stubs should be limited to 130 characters per line. - -Blank Lines ------------ - -Do not use empty lines between functions, methods, and fields, except to -group them with one empty line. Use one empty line around classes, but do not -use empty lines between body-less classes, except for grouping. - -Yes:: - - def time_func() -> None: ... - def date_func() -> None: ... - - def ip_func() -> None: ... - - class Foo: - x: int - y: int - def __init__(self) -> None: ... - - class MyError(Exception): ... - class AnotherError(Exception): ... - -No:: - - def time_func() -> None: ... - - def date_func() -> None: ... # do no leave unnecessary empty lines - - def ip_func() -> None: ... - - - class Foo: # leave only one empty line above - x: int - class MyError(Exception): ... # leave an empty line between the classes - -Module Level Attributes ------------------------ - -Do not use an assignment for module-level attributes. - -Yes:: - - CONST: Literal["const"] - x: int - -No:: - - CONST = "const" - x: int = 0 - y: float = ... - z = 0 # type: int - a = ... # type: int - -.. _stub-style-classes: - -Classes -------- - -Classes without bodies should use the ellipsis literal ``...`` in place -of the body on the same line as the class definition. - -Yes:: - - class MyError(Exception): ... - -No:: - - class MyError(Exception): - ... - class AnotherError(Exception): pass - -Instance attributes and class variables follow the same recommendations as -module level attributes: - -Yes:: - - class Foo: - c: ClassVar[str] - x: int - -No:: - - class Foo: - c: ClassVar[str] = "" - d: ClassVar[int] = ... - x = 4 - y: int = ... - -Functions and Methods ---------------------- - -Use the same argument names as in the implementation, because -otherwise using keyword arguments will fail. Of course, this -does not apply to positional-only arguments, which are marked with a double -underscore. - -Use the ellipsis literal ``...`` in place of actual default argument -values. Use an explicit ``X | None`` annotation instead of -a ``None`` default. - -Yes:: - - def foo(x: int = ...) -> None: ... - def bar(y: str | None = ...) -> None: ... - -No:: - - def foo(x: int = 0) -> None: ... - def bar(y: str = None) -> None: ... - def baz(z: str | None = None) -> None: ... - -Do not annotate ``self`` and ``cls`` in method definitions, except when -referencing a type variable. - -Yes:: - - _T = TypeVar("_T") - class Foo: - def bar(self) -> None: ... - @classmethod - def create(cls: type[_T]) -> _T: ... - -No:: - - class Foo: - def bar(self: Foo) -> None: ... - @classmethod - def baz(cls: type[Foo]) -> int: ... - -The bodies of functions and methods should consist of only the ellipsis -literal ``...`` on the same line as the closing parenthesis and colon. - -Yes:: - - def to_int1(x: str) -> int: ... - def to_int2( - x: str, - ) -> int: ... - -No:: - - def to_int1(x: str) -> int: - return int(x) - def to_int2(x: str) -> int: - ... - def to_int3(x: str) -> int: pass - -.. _private-definitions: - -Private Definitions -------------------- - -Type variables, type aliases, and other definitions that should not -be used outside the stub should be marked as private by prefixing them -with an underscore. - -Yes:: - - _T = TypeVar("_T") - _DictList = Dict[str, List[Optional[int]] - -No:: - - T = TypeVar("T") - DictList = Dict[str, List[Optional[int]]] - -Language Features ------------------ - -Use the latest language features available as outlined -in the Syntax_ section, even for stubs targeting older Python versions. -Do not use quotes around forward references and do not use ``__future__`` -imports. - -Yes:: - - class Py35Class: - x: int - forward_reference: OtherClass - class OtherClass: ... - -No:: - - class Py35Class: - x = 0 # type: int - forward_reference: 'OtherClass' - class OtherClass: ... - -NamedTuple and TypedDict ------------------------- - -Use the class-based syntax for ``typing.NamedTuple`` and -``typing.TypedDict``, following the :ref:`stub-style-classes` section of this style guide. - -Yes:: - - from typing import NamedTuple, TypedDict - class Point(NamedTuple): - x: float - y: float - - class Thing(TypedDict): - stuff: str - index: int - -No:: - - from typing import NamedTuple, TypedDict - Point = NamedTuple("Point", [('x', float), ('y', float)]) - Thing = TypedDict("Thing", {'stuff': str, 'index': int}) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 317621eef..fa851695c 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -33,6 +33,7 @@ can elect to ignore unsupported ones. Additionally, type checkers can support constructs not decribed here, and tool authors are encouraged to experiment with additional features. +.. _stub-file-syntax: Syntax ^^^^^^ From d283c387c5c6309e880788d3ba4b4e846635b8a6 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 1 Jul 2024 22:10:28 -0700 Subject: [PATCH 04/41] Change :ref:`stubs` occurrences to :ref:`stub-files`. --- docs/guides/libraries.rst | 2 +- docs/guides/writing_stubs.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/libraries.rst b/docs/guides/libraries.rst index 376811ad0..417c86b47 100644 --- a/docs/guides/libraries.rst +++ b/docs/guides/libraries.rst @@ -43,7 +43,7 @@ library: Inline type annotations simply refers to the use of annotations within your ``.py`` files. In contrast, with type stub files, type information lives in -separate ``.pyi`` files; see :ref:`stubs` and :ref:`writing_stubs` for more +separate ``.pyi`` files; see :ref:`stub-files` and :ref:`writing_stubs` for more details. We recommend using the inline type annotations approach, since it has the diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 6016a926a..0af304480 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -5,7 +5,7 @@ Writing and Maintaining Stub Files ********************************** Stub files are a means of providing type information for Python modules. -For a full reference, refer to :ref:`stubs`. +For a full reference, refer to :ref:`stub-files`. Maintaining stubs can be a little cumbersome because they are separated from the implementation. This page lists some tools that make writing and maintaining From 01b34caee0e5d877906df58b2eb26eace0937e21 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Wed, 3 Jul 2024 14:12:19 -0700 Subject: [PATCH 05/41] Move a few "Supported Constructs" sections into spec. The only substantive change is an update to "Comments" to note that many formats of error suppression comments are allowed. --- docs/reference/stubs.rst | 140 ------------------------------------- docs/spec/distributing.rst | 125 +++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+), 140 deletions(-) diff --git a/docs/reference/stubs.rst b/docs/reference/stubs.rst index 21d9f15aa..926396cb8 100644 --- a/docs/reference/stubs.rst +++ b/docs/reference/stubs.rst @@ -7,21 +7,6 @@ Type Stubs Supported Constructs ==================== -Comments --------- - -Standard Python comments are accepted everywhere Python syntax allows them. - -Two kinds of structured comments are accepted: - -* A ``# type: X`` comment at the end of a line that defines a variable, - declaring that the variable has type ``X``. However, :pep:`526`-style - variable annotations are preferred over type comments. -* A ``# type: ignore`` comment at the end of any line, which suppresses all type - errors in that line. The type checker mypy supports suppressing certain - type errors by using ``# type: ignore[error-type]``. This is not supported - by other type checkers and should not be used in stubs. - Imports ------- @@ -75,33 +60,6 @@ specified in ``__all__`` are imported:: Type checkers support cyclic imports in stub files. -Built-in Generics ------------------ - -:pep:`585` built-in generics are supported and should be used instead -of the corresponding types from ``typing``:: - - from collections import defaultdict - - def foo(t: type[MyClass]) -> list[int]: ... - x: defaultdict[int] - -Using imports from ``collections.abc`` instead of ``typing`` is -generally possible and recommended:: - - from collections.abc import Iterable - - def foo(iter: Iterable[int]) -> None: ... - -Unions ------- - -Declaring unions with the shorthand syntax or ``Union`` and ``Optional`` is -supported by all type checkers:: - - def foo(x: int | str) -> int | None: ... # recommended - def foo(x: Union[int, str]) -> Optional[int]: ... # ok - Module Level Attributes ----------------------- @@ -252,42 +210,6 @@ type stubs:: def foo(x: float) -> int: ... def foo(x: str | float) -> Any: ... -Aliases and NewType -------------------- - -Type checkers should accept module-level type aliases, optionally using -``TypeAlias`` (:pep:`613`), e.g.:: - - _IntList = list[int] - _StrList: TypeAlias = list[str] - -Type checkers should also accept regular module-level or class-level aliases, -e.g.:: - - def a() -> None: ... - b = a - - class C: - def f(self) -> int: ... - g = f - -A type alias may contain type variables. As per :pep:`484`, -all type variables must be substituted when the alias is used:: - - _K = TypeVar("_K") - _V = TypeVar("_V") - _MyMap: TypeAlias = dict[str, dict[_K, _V]] - - # either concrete types or other type variables can be substituted - def f(x: _MyMap[str, _V]) -> _V: ... - # explicitly substitute in Any rather than using a bare alias - def g(x: _MyMap[Any, Any]) -> Any: ... - -Otherwise, type variables in aliases follow the same rules as type variables in -generic class definitions. - -``typing.NewType`` is also supported in stubs. - Decorators ---------- @@ -314,68 +236,6 @@ the stub definition should be:: from contextlib import AbstractContextManager def f() -> AbstractContextManager[int]: ... -Version and Platform Checks ---------------------------- - -Type stubs for libraries that support multiple Python versions can use version -checks to supply version-specific type hints. Type stubs for different Python -versions should still conform to the most recent supported Python version's -syntax, as explain in the Syntax_ section above. - -Version checks are if-statements that use ``sys.version_info`` to determine the -current Python version. Version checks should only check against the ``major`` and -``minor`` parts of ``sys.version_info``. Type checkers are only required to -support the tuple-based version check syntax:: - - if sys.version_info >= (3,): - # Python 3-specific type hints. This tuple-based syntax is recommended. - else: - # Python 2-specific type hints. - - if sys.version_info >= (3, 5): - # Specific minor version features can be easily checked with tuples. - - if sys.version_info < (3,): - # This is only necessary when a feature has no Python 3 equivalent. - -Type stubs should avoid checking against ``sys.version_info.major`` -directly and should not use comparison operators other than ``<`` and ``>=``. - -No:: - - if sys.version_info.major >= 3: - # Semantically the same as the first tuple check. - - if sys.version_info[0] >= 3: - # This is also the same. - - if sys.version_info <= (2, 7): - # This does not work because e.g. (2, 7, 1) > (2, 7). - -Some type stubs also may need to specify type hints for different platforms. -Platform checks must be equality comparisons between ``sys.platform`` and the name -of a platform as a string literal: - -Yes:: - - if sys.platform == 'win32': - # Windows-specific type hints. - else: - # Posix-specific type hints. - -No:: - - if sys.platform.startswith('linux'): - # Not necessary since Python 3.3. - - if sys.platform in ['linux', 'cygwin', 'darwin']: - # Only '==' or '!=' should be used in platform checks. - -Version and platform comparisons can be chained using the ``and`` and ``or`` -operators:: - - if sys.platform == 'linux' and (sys.version_info < (3,) or sys,version_info >= (3, 7)): ... - Enums ----- diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index fa851695c..f5b4c14bf 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -87,15 +87,48 @@ for a library supporting Python 3.7+:: Comments """""""" +Standard Python comments are accepted everywhere Python syntax allows them. + +Two kinds of structured comments are accepted: + +* A ``# type: X`` comment at the end of a line that defines a variable, + declaring that the variable has type ``X``. However, :pep:`526`-style + variable annotations are preferred over type comments. +* An error suppression comment at the end of any line. Common error suppression + formats are ``# type: ignore``, ``# type: ignore[error-class]``, and + ``# pyright: ignore[error-class]``. Type checkers may ignore error + suppressions that they don't support but should not error on them. + Imports """"""" Built-in Generics """"""""""""""""" +:pep:`585` built-in generics are supported and should be used instead +of the corresponding types from ``typing``:: + + from collections import defaultdict + + def foo(t: type[MyClass]) -> list[int]: ... + x: defaultdict[int] + +Using imports from ``collections.abc`` instead of ``typing`` is +generally possible and recommended:: + + from collections.abc import Iterable + + def foo(iter: Iterable[int]) -> None: ... + Unions """""" +Declaring unions with the shorthand syntax or ``Union`` and ``Optional`` is +supported by all type checkers:: + + def foo(x: int | str) -> int | None: ... # recommended + def foo(x: Union[int, str]) -> Optional[int]: ... # ok + Module Level Attributes """"""""""""""""""""""" @@ -108,12 +141,104 @@ Functions and Methods Aliases and NewType """"""""""""""""""" +Type checkers should accept module-level type aliases, optionally using +``TypeAlias`` (:pep:`613`), e.g.:: + + _IntList = list[int] + _StrList: TypeAlias = list[str] + +Type checkers should also accept regular module-level or class-level aliases, +e.g.:: + + def a() -> None: ... + b = a + + class C: + def f(self) -> int: ... + g = f + +A type alias may contain type variables. As per :pep:`484`, +all type variables must be substituted when the alias is used:: + + _K = TypeVar("_K") + _V = TypeVar("_V") + _MyMap: TypeAlias = dict[str, dict[_K, _V]] + + # either concrete types or other type variables can be substituted + def f(x: _MyMap[str, _V]) -> _V: ... + # explicitly substitute in Any rather than using a bare alias + def g(x: _MyMap[Any, Any]) -> Any: ... + +Otherwise, type variables in aliases follow the same rules as type variables in +generic class definitions. + +``typing.NewType`` is also supported in stubs. + Decorators """""""""" Version and Platform Checks """"""""""""""""""""""""""" +Stub files for libraries that support multiple Python versions can use version +checks to supply version-specific type hints. Stubs for different Python +versions should still conform to the most recent supported Python version's +syntax, as explain in the :ref:`Syntax` section above. + +Version checks are if-statements that use ``sys.version_info`` to determine the +current Python version. Version checks should only check against the ``major`` and +``minor`` parts of ``sys.version_info``. Type checkers are only required to +support the tuple-based version check syntax:: + + if sys.version_info >= (3,): + # Python 3-specific type hints. This tuple-based syntax is recommended. + else: + # Python 2-specific type hints. + + if sys.version_info >= (3, 5): + # Specific minor version features can be easily checked with tuples. + + if sys.version_info < (3,): + # This is only necessary when a feature has no Python 3 equivalent. + +Stubs should avoid checking against ``sys.version_info.major`` directly and +should not use comparison operators other than ``<`` and ``>=``. + +No:: + + if sys.version_info.major >= 3: + # Semantically the same as the first tuple check. + + if sys.version_info[0] >= 3: + # This is also the same. + + if sys.version_info <= (2, 7): + # This does not work because e.g. (2, 7, 1) > (2, 7). + +Some stubs also may need to specify type hints for different platforms. Platform +checks must be equality comparisons between ``sys.platform`` and the name of a +platform as a string literal: + +Yes:: + + if sys.platform == 'win32': + # Windows-specific type hints. + else: + # Posix-specific type hints. + +No:: + + if sys.platform.startswith('linux'): + # Not necessary since Python 3.3. + + if sys.platform in ['linux', 'cygwin', 'darwin']: + # Only '==' or '!=' should be used in platform checks. + +Version and platform comparisons can be chained using the ``and`` and ``or`` +operators:: + + if sys.platform == 'linux' and (sys.version_info < (3,) or sys,version_info >= (3, 7)): ... + Enums """"" From be35ffe5f7d9e40bd6eeb60d09808c5e03bc76bf Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Fri, 12 Jul 2024 03:35:24 -0700 Subject: [PATCH 06/41] Migrated "Imports" and "Module Level Attributes" supported constructs. Added `x: Final = ` to module-level attributes section. No other changes. --- docs/reference/stubs.rst | 74 -------------------------------------- docs/spec/distributing.rst | 74 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 74 deletions(-) diff --git a/docs/reference/stubs.rst b/docs/reference/stubs.rst index 926396cb8..bb814072d 100644 --- a/docs/reference/stubs.rst +++ b/docs/reference/stubs.rst @@ -7,80 +7,6 @@ Type Stubs Supported Constructs ==================== -Imports -------- - -Type stubs distinguish between imports that are re-exported and those -that are only used internally. Imports are re-exported if they use one of these -forms (:pep:`484`): - -* ``import X as X`` -* ``from Y import X as X`` -* ``from Y import *`` - -Here are some examples of imports that make names available for internal use in -a stub but do not re-export them:: - - import X - from Y import X - from Y import X as OtherX - -Type aliases can be used to re-export an import under a different name:: - - from foo import bar as _bar - new_bar = _bar # "bar" gets re-exported with the name "new_bar" - -Sub-modules are always exported when they are imported in a module. -For example, consider the following file structure:: - - foo/ - __init__.pyi - bar.pyi - -Then ``foo`` will export ``bar`` when one of the following constructs is used in -``__init__.pyi``:: - - from . import bar - from .bar import Bar - -Stubs support customizing star import semantics by defining a module-level -variable called ``__all__``. In stubs, this must be a string list literal. -Other types are not supported. Neither is the dynamic creation of this -variable (for example by concatenation). - -By default, ``from foo import *`` imports all names in ``foo`` that -do not begin with an underscore. When ``__all__`` is defined, only those names -specified in ``__all__`` are imported:: - - __all__ = ['public_attr', '_private_looking_public_attr'] - - public_attr: int - _private_looking_public_attr: int - private_attr: int - -Type checkers support cyclic imports in stub files. - -Module Level Attributes ------------------------ - -Module level variables and constants can be annotated using either -type comments or variable annotation syntax:: - - x: int # recommended - x: int = 0 - x = 0 # type: int - x = ... # type: int - -The type of a variable is unspecified when the variable is unannotated or -when the annotation -and the assigned value disagree. As an exception, the ellipsis literal can -stand in for any type:: - - x = 0 # type is unspecified - x = ... # type is unspecified - x: int = "" # type is unspecified - x: int = ... # type is int - Classes ------- diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index f5b4c14bf..caab78cbc 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -102,6 +102,56 @@ Two kinds of structured comments are accepted: Imports """"""" +Stub files distinguish between imports that are re-exported and those +that are only used internally. Imports are re-exported if they use one of these +forms (:pep:`484`): + +* ``import X as X`` +* ``from Y import X as X`` +* ``from Y import *`` + +Here are some examples of imports that make names available for internal use in +a stub but do not re-export them:: + + import X + from Y import X + from Y import X as OtherX + +Type aliases can be used to re-export an import under a different name:: + + from foo import bar as _bar + new_bar = _bar # "bar" gets re-exported with the name "new_bar" + +Sub-modules are always exported when they are imported in a module. +For example, consider the following file structure:: + + foo/ + __init__.pyi + bar.pyi + +Then ``foo`` will export ``bar`` when one of the following constructs is used in +``__init__.pyi``:: + + from . import bar + from .bar import Bar + +Stubs support customizing star import semantics by defining a module-level +variable called ``__all__``. In stubs, this must be a string list literal. +Other types are not supported. Neither is the dynamic creation of this +variable (for example by concatenation). + +By default, ``from foo import *`` imports all names in ``foo`` that +do not begin with an underscore. When ``__all__`` is defined, only those names +specified in ``__all__`` are imported:: + + __all__ = ['public_attr', '_private_looking_public_attr'] + + public_attr: int + _private_looking_public_attr: int + private_attr: int + +Type checkers support cyclic imports in stub files. + Built-in Generics """"""""""""""""" @@ -132,6 +182,30 @@ supported by all type checkers:: Module Level Attributes """"""""""""""""""""""" +Module level variables and constants can be annotated using either +type comments or variable annotation syntax:: + + x: int # recommended + x: int = 0 + x = 0 # type: int + x = ... # type: int + +The ellipsis literal can stand in for any value:: + + x: int = ... # type is int + +A variable annotated as ``Final`` and assigned a literal value has the +corresponding ``Literal`` type:: + + x: Final = 0 # type is Literal[0] + +In all other cases, the type of a variable is unspecified when the variable is +unannotated or when the annotation and the assigned value disagree:: + + x = 0 # type is unspecified + x = ... # type is unspecified + x: int = "" # type is unspecified + Classes """"""" From 00766db71634f61c0dd2a41a71893ec04d552b66 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Sat, 13 Jul 2024 02:00:43 -0700 Subject: [PATCH 07/41] Migrate "Enums" supported construct section. Replaces outdated info with a link to the enums section of the spec. --- docs/reference/stubs.rst | 40 -------------------------------------- docs/spec/distributing.rst | 7 +++++++ docs/spec/enums.rst | 1 + 3 files changed, 8 insertions(+), 40 deletions(-) diff --git a/docs/reference/stubs.rst b/docs/reference/stubs.rst index bb814072d..32b486d3d 100644 --- a/docs/reference/stubs.rst +++ b/docs/reference/stubs.rst @@ -161,43 +161,3 @@ the stub definition should be:: from contextlib import AbstractContextManager def f() -> AbstractContextManager[int]: ... - -Enums ------ - -Enum classes are supported in stubs, regardless of the Python version targeted by -the stubs. - -Enum members may be specified just like other forms of assignments, for example as -``x: int``, ``x = 0``, or ``x = ...``. The first syntax is preferred because it -allows type checkers to correctly type the ``.value`` attribute of enum members, -without providing unnecessary information like the runtime value of the enum member. - -Additional properties on enum members should be specified with ``@property``, so they -do not get interpreted by type checkers as enum members. - -Yes:: - - from enum import Enum - - class Color(Enum): - RED: int - BLUE: int - @property - def rgb_value(self) -> int: ... - - class Color(Enum): - # discouraged; type checkers will not understand that Color.RED.value is an int - RED = ... - BLUE = ... - @property - def rgb_value(self) -> int: ... - -No:: - - from enum import Enum - - class Color(Enum): - RED: int - BLUE: int - rgb_value: int # no way for type checkers to know that this is not an enum member diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index caab78cbc..05d097382 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -316,6 +316,13 @@ operators:: Enums """"" +Enum classes are supported in stubs, regardless of the Python version targeted by +the stubs. + +Enum members should be specified with an unannotated assignment, for example as +``x = 0`` or ``x = ...``. Non-member attributes should be specified with a type +annotation and no assigned value. See :ref:`enum-members` for details. + The Typeshed Project ^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/spec/enums.rst b/docs/spec/enums.rst index a3fa1ccd8..39e2598b0 100644 --- a/docs/spec/enums.rst +++ b/docs/spec/enums.rst @@ -94,6 +94,7 @@ implicitly "final". Type checkers should enforce this:: class ExtendedShape(Shape): # Type checker error: Shape is implicitly final TRIANGLE = 3 +.. _enum-members: Defining Members ---------------- From c524da444986288ce14c2fd9fd0a5e6781b69c2f Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Fri, 19 Jul 2024 18:55:02 -0700 Subject: [PATCH 08/41] Move "Classes" and "Decorators" supported constructs sections. Added an example of an inner class to "Classes." Removed mention of asyncio.coroutine from "Decorators" as it's very deprecated. Added functions decorated with `@dataclass_transform` to list of decorators that type checkers should recognize. --- docs/reference/stubs.rst | 66 -------------------------------------- docs/spec/distributing.rst | 63 ++++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 66 deletions(-) diff --git a/docs/reference/stubs.rst b/docs/reference/stubs.rst index 32b486d3d..6cda2abd1 100644 --- a/docs/reference/stubs.rst +++ b/docs/reference/stubs.rst @@ -7,46 +7,6 @@ Type Stubs Supported Constructs ==================== -Classes -------- - -Class definition syntax follows general Python syntax, but type checkers -are only expected to understand the following constructs in class bodies: - -* The ellipsis literal ``...`` is ignored and used for empty - class bodies. Using ``pass`` in class bodies is undefined. -* Instance attributes follow the same rules as module level attributes - (see above). -* Method definitions (see below) and properties. -* Method aliases. -* Inner class definitions. - -More complex statements don't need to be supported:: - - class Simple: ... - - class Complex(Base): - read_write: int - @property - def read_only(self) -> int: ... - def do_stuff(self, y: str) -> None: ... - doStuff = do_stuff - -The type of generic classes can be narrowed by annotating the ``self`` -argument of the ``__init__`` method:: - - class Foo(Generic[_T]): - @overload - def __init__(self: Foo[str], type: Literal["s"]) -> None: ... - @overload - def __init__(self: Foo[int], type: Literal["i"]) -> None: ... - @overload - def __init__(self, type: str) -> None: ... - -The class must match the class in which it is declared. Using other classes, -including sub or super classes, will not work. In addition, the ``self`` -annotation cannot contain type variables. - .. _supported-functions: Functions and Methods @@ -135,29 +95,3 @@ type stubs:: @overload def foo(x: float) -> int: ... def foo(x: str | float) -> Any: ... - -Decorators ----------- - -Type stubs may only use decorators defined in the ``typing`` module, plus a -fixed set of additional ones: - -* ``classmethod`` -* ``staticmethod`` -* ``property`` (including ``.setter``) -* ``abc.abstractmethod`` -* ``dataclasses.dataclass`` -* ``asyncio.coroutine`` (although ``async`` should be used instead) - -The behavior of other decorators should instead be incorporated into the types. -For example, for the following function:: - - import contextlib - @contextlib.contextmanager - def f(): - yield 42 - -the stub definition should be:: - - from contextlib import AbstractContextManager - def f() -> AbstractContextManager[int]: ... diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 05d097382..3e37f9e98 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -209,6 +209,46 @@ unannotated or when the annotation and the assigned value disagree:: Classes """"""" +Class definition syntax follows general Python syntax, but type checkers +are only expected to understand the following constructs in class bodies: + +* The ellipsis literal ``...`` is ignored and used for empty + class bodies. Using ``pass`` in class bodies is undefined. +* Instance attributes follow the same rules as module level attributes + (see above). +* Method definitions (see below) and properties. +* Method aliases. +* Inner class definitions. + +Yes:: + + class Simple: ... + + class Complex(Base): + read_write: int + @property + def read_only(self) -> int: ... + def do_stuff(self, y: str) -> None: ... + doStuff = do_stuff + class Inner: ... + +More complex statements don't need to be supported. + +The type of generic classes can be narrowed by annotating the ``self`` +argument of the ``__init__`` method:: + + class Foo(Generic[_T]): + @overload + def __init__(self: Foo[str], type: Literal["s"]) -> None: ... + @overload + def __init__(self: Foo[int], type: Literal["i"]) -> None: ... + @overload + def __init__(self, type: str) -> None: ... + +The class must match the class in which it is declared. Using other classes, +including sub or super classes, will not work. In addition, the ``self`` +annotation cannot contain type variables. + Functions and Methods """"""""""""""""""""" @@ -251,6 +291,29 @@ generic class definitions. Decorators """""""""" +Type checkers are expected to understand the effects of all decorators defined +in the ``typing`` module, plus these additional ones: + + * ``classmethod`` + * ``staticmethod`` + * ``property`` (including ``.setter``) + * ``abc.abstractmethod`` + * ``dataclasses.dataclass`` + * functions decorated with ``@typing.dataclass_transform`` + +The behavior of other decorators should instead be incorporated into the types. +For example, for the following function:: + + import contextlib + @contextlib.contextmanager + def f(): + yield 42 + +the stub definition should be:: + + from contextlib import AbstractContextManager + def f() -> AbstractContextManager[int]: ... + Version and Platform Checks """"""""""""""""""""""""""" From abd198f3aba4d63c0fd11767be8037108cc1c324 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Sat, 20 Jul 2024 02:23:36 -0700 Subject: [PATCH 09/41] Port "Functions and Methods" supported constructs section. Removed mention of individual supported/unsupported features from this section. Where we state that all typing features from the latest released version are supported, added a caveat with a link to the typeshed feature tracker. --- docs/reference/stubs.rst | 97 -------------------------------------- docs/spec/distributing.rst | 80 +++++++++++++++++++++++++++++-- 2 files changed, 76 insertions(+), 101 deletions(-) delete mode 100644 docs/reference/stubs.rst diff --git a/docs/reference/stubs.rst b/docs/reference/stubs.rst deleted file mode 100644 index 6cda2abd1..000000000 --- a/docs/reference/stubs.rst +++ /dev/null @@ -1,97 +0,0 @@ -.. _stubs: - -********** -Type Stubs -********** - -Supported Constructs -==================== - -.. _supported-functions: - -Functions and Methods ---------------------- - -Function and method definition syntax follows general Python syntax. -For backwards compatibility, positional-only parameters can also be marked by -prefixing their name with two underscores (but not suffixing it with two -underscores):: - - # x is positional-only - # y can be used positionally or as keyword argument - # z is keyword-only - def foo(x, /, y, *, z): ... # recommended - def foo(__x, y, *, z): ... # backwards compatible syntax - -If an argument or return type is unannotated, per :pep:`484` its -type is assumed to be ``Any``. It is preferred to leave unknown -types unannotated rather than explicitly marking them as ``Any``, as some -type checkers can optionally warn about unannotated arguments. - -If an argument has a literal or constant default value, it must match the implementation -and the type of the argument (if specified) must match the default value. -Alternatively, ``...`` can be used in place of any default value:: - - # The following arguments all have type Any. - def unannotated(a, b=42, c=...): ... - # The following arguments all have type int. - def annotated(a: int, b: int = 42, c: int = ...): ... - # The following default values are invalid and the types are unspecified. - def invalid(a: int = "", b: Foo = Foo()): ... - -For a class ``C``, the type of the first argument to a classmethod is -assumed to be ``type[C]``, if unannotated. For other non-static methods, -its type is assumed to be ``C``:: - - class Foo: - def do_things(self): ... # self has type Foo - @classmethod - def create_it(cls): ... # cls has type Type[Foo] - @staticmethod - def utility(x): ... # x has type Any - -But:: - - _T = TypeVar("_T") - - class Foo: - def do_things(self: _T) -> _T: ... # self has type _T - @classmethod - def create_it(cls: _T) -> _T: ... # cls has type _T - -:pep:`612` parameter specification variables (``ParamSpec``) -are supported in argument and return types:: - - _P = ParamSpec("_P") - _R = TypeVar("_R") - - def foo(cb: Callable[_P, _R], *args: _P.args, **kwargs: _P.kwargs) -> _R: ... - -However, ``Concatenate`` from PEP 612 is not yet supported; nor is using -a ``ParamSpec`` to parameterize a generic class. - -:pep:`647` type guards are supported. - -Using a function or method body other than the ellipsis literal is currently -unspecified. Stub authors may experiment with other bodies, but it is up to -individual type checkers how to interpret them:: - - def foo(): ... # compatible - def bar(): pass # behavior undefined - -All variants of overloaded functions and methods must have an ``@overload`` -decorator:: - - @overload - def foo(x: str) -> str: ... - @overload - def foo(x: float) -> int: ... - -The following (which would be used in the implementation) is wrong in -type stubs:: - - @overload - def foo(x: str) -> str: ... - @overload - def foo(x: float) -> int: ... - def foo(x: str | float) -> Any: ... diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 3e37f9e98..0b8dcd99c 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -72,10 +72,12 @@ authors can safely use these constructs. If a construct is marked as error. Linters should usually flag those constructs. Stub authors should avoid using them to ensure compatibility across type checkers. -Unless otherwise mentioned, stub files support all features from the ``typing`` -module of the latest released Python version. If a stub uses typing features -from a later Python version than what the implementation supports, these -features can be imported from ``typing_extensions`` instead of ``typing``. +The `typeshed feature tracker `_ tracks features from the ``typing`` module that are +not yet supported by all major type checkers. Unless otherwise noted in this +tracker, stub files support all features from the ``typing`` module of the +latest released Python version. If a stub uses typing features from a later +Python version than what the implementation supports, these features can be +imported from ``typing_extensions`` instead of ``typing`` For example, a stub could use ``Literal``, introduced in Python 3.8, for a library supporting Python 3.7+:: @@ -252,6 +254,76 @@ annotation cannot contain type variables. Functions and Methods """"""""""""""""""""" +Function and method definition syntax follows general Python syntax. +For backwards compatibility, positional-only parameters can also be marked by +prefixing their name with two underscores (but not suffixing it with two +underscores):: + + # x is positional-only + # y can be used positionally or as keyword argument + # z is keyword-only + def foo(x, /, y, *, z): ... # recommended + def foo(__x, y, *, z): ... # backwards compatible syntax + +If an argument or return type is unannotated, per :pep:`484` its +type is assumed to be ``Any``. It is preferred to leave unknown +types unannotated rather than explicitly marking them as ``Any``, as some +type checkers can optionally warn about unannotated arguments. + +If an argument has a literal or constant default value, it must match the implementation +and the type of the argument (if specified) must match the default value. +Alternatively, ``...`` can be used in place of any default value:: + + # The following arguments all have type Any. + def unannotated(a, b=42, c=...): ... + # The following arguments all have type int. + def annotated(a: int, b: int = 42, c: int = ...): ... + # The following default values are invalid and the types are unspecified. + def invalid(a: int = "", b: Foo = Foo()): ... + +For a class ``C``, the type of the first argument to a classmethod is +assumed to be ``type[C]``, if unannotated. For other non-static methods, +its type is assumed to be ``C``:: + + class Foo: + def do_things(self): ... # self has type Foo + @classmethod + def create_it(cls): ... # cls has type type[Foo] + @staticmethod + def utility(x): ... # x has type Any + +But:: + + _T = TypeVar("_T") + + class Foo: + def do_things(self: _T) -> _T: ... # self has type _T + @classmethod + def create_it(cls: _T) -> _T: ... # cls has type _T + +Using a function or method body other than the ellipsis literal is currently +unspecified. Stub authors may experiment with other bodies, but it is up to +individual type checkers how to interpret them:: + + def foo(): ... # compatible + def bar(): pass # behavior undefined + +All variants of overloaded functions and methods must have an ``@overload`` +decorator:: + + @overload + def foo(x: str) -> str: ... + @overload + def foo(x: float) -> int: ... + +The following (which would be used in the implementation) is wrong in stubs:: + + @overload + def foo(x: str) -> str: ... + @overload + def foo(x: float) -> int: ... + def foo(x: str | float) -> Any: ... + Aliases and NewType """"""""""""""""""" From e0b16d9f5122bfe3148244ca074665148491dc0c Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Sat, 20 Jul 2024 05:26:05 -0700 Subject: [PATCH 10/41] Adds an "Import Conventions" section (deduplication). --- docs/guides/libraries.rst | 12 +++------ docs/spec/distributing.rst | 52 +++++++++++++++++--------------------- 2 files changed, 27 insertions(+), 37 deletions(-) diff --git a/docs/guides/libraries.rst b/docs/guides/libraries.rst index 417c86b47..7d13e98a8 100644 --- a/docs/guides/libraries.rst +++ b/docs/guides/libraries.rst @@ -211,13 +211,8 @@ to determine which symbols are visible outside of the package. - Symbols whose names begin with an underscore (but are not dunder names) are considered private. -- Imported symbols are considered private by default. If they use the - ``import A as A`` (a redundant module alias), ``from X import A as A`` (a - redundant symbol alias), or ``from . import A`` forms, symbol ``A`` is - not private unless the name begins with an underscore. If a file - ``__init__.py`` uses form ``from .A import X``, symbol ``A`` is treated - likewise. If a wildcard import (of the form ``from X import *``) is - used, all symbols referenced by the wildcard are not private. +- Imported symbols are considered private by default. A fixed set of + :ref:`import forms ` re-export imported symbols. - A module can expose an ``__all__`` symbol at the module level that provides a list of names that are considered part of the interface. This overrides all other rules above, allowing imported symbols or @@ -226,7 +221,8 @@ to determine which symbols are visible outside of the package. - Local variables within a function (including nested functions) are always considered private. -The following idioms are supported for defining the values contained +In stub files, ``__all__`` must be a string list literal. In ``.py`` files, +the following idioms are supported for defining the values contained within ``__all__``. These restrictions allow type checkers to statically determine the value of ``__all__``. diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 0b8dcd99c..cec9feb9b 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -105,46 +105,20 @@ Imports """"""" Stub files distinguish between imports that are re-exported and those -that are only used internally. Imports are re-exported if they use one of these -forms (:pep:`484`): - -* ``import X as X`` -* ``from Y import X as X`` -* ``from Y import *`` - -Here are some examples of imports that make names available for internal use in -a stub but do not re-export them:: - - import X - from Y import X - from Y import X as OtherX +that are only used internally. See :ref:`import-conventions`. Type aliases can be used to re-export an import under a different name:: from foo import bar as _bar new_bar = _bar # "bar" gets re-exported with the name "new_bar" -Sub-modules are always exported when they are imported in a module. -For example, consider the following file structure:: - - foo/ - __init__.pyi - bar.pyi - -Then ``foo`` will export ``bar`` when one of the following constructs is used in -``__init__.pyi``:: - - from . import bar - from .bar import Bar - Stubs support customizing star import semantics by defining a module-level variable called ``__all__``. In stubs, this must be a string list literal. Other types are not supported. Neither is the dynamic creation of this variable (for example by concatenation). -By default, ``from foo import *`` imports all names in ``foo`` that -do not begin with an underscore. When ``__all__`` is defined, only those names -specified in ``__all__`` are imported:: +When ``__all__`` is defined, exactly those names specified in ``__all__`` are +imported:: __all__ = ['public_attr', '_private_looking_public_attr'] @@ -647,3 +621,23 @@ of that Python version. This can be queried e.g. ``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is also recommended that the type checker allow for the user to point to a particular Python binary, in case it is not in the path. + +.. _import-conventions: + +Import Conventions +------------------ + +By convention, certain import forms indicate to type checkers that an imported +symbol is re-exported and should be considered part of the importing module's +public interface. All other imported symbols are considered private by default. + +The following import forms re-export symbols: + +* ``import X as X`` (a redundant module alias): re-exports ``X``. +* ``from Y import X as X`` (a redundant symbol alias): re-exports ``X``. +* ``from Y import *``: re-exports all symbols in ``Y`` that do not begin with + an underscore. +* ``from . import bar`` in an ``__init__`` module: re-exports ``bar`` if it does + not begin with an underscore. +* ``from .bar import Bar`` in an ``__init__`` module: re-exports ``Bar`` if it + does not begin with an underscore. From b3e9b9c422cf78dadc0e7cefcb640c34d62f0344 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 00:45:44 -0700 Subject: [PATCH 11/41] Remove missed reference to deleted stubs doc. --- docs/reference/index.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/index.rst b/docs/reference/index.rst index 010f3b498..720e21277 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -8,7 +8,6 @@ Type System Reference generics protocols - stubs best_practices quality typing Module Documentation From 31c52c0717234f145e1ec0b14fd96c9f0448796a Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 00:50:27 -0700 Subject: [PATCH 12/41] Fix broken refs. --- docs/spec/distributing.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index cec9feb9b..0169731d6 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -24,7 +24,7 @@ Python packages and modules. Stub files serve multiple purposes: API of a package, without including the implementation or private members. -Stub files that only use the constructs described in :ref:`Supported Constructs` +Stub files that only use the constructs described in :ref:`stub-file-supported-constructs` below should work with all type checkers that conform to this specification. A conformant type checker will parse a stub that only uses such constructs without error and will not interpret any construct in a contradictory manner. However, @@ -60,8 +60,9 @@ references can be used. Type checkers should have a configurable search path for stub files. If a stub file is found, the type checker should not read the corresponding "real" module. -See :ref:`Import resolution ordering` for more information. +See :ref:`mro` for more information. +.. _stub-file-supported-constructs: Supported Constructs ^^^^^^^^^^^^^^^^^^^^ @@ -366,7 +367,7 @@ Version and Platform Checks Stub files for libraries that support multiple Python versions can use version checks to supply version-specific type hints. Stubs for different Python versions should still conform to the most recent supported Python version's -syntax, as explain in the :ref:`Syntax` section above. +syntax, as explain in the :ref:`stub-file-syntax` section above. Version checks are if-statements that use ``sys.version_info`` to determine the current Python version. Version checks should only check against the ``major`` and From 90170aa7c56e7fa325826cf1a09049bb0f930708 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 23:37:15 -0700 Subject: [PATCH 13/41] Formatting fix to docs/guides/writing_stubs.rst Co-authored-by: Sebastian Rittau --- docs/guides/writing_stubs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 0af304480..df37586be 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -174,7 +174,7 @@ Stub-Only Objects Definitions that do not exist at runtime may be included in stubs to aid in expressing types. Sometimes, it is desirable to make a stub-only class available -to a stub's users - for example, to allow them to type the return value of a +to a stub's users — for example, to allow them to type the return value of a public method for which a library does not provided a usable runtime type:: from typing import Protocol From e9d3211038b095eff0c501365d10c133bcce8ec0 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 23:37:58 -0700 Subject: [PATCH 14/41] Reword paragraph on argument names to be more concise. Co-authored-by: Sebastian Rittau --- docs/guides/writing_stubs.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index df37586be..40e25d7da 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -454,10 +454,9 @@ No:: Functions and Methods --------------------- -Use the same argument names as in the implementation, because -otherwise using keyword arguments will fail. Of course, this -does not apply to positional-only arguments marked with the historical double -underscore convention. +For keyword-only and positional-or-keyword arguments, use the same +argument names as in the implementation, because otherwise using +keyword arguments will fail. Use the ellipsis literal ``...`` in place of actual default argument values. Use an explicit ``X | None`` annotation instead of From 4c7fee21721ebfbb7a14cb1f7de81f538027ff2e Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 23:40:09 -0700 Subject: [PATCH 15/41] Wording clarification in docs/guides/writing_stubs.rst Co-authored-by: Sebastian Rittau --- docs/guides/writing_stubs.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 40e25d7da..1362bc5f3 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -515,8 +515,8 @@ No:: Private Definitions ------------------- -Type variables, type aliases, and other definitions that should not -be used outside the stub should be marked as private by prefixing them +Type variables, type aliases, and other definitions that don't exist at +runtime should be marked as private by prefixing them with an underscore. Yes:: From 60d80d8c87581b67243ccc2fd7db98f5b5693554 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 23:40:42 -0700 Subject: [PATCH 16/41] Modernize code example in docs/guides/writing_stubs.rst Co-authored-by: Sebastian Rittau --- docs/guides/writing_stubs.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 1362bc5f3..27c490d17 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -522,12 +522,12 @@ with an underscore. Yes:: _T = TypeVar("_T") - _DictList = Dict[str, List[Optional[int]] + _DictList: TypeAlias = dict[str, list[int | None]] No:: T = TypeVar("T") - DictList = Dict[str, List[Optional[int]]] + DictList: TypeAlias = dict[str, list[int | None]] Language Features ----------------- From a2a9d72e4eea070fdafcf2d7ca7c712be688a409 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 23:41:13 -0700 Subject: [PATCH 17/41] Typo fix in docs/spec/distributing.rst Co-authored-by: Sebastian Rittau --- docs/spec/distributing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 0169731d6..e0cc329a1 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -51,7 +51,7 @@ to maintain compatibility with all type checkers. For example, Python 3.7 added the ``async`` keyword (see :pep:`492`). Stub authors should use it to mark coroutines, even if the implementation still uses the ``@coroutine`` decorator. On the other hand, stubs should not use -the ``type`` soft keyword from :pep:`695`, introduced in Python 3.12, util +the ``type`` soft keyword from :pep:`695`, introduced in Python 3.12, until Python 3.11 reaches end-of-life in October 2027. Stubs are treated as if ``from __future__ import annotations`` is enabled. In From 0c3bd2cd7d4c2c0831856e72793b7a5176bc4640 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 23:41:31 -0700 Subject: [PATCH 18/41] Typo fix in docs/spec/distributing.rst Co-authored-by: Sebastian Rittau --- docs/spec/distributing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index e0cc329a1..5bc02e029 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -367,7 +367,7 @@ Version and Platform Checks Stub files for libraries that support multiple Python versions can use version checks to supply version-specific type hints. Stubs for different Python versions should still conform to the most recent supported Python version's -syntax, as explain in the :ref:`stub-file-syntax` section above. +syntax, as explained in the :ref:`stub-file-syntax` section above. Version checks are if-statements that use ``sys.version_info`` to determine the current Python version. Version checks should only check against the ``major`` and From da4571e96974d1bb96e260e168c3a857e1dd329a Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 23:45:45 -0700 Subject: [PATCH 19/41] Grammar fixes --- docs/spec/distributing.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 5bc02e029..fb4313059 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -24,9 +24,9 @@ Python packages and modules. Stub files serve multiple purposes: API of a package, without including the implementation or private members. -Stub files that only use the constructs described in :ref:`stub-file-supported-constructs` +Stub files that use only the constructs described in :ref:`stub-file-supported-constructs` below should work with all type checkers that conform to this specification. A -conformant type checker will parse a stub that only uses such constructs without +conformant type checker will parse a stub that uses only such constructs without error and will not interpret any construct in a contradictory manner. However, type checkers are not required to implement checks for all these constructs and can elect to ignore unsupported ones. Additionally, type checkers can support @@ -187,7 +187,7 @@ Classes """"""" Class definition syntax follows general Python syntax, but type checkers -are only expected to understand the following constructs in class bodies: +are expected to understand only the following constructs in class bodies: * The ellipsis literal ``...`` is ignored and used for empty class bodies. Using ``pass`` in class bodies is undefined. From 1e609fef9c9bd9957a37a861df3c2b8416b83036 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 22 Jul 2024 23:47:38 -0700 Subject: [PATCH 20/41] Property deleters should also be understood. --- docs/spec/distributing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index fb4313059..1910c8218 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -343,7 +343,7 @@ in the ``typing`` module, plus these additional ones: * ``classmethod`` * ``staticmethod`` - * ``property`` (including ``.setter``) + * ``property`` (including ``.setter`` and ``.deleter``) * ``abc.abstractmethod`` * ``dataclasses.dataclass`` * functions decorated with ``@typing.dataclass_transform`` From 05434ed21be097da2f8bd73ef8b2aeed8fe66780 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Tue, 23 Jul 2024 22:52:02 -0700 Subject: [PATCH 21/41] Address future notes. --- docs/guides/writing_stubs.rst | 67 +++++++++++++++++++++-------------- 1 file changed, 41 insertions(+), 26 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 27c490d17..258426523 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -123,6 +123,7 @@ It is recommended to exclude: 1. Implementation details, with `multiprocessing/popen_spawn_win32.py `_ as a notable example 2. Modules that are not supposed to be imported, such as ``__main__.py`` 3. Protected modules that start with a single ``_`` char. However, when needed protected modules can still be added (see :ref:`undocumented-objects` section below) +4. Tests Public Interface ---------------- @@ -175,14 +176,16 @@ Stub-Only Objects Definitions that do not exist at runtime may be included in stubs to aid in expressing types. Sometimes, it is desirable to make a stub-only class available to a stub's users — for example, to allow them to type the return value of a -public method for which a library does not provided a usable runtime type:: +public method for which a library does not provided a usable runtime type. Use +the ``typing.type_check_only`` decorator to mark such objects:: - from typing import Protocol + from typing import Protocol, type_check_only - class _Readable(Protocol): + @type_check_only + class Readable(Protocol): def read(self) -> str: ... - def get_reader() -> _Readable: ... + def get_reader() -> Readable: ... Structural Types ---------------- @@ -202,12 +205,25 @@ follow the following guidelines: * Included functions and methods should list all arguments, but the arguments can be left unannotated. -* Do not use ``Any`` to mark unannotated arguments or return values. -* Partial classes should include a ``__getattr__()`` method marked with an - ``# incomplete`` comment (see example below). +* Do not use ``Any`` to mark unannotated or partially annotated values. Leave + function parameters and return values unannotated and mark the function with + an ``# incomplete`` comment. This comment is mainly intended as a reminder for + stub authors, but can be used by tools to flag such functions. In all other + cases, use ``_typeshed.Incomplete`` + (`documentation `_):: + + from _typeshed import Incomplete + + field1: Incomplete + field2: dict[str, Incomplete] + + def foo(x): ... # incomplete + +* Partial classes should include a ``__getattr__()`` method marked with + ``_typeshed.Incomplete`` (see example below). * Partial modules (i.e. modules that are missing some or all classes, functions, or attributes) should include a top-level ``__getattr__()`` - function marked with an ``# incomplete`` comment (see example below). + function marked with ``_typeshed.Incomplete`` (see example below). * Partial packages (i.e. packages that are missing one or more sub-modules) should have a ``__init__.pyi`` stub that is marked as incomplete (see above). A better alternative is to create empty stubs for all sub-modules and @@ -216,18 +232,17 @@ follow the following guidelines: Example of a partial module with a partial class ``Foo`` and a partially annotated function ``bar()``:: - def __getattr__(name: str) -> Any: ... # incomplete + from _typeshed import Incomplete + + def __getattr__(name: str) -> Incomplete: ... class Foo: - def __getattr__(self, name: str) -> Any: ... # incomplete + def __getattr__(self, name: str) -> Incomplete: ... x: int y: str def bar(x: str, y, *, z=...): ... -The ``# incomplete`` comment is mainly intended as a reminder for stub -authors, but can be used by tools to flag such items. - Attribute Access ---------------- @@ -295,15 +310,15 @@ still recommended to separately define known attributes. Constants --------- -When the value of a constant is important, annotate it using ``Literal`` -instead of its type. +When the value of a constant is important, mark it as ``Final`` and assign it +to its value:: Yes:: - TEL_LANDLINE: Literal["landline"] - TEL_MOBILE: Literal["mobile"] - DAY_FLAG: Literal[0x01] - NIGHT_FLAG: Literal[0x02] + TEL_LANDLINE: Final = "landline" + TEL_MOBILE: Final = "mobile" + DAY_FLAG: Final = 0x01 + NIGHT_FLAG: Final = 0x02 No:: @@ -458,20 +473,20 @@ For keyword-only and positional-or-keyword arguments, use the same argument names as in the implementation, because otherwise using keyword arguments will fail. -Use the ellipsis literal ``...`` in place of actual default argument -values. Use an explicit ``X | None`` annotation instead of -a ``None`` default. +For default values, use the literal values of "simple" default values (``None``, +bools, ints, bytes, strings, and floats). Use the ellipsis literal ``...`` in +place of more complex default values. Use an explicit ``X | None`` annotation +when the default is ``None``. Yes:: - def foo(x: int = ...) -> None: ... - def bar(y: str | None = ...) -> None: ... + def foo(x: int = 0) -> None: ... + def bar(y: str | None = None) -> None: ... No:: - def foo(x: int = 0) -> None: ... + def foo(x: X = X()) -> None: ... def bar(y: str = None) -> None: ... - def baz(z: str | None = None) -> None: ... Do not annotate ``self`` and ``cls`` in method definitions, except when referencing a type variable. From 6a8a26ef5f7dff2231f0fca8ed463353742b80e8 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Tue, 23 Jul 2024 22:55:01 -0700 Subject: [PATCH 22/41] Fix silly formatting error. --- docs/guides/writing_stubs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 258426523..609edaf2c 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -311,7 +311,7 @@ Constants --------- When the value of a constant is important, mark it as ``Final`` and assign it -to its value:: +to its value. Yes:: From 10cb9693b347a2f13609ca9b2fb515a908c580b6 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Fri, 26 Jul 2024 19:42:45 -0700 Subject: [PATCH 23/41] Undelete stubs.rst. --- docs/reference/stubs.rst | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 docs/reference/stubs.rst diff --git a/docs/reference/stubs.rst b/docs/reference/stubs.rst new file mode 100644 index 000000000..b0ce56711 --- /dev/null +++ b/docs/reference/stubs.rst @@ -0,0 +1,8 @@ +:orphan: + +********** +Type Stubs +********** + +The contents of this document have been moved to :ref:`stub-files` and +:ref:`writing_stubs`. From 46e133f1527c800666495da4c5467d6a42b38c03 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Sun, 28 Jul 2024 21:18:15 -0700 Subject: [PATCH 24/41] Address review comments. * Reword some sections to focus on type checker behavior. * Move or remove some sections that fit better in the writing_stubs guide or are redundant. --- docs/guides/writing_stubs.rst | 27 ++++++++ docs/spec/distributing.rst | 119 +++++++++------------------------- 2 files changed, 58 insertions(+), 88 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index f59a2607a..1a94ba083 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -581,3 +581,30 @@ No:: from typing import NamedTuple, TypedDict Point = NamedTuple("Point", [('x', float), ('y', float)]) Thing = TypedDict("Thing", {'stuff': str, 'index': int}) + +Built-in Generics +""""""""""""""""" + +:pep:`585` built-in generics are supported and should be used instead +of the corresponding types from ``typing``:: + + from collections import defaultdict + + def foo(t: type[MyClass]) -> list[int]: ... + x: defaultdict[int] + +Using imports from ``collections.abc`` instead of ``typing`` is +generally possible and recommended:: + + from collections.abc import Iterable + + def foo(iter: Iterable[int]) -> None: ... + +Unions +"""""" + +Declaring unions with the shorthand `|` syntax is recommended and supported by +all type checkers:: + + def foo(x: int | str) -> int | None: ... # recommended + def foo(x: Union[int, str]) -> Optional[int]: ... # ok diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 1910c8218..8eb23a2d0 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -24,14 +24,16 @@ Python packages and modules. Stub files serve multiple purposes: API of a package, without including the implementation or private members. -Stub files that use only the constructs described in :ref:`stub-file-supported-constructs` -below should work with all type checkers that conform to this specification. A -conformant type checker will parse a stub that uses only such constructs without -error and will not interpret any construct in a contradictory manner. However, -type checkers are not required to implement checks for all these constructs and -can elect to ignore unsupported ones. Additionally, type checkers can support -constructs not decribed here, and tool authors are encouraged to experiment with -additional features. +Stub files use a subset of the constructs used in Python source files, as +described in :ref:`stub-file-supported-constructs` below. Type checkers should +parse a stub that uses only such constructs without error and not interpret any +construct in a contradictory manner. However, type checkers are not required to +implement checks for all of these constructs and can elect to ignore unsupported +ones. Additionally, type checkers can support constructs not described here. + +Type checkers should have a configurable search path for stub files. If a stub +file is found, the type checker should not read the corresponding "real" module. +See :ref:`mro` for more information. .. _stub-file-syntax: @@ -42,65 +44,43 @@ Stub files are syntactically valid Python files in the earliest Python version that is not yet end-of-life. They use a ``.pyi`` suffix. The Python syntax used by stub files is independent from the Python versions supported by the implementation, and from the Python version the type checker runs under (if -any). Therefore, stub authors should use the latest syntax features available in +any). Therefore, stubs may use the latest syntax features available in the earliest supported version, even if the implementation supports older -versions. Type checker authors are encouraged to support syntax features from -newer versions, although stub authors should not use such features if they wish -to maintain compatibility with all type checkers. +versions. -For example, Python 3.7 added the ``async`` keyword (see :pep:`492`). -Stub authors should use it to mark coroutines, even if the implementation -still uses the ``@coroutine`` decorator. On the other hand, stubs should not use -the ``type`` soft keyword from :pep:`695`, introduced in Python 3.12, until -Python 3.11 reaches end-of-life in October 2027. +For example, Python 3.7 added the ``async`` keyword (see :pep:`492`). Stubs may +use it to mark coroutines, even if the implementation still uses the +``@coroutine`` decorator. On the other hand, the ``type`` soft keyword from +:pep:`695`, introduced in Python 3.12, is not valid in stubs until Python 3.11 +reaches end-of-life in October 2027. Stubs are treated as if ``from __future__ import annotations`` is enabled. In -particular, built-in generics, pipe union syntax (``X | Y``), and forward -references can be used. - -Type checkers should have a configurable search path for stub files. If a stub -file is found, the type checker should not read the corresponding "real" module. -See :ref:`mro` for more information. +particular, forward references do not need to be quoted, and syntax from newer +versions than otherwise supported may be used in annotation contexts. For +example, the pipe union syntax (``X | Y``) introduced in Python 3.10 may be used +in annotation contexts even before Python 3.9 reaches end-of-life. .. _stub-file-supported-constructs: Supported Constructs ^^^^^^^^^^^^^^^^^^^^ -This section lists constructs that type checkers will accept in stub files. Stub -authors can safely use these constructs. If a construct is marked as -"unspecified", type checkers may handle it as they best see fit or report an -error. Linters should usually flag those constructs. Stub authors should avoid -using them to ensure compatibility across type checkers. - -The `typeshed feature tracker `_ tracks features from the ``typing`` module that are -not yet supported by all major type checkers. Unless otherwise noted in this -tracker, stub files support all features from the ``typing`` module of the -latest released Python version. If a stub uses typing features from a later -Python version than what the implementation supports, these features can be -imported from ``typing_extensions`` instead of ``typing`` - -For example, a stub could use ``Literal``, introduced in Python 3.8, -for a library supporting Python 3.7+:: +This section lists constructs that type checkers should accept in stub files. If +a construct is marked as "unspecified", type checkers may handle it as they best +see fit or report an error. - from typing_extensions import Literal +Typing Features +""""""""""""""" - def foo(x: Literal[""]) -> int: ... +Type checkers should support all features from the ``typing`` module of the +latest released Python version. Comments """""""" -Standard Python comments are accepted everywhere Python syntax allows them. - -Two kinds of structured comments are accepted: - -* A ``# type: X`` comment at the end of a line that defines a variable, - declaring that the variable has type ``X``. However, :pep:`526`-style - variable annotations are preferred over type comments. -* An error suppression comment at the end of any line. Common error suppression - formats are ``# type: ignore``, ``# type: ignore[error-class]``, and - ``# pyright: ignore[error-class]``. Type checkers may ignore error - suppressions that they don't support but should not error on them. +Standard Python comments should be accepted everywhere Python syntax allows +them. Type declaration (``# type: X``) and error suppression (``type: ignore``) +comments should be supported. Imports """"""" @@ -129,33 +109,6 @@ imported:: Type checkers support cyclic imports in stub files. -Built-in Generics -""""""""""""""""" - -:pep:`585` built-in generics are supported and should be used instead -of the corresponding types from ``typing``:: - - from collections import defaultdict - - def foo(t: type[MyClass]) -> list[int]: ... - x: defaultdict[int] - -Using imports from ``collections.abc`` instead of ``typing`` is -generally possible and recommended:: - - from collections.abc import Iterable - - def foo(iter: Iterable[int]) -> None: ... - -Unions -"""""" - -Declaring unions with the shorthand syntax or ``Union`` and ``Optional`` is -supported by all type checkers:: - - def foo(x: int | str) -> int | None: ... # recommended - def foo(x: Union[int, str]) -> Optional[int]: ... # ok - Module Level Attributes """"""""""""""""""""""" @@ -423,16 +376,6 @@ operators:: if sys.platform == 'linux' and (sys.version_info < (3,) or sys,version_info >= (3, 7)): ... -Enums -""""" - -Enum classes are supported in stubs, regardless of the Python version targeted by -the stubs. - -Enum members should be specified with an unannotated assignment, for example as -``x = 0`` or ``x = ...``. Non-member attributes should be specified with a type -annotation and no assigned value. See :ref:`enum-members` for details. - The Typeshed Project ^^^^^^^^^^^^^^^^^^^^ From 8a9eda8e0d0ce6e6d15884ad75ff176db92b9dc0 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Tue, 30 Jul 2024 00:28:26 -0700 Subject: [PATCH 25/41] Address more review comments. Cuts redundant sections, moves things to writing_stubs, tries out a new "these should be supported fully" and "these should be supported to at least the described minimum" structure. --- docs/guides/writing_stubs.rst | 43 ++++++- docs/spec/directives.rst | 2 + docs/spec/distributing.rst | 233 +++------------------------------- 3 files changed, 61 insertions(+), 217 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 1a94ba083..33a6640db 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -337,6 +337,45 @@ No:: DAY_FLAG: int NIGHT_FLAG: int +Overloads +--------- + +All variants of overloaded functions and methods must have an ``@overload`` +decorator. Do not include the implementation's final non-`@overload`-decorated +definition. + +Yes:: + + @overload + def foo(x: str) -> str: ... + @overload + def foo(x: float) -> int: ... + +No:: + + @overload + def foo(x: str) -> str: ... + @overload + def foo(x: float) -> int: ... + def foo(x: str | float) -> Any: ... + +Decorators +---------- + +Include only those decorators whose effects type checkers understand, enumerated +:ref:`here `. The behavior of other decorators should instead +be incorporated into the types. For example, for the following function:: + + import contextlib + @contextlib.contextmanager + def f(): + yield 42 + +the stub definition should be:: + + from contextlib import AbstractContextManager + def f() -> AbstractContextManager[int]: ... + Documentation or Implementation ------------------------------- @@ -583,7 +622,7 @@ No:: Thing = TypedDict("Thing", {'stuff': str, 'index': int}) Built-in Generics -""""""""""""""""" +----------------- :pep:`585` built-in generics are supported and should be used instead of the corresponding types from ``typing``:: @@ -601,7 +640,7 @@ generally possible and recommended:: def foo(iter: Iterable[int]) -> None: ... Unions -"""""" +------ Declaring unions with the shorthand `|` syntax is recommended and supported by all type checkers:: diff --git a/docs/spec/directives.rst b/docs/spec/directives.rst index 7fca698a3..62e1b3c4f 100644 --- a/docs/spec/directives.rst +++ b/docs/spec/directives.rst @@ -128,6 +128,8 @@ and return type annotations and treat the function as if it were unannotated. The behavior for the ``no_type_check`` decorator when applied to a class is left undefined by the typing spec at this time. +.. _`version-and-platform-checks`: + Version and platform checking ----------------------------- diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 8eb23a2d0..56bea9876 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -69,45 +69,22 @@ This section lists constructs that type checkers should accept in stub files. If a construct is marked as "unspecified", type checkers may handle it as they best see fit or report an error. -Typing Features -""""""""""""""" - -Type checkers should support all features from the ``typing`` module of the -latest released Python version. - -Comments -"""""""" - -Standard Python comments should be accepted everywhere Python syntax allows -them. Type declaration (``# type: X``) and error suppression (``type: ignore``) -comments should be supported. - -Imports -""""""" - -Stub files distinguish between imports that are re-exported and those -that are only used internally. See :ref:`import-conventions`. - -Type aliases can be used to re-export an import under a different name:: - - from foo import bar as _bar - new_bar = _bar # "bar" gets re-exported with the name "new_bar" - -Stubs support customizing star import semantics by defining a module-level -variable called ``__all__``. In stubs, this must be a string list literal. -Other types are not supported. Neither is the dynamic creation of this -variable (for example by concatenation). - -When ``__all__`` is defined, exactly those names specified in ``__all__`` are -imported:: - - __all__ = ['public_attr', '_private_looking_public_attr'] - - public_attr: int - _private_looking_public_attr: int - private_attr: int - -Type checkers support cyclic imports in stub files. +Type checkers should fully support all features from the ``typing`` module of +the latest released Python version, as well as these constructs: + +* Comments, including type declaration (``# type: X``) and error suppression + (``type: ignore``) comments. +* Import statements, including the standard :ref:`import-conventions` and cyclic + imports. +* Module-level type aliases (e.g., ``X: TypeAlias = int``, + ``Y: TypeAlias = dict[str, _V]``). +* Regular aliases (e.g., ``function_alias = some_function``) at both the module- + and class-level. +* Simple version and platform checks, as described + :ref:`here `. + +The constructs in the following subsections may be supported in a more limited +fashion, as described below. Module Level Attributes """"""""""""""""""""""" @@ -162,41 +139,12 @@ Yes:: doStuff = do_stuff class Inner: ... -More complex statements don't need to be supported. - -The type of generic classes can be narrowed by annotating the ``self`` -argument of the ``__init__`` method:: - - class Foo(Generic[_T]): - @overload - def __init__(self: Foo[str], type: Literal["s"]) -> None: ... - @overload - def __init__(self: Foo[int], type: Literal["i"]) -> None: ... - @overload - def __init__(self, type: str) -> None: ... - -The class must match the class in which it is declared. Using other classes, -including sub or super classes, will not work. In addition, the ``self`` -annotation cannot contain type variables. - Functions and Methods """"""""""""""""""""" Function and method definition syntax follows general Python syntax. -For backwards compatibility, positional-only parameters can also be marked by -prefixing their name with two underscores (but not suffixing it with two -underscores):: - - # x is positional-only - # y can be used positionally or as keyword argument - # z is keyword-only - def foo(x, /, y, *, z): ... # recommended - def foo(__x, y, *, z): ... # backwards compatible syntax - If an argument or return type is unannotated, per :pep:`484` its -type is assumed to be ``Any``. It is preferred to leave unknown -types unannotated rather than explicitly marking them as ``Any``, as some -type checkers can optionally warn about unannotated arguments. +type is assumed to be ``Any``. If an argument has a literal or constant default value, it must match the implementation and the type of the argument (if specified) must match the default value. @@ -209,26 +157,6 @@ Alternatively, ``...`` can be used in place of any default value:: # The following default values are invalid and the types are unspecified. def invalid(a: int = "", b: Foo = Foo()): ... -For a class ``C``, the type of the first argument to a classmethod is -assumed to be ``type[C]``, if unannotated. For other non-static methods, -its type is assumed to be ``C``:: - - class Foo: - def do_things(self): ... # self has type Foo - @classmethod - def create_it(cls): ... # cls has type type[Foo] - @staticmethod - def utility(x): ... # x has type Any - -But:: - - _T = TypeVar("_T") - - class Foo: - def do_things(self: _T) -> _T: ... # self has type _T - @classmethod - def create_it(cls: _T) -> _T: ... # cls has type _T - Using a function or method body other than the ellipsis literal is currently unspecified. Stub authors may experiment with other bodies, but it is up to individual type checkers how to interpret them:: @@ -236,57 +164,7 @@ individual type checkers how to interpret them:: def foo(): ... # compatible def bar(): pass # behavior undefined -All variants of overloaded functions and methods must have an ``@overload`` -decorator:: - - @overload - def foo(x: str) -> str: ... - @overload - def foo(x: float) -> int: ... - -The following (which would be used in the implementation) is wrong in stubs:: - - @overload - def foo(x: str) -> str: ... - @overload - def foo(x: float) -> int: ... - def foo(x: str | float) -> Any: ... - -Aliases and NewType -""""""""""""""""""" - -Type checkers should accept module-level type aliases, optionally using -``TypeAlias`` (:pep:`613`), e.g.:: - - _IntList = list[int] - _StrList: TypeAlias = list[str] - -Type checkers should also accept regular module-level or class-level aliases, -e.g.:: - - def a() -> None: ... - b = a - - class C: - def f(self) -> int: ... - g = f - -A type alias may contain type variables. As per :pep:`484`, -all type variables must be substituted when the alias is used:: - - _K = TypeVar("_K") - _V = TypeVar("_V") - _MyMap: TypeAlias = dict[str, dict[_K, _V]] - - # either concrete types or other type variables can be substituted - def f(x: _MyMap[str, _V]) -> _V: ... - # explicitly substitute in Any rather than using a bare alias - def g(x: _MyMap[Any, Any]) -> Any: ... - -Otherwise, type variables in aliases follow the same rules as type variables in -generic class definitions. - -``typing.NewType`` is also supported in stubs. +.. _stub-decorators: Decorators """""""""" @@ -301,81 +179,6 @@ in the ``typing`` module, plus these additional ones: * ``dataclasses.dataclass`` * functions decorated with ``@typing.dataclass_transform`` -The behavior of other decorators should instead be incorporated into the types. -For example, for the following function:: - - import contextlib - @contextlib.contextmanager - def f(): - yield 42 - -the stub definition should be:: - - from contextlib import AbstractContextManager - def f() -> AbstractContextManager[int]: ... - -Version and Platform Checks -""""""""""""""""""""""""""" - -Stub files for libraries that support multiple Python versions can use version -checks to supply version-specific type hints. Stubs for different Python -versions should still conform to the most recent supported Python version's -syntax, as explained in the :ref:`stub-file-syntax` section above. - -Version checks are if-statements that use ``sys.version_info`` to determine the -current Python version. Version checks should only check against the ``major`` and -``minor`` parts of ``sys.version_info``. Type checkers are only required to -support the tuple-based version check syntax:: - - if sys.version_info >= (3,): - # Python 3-specific type hints. This tuple-based syntax is recommended. - else: - # Python 2-specific type hints. - - if sys.version_info >= (3, 5): - # Specific minor version features can be easily checked with tuples. - - if sys.version_info < (3,): - # This is only necessary when a feature has no Python 3 equivalent. - -Stubs should avoid checking against ``sys.version_info.major`` directly and -should not use comparison operators other than ``<`` and ``>=``. - -No:: - - if sys.version_info.major >= 3: - # Semantically the same as the first tuple check. - - if sys.version_info[0] >= 3: - # This is also the same. - - if sys.version_info <= (2, 7): - # This does not work because e.g. (2, 7, 1) > (2, 7). - -Some stubs also may need to specify type hints for different platforms. Platform -checks must be equality comparisons between ``sys.platform`` and the name of a -platform as a string literal: - -Yes:: - - if sys.platform == 'win32': - # Windows-specific type hints. - else: - # Posix-specific type hints. - -No:: - - if sys.platform.startswith('linux'): - # Not necessary since Python 3.3. - - if sys.platform in ['linux', 'cygwin', 'darwin']: - # Only '==' or '!=' should be used in platform checks. - -Version and platform comparisons can be chained using the ``and`` and ``or`` -operators:: - - if sys.platform == 'linux' and (sys.version_info < (3,) or sys,version_info >= (3, 7)): ... - The Typeshed Project ^^^^^^^^^^^^^^^^^^^^ From 14966d248c5271c7804dba4a1afcfcd5474187ff Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Tue, 30 Jul 2024 00:35:56 -0700 Subject: [PATCH 26/41] Remove no-longer-needed anchor. --- docs/spec/enums.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/spec/enums.rst b/docs/spec/enums.rst index 39e2598b0..429f703b1 100644 --- a/docs/spec/enums.rst +++ b/docs/spec/enums.rst @@ -94,8 +94,6 @@ implicitly "final". Type checkers should enforce this:: class ExtendedShape(Shape): # Type checker error: Shape is implicitly final TRIANGLE = 3 -.. _enum-members: - Defining Members ---------------- From 1ab8a33351d5f2391c335e9fbf0217d8333f150c Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Tue, 30 Jul 2024 00:36:44 -0700 Subject: [PATCH 27/41] And add back an accidentally deleted newline... --- docs/spec/enums.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/spec/enums.rst b/docs/spec/enums.rst index 429f703b1..a3fa1ccd8 100644 --- a/docs/spec/enums.rst +++ b/docs/spec/enums.rst @@ -94,6 +94,7 @@ implicitly "final". Type checkers should enforce this:: class ExtendedShape(Shape): # Type checker error: Shape is implicitly final TRIANGLE = 3 + Defining Members ---------------- From df94e57aee1bf711ac5800abaad611cfe5eff0db Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Tue, 30 Jul 2024 22:16:43 -0700 Subject: [PATCH 28/41] Slight rewording of Decorators advice. --- docs/guides/writing_stubs.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 33a6640db..7000ca045 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -362,9 +362,10 @@ No:: Decorators ---------- -Include only those decorators whose effects type checkers understand, enumerated -:ref:`here `. The behavior of other decorators should instead -be incorporated into the types. For example, for the following function:: +Include only the decorators listed :ref:`here `, whose effects +are understood by all of the major type checkers. The behavior of other +decorators should instead be incorporated into the types. For example, for the +following function:: import contextlib @contextlib.contextmanager From 93ce31d69247f31b9b8c51a9c04d4e8005e3a137 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Tue, 30 Jul 2024 23:51:11 -0700 Subject: [PATCH 29/41] Address more reviewer comments. Moves "Library Interface" into spec, further trims redundant text. --- docs/guides/libraries.rst | 43 +------------- docs/spec/distributing.rst | 119 +++++++++++++++++++++++-------------- docs/spec/literal.rst | 1 + 3 files changed, 76 insertions(+), 87 deletions(-) diff --git a/docs/guides/libraries.rst b/docs/guides/libraries.rst index 7d13e98a8..4b90da498 100644 --- a/docs/guides/libraries.rst +++ b/docs/guides/libraries.rst @@ -193,47 +193,8 @@ How much of my library needs types? A "py.typed" library should aim to be type complete so that type checking and inspection can work to their full extent. Here we say that a library is “type complete” if all of the symbols -that comprise its interface have type annotations that refer to types -that are fully known. Private symbols are exempt. - -Library interface (public and private symbols) ----------------------------------------------- - -If a ``py.typed`` module is present, a type checker will treat all modules -within that package (i.e. all files that end in ``.py`` or ``.pyi``) as -importable unless the file name begins with an underscore. These modules -comprise the supported interface for the library. - -Each module exposes a set of symbols. Some of these symbols are -considered "private” — implementation details that are not part of the -library’s interface. Type checkers can use the following rules -to determine which symbols are visible outside of the package. - -- Symbols whose names begin with an underscore (but are not dunder - names) are considered private. -- Imported symbols are considered private by default. A fixed set of - :ref:`import forms ` re-export imported symbols. -- A module can expose an ``__all__`` symbol at the module level that - provides a list of names that are considered part of the interface. - This overrides all other rules above, allowing imported symbols or - symbols whose names begin with an underscore to be included in the - interface. -- Local variables within a function (including nested functions) are - always considered private. - -In stub files, ``__all__`` must be a string list literal. In ``.py`` files, -the following idioms are supported for defining the values contained -within ``__all__``. These restrictions allow type checkers to statically -determine the value of ``__all__``. - -- ``__all__ = ('a', b')`` -- ``__all__ = ['a', b']`` -- ``__all__ += ['a', b']`` -- ``__all__ += submodule.__all__`` -- ``__all__.extend(['a', b'])`` -- ``__all__.extend(submodule.__all__)`` -- ``__all__.append('a')`` -- ``__all__.remove('a')`` +that comprise its :ref:`interface ` have type annotations +that refer to types that are fully known. Private symbols are exempt. Type Completeness ----------------- diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 56bea9876..566757c93 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -65,53 +65,55 @@ in annotation contexts even before Python 3.9 reaches end-of-life. Supported Constructs ^^^^^^^^^^^^^^^^^^^^ -This section lists constructs that type checkers should accept in stub files. If -a construct is marked as "unspecified", type checkers may handle it as they best -see fit or report an error. - -Type checkers should fully support all features from the ``typing`` module of -the latest released Python version, as well as these constructs: +Type checkers should fully support these constructs:: +* All features from the ``typing`` module of the latest released Python version * Comments, including type declaration (``# type: X``) and error suppression - (``type: ignore``) comments. + (``type: ignore``) comments * Import statements, including the standard :ref:`import-conventions` and cyclic - imports. + imports * Module-level type aliases (e.g., ``X: TypeAlias = int``, - ``Y: TypeAlias = dict[str, _V]``). + ``Y: TypeAlias = dict[str, _V]``) * Regular aliases (e.g., ``function_alias = some_function``) at both the module- - and class-level. -* Simple version and platform checks, as described - :ref:`here `. + and class-level +* :ref:`Simple version and platform checks ` The constructs in the following subsections may be supported in a more limited fashion, as described below. +Value Expressions +""""""""""""""""" + +In locations where value expressions can appear, such as the right-hand side of +assignment statements and function parameter defaults, type checkers should +support the following expressions: + +* The ellipsis literal, ``...``, which can stand in for any value +* Any value that is a + :ref:`legal parameter for typing.Literal ` + Module Level Attributes """"""""""""""""""""""" -Module level variables and constants can be annotated using either -type comments or variable annotation syntax:: +Type checkers should support module-level variable annotations, with and without +assignments:: - x: int # recommended + x: int x: int = 0 x = 0 # type: int x = ... # type: int -The ellipsis literal can stand in for any value:: - - x: int = ... # type is int - A variable annotated as ``Final`` and assigned a literal value has the corresponding ``Literal`` type:: x: Final = 0 # type is Literal[0] -In all other cases, the type of a variable is unspecified when the variable is -unannotated or when the annotation and the assigned value disagree:: +When the type of a variable is omitted or disagrees from the assigned value, +type checker behavior is undefined:: - x = 0 # type is unspecified - x = ... # type is unspecified - x: int = "" # type is unspecified + x = 0 # behavior undefined + x: Final = ... # behavior undefined + x: int = "" # behavior undefined Classes """"""" @@ -119,8 +121,8 @@ Classes Class definition syntax follows general Python syntax, but type checkers are expected to understand only the following constructs in class bodies: -* The ellipsis literal ``...`` is ignored and used for empty - class bodies. Using ``pass`` in class bodies is undefined. +* The ellipsis literal ``...`` is used for empty class bodies. Using ``pass`` in + class bodies is undefined. * Instance attributes follow the same rules as module level attributes (see above). * Method definitions (see below) and properties. @@ -142,24 +144,8 @@ Yes:: Functions and Methods """"""""""""""""""""" -Function and method definition syntax follows general Python syntax. -If an argument or return type is unannotated, per :pep:`484` its -type is assumed to be ``Any``. - -If an argument has a literal or constant default value, it must match the implementation -and the type of the argument (if specified) must match the default value. -Alternatively, ``...`` can be used in place of any default value:: - - # The following arguments all have type Any. - def unannotated(a, b=42, c=...): ... - # The following arguments all have type int. - def annotated(a: int, b: int = 42, c: int = ...): ... - # The following default values are invalid and the types are unspecified. - def invalid(a: int = "", b: Foo = Foo()): ... - -Using a function or method body other than the ellipsis literal is currently -unspecified. Stub authors may experiment with other bodies, but it is up to -individual type checkers how to interpret them:: +Function and method definition follows general Python syntax. Using a function +or method body other than the ellipsis literal is undefined:: def foo(): ... # compatible def bar(): pass # behavior undefined @@ -369,6 +355,46 @@ of that Python version. This can be queried e.g. that the type checker allow for the user to point to a particular Python binary, in case it is not in the path. +.. _library-interface: + +Library interface (public and private symbols) +---------------------------------------------- + +If a ``py.typed`` module is present, a type checker will treat all modules +within that package (i.e. all files that end in ``.py`` or ``.pyi``) as +importable unless the file name begins with an underscore. These modules +comprise the supported interface for the library. + +Each module exposes a set of symbols. Some of these symbols are +considered "private” — implementation details that are not part of the +library’s interface. Type checkers can use the following rules +to determine which symbols are visible outside of the package. + +- Symbols whose names begin with an underscore (but are not dunder + names) are considered private. +- Imported symbols are considered private by default. A fixed set of + :ref:`import forms ` re-export imported symbols. +- A module can expose an ``__all__`` symbol at the module level that + provides a list of names that are considered part of the interface. + This overrides all other rules above, allowing imported symbols or + symbols whose names begin with an underscore to be included in the + interface. +- Local variables within a function (including nested functions) are + always considered private. + +The following idioms are supported for defining the values contained +within ``__all__``. These restrictions allow type checkers to statically +determine the value of ``__all__``. + +- ``__all__ = ('a', b')`` +- ``__all__ = ['a', b']`` +- ``__all__ += ['a', b']`` +- ``__all__ += submodule.__all__`` +- ``__all__.extend(['a', b'])`` +- ``__all__.extend(submodule.__all__)`` +- ``__all__.append('a')`` +- ``__all__.remove('a')`` + .. _import-conventions: Import Conventions @@ -382,8 +408,9 @@ The following import forms re-export symbols: * ``import X as X`` (a redundant module alias): re-exports ``X``. * ``from Y import X as X`` (a redundant symbol alias): re-exports ``X``. -* ``from Y import *``: re-exports all symbols in ``Y`` that do not begin with - an underscore. +* ``from Y import *``: if ``Y`` defines a module-level ``__all__`` list, + re-exports all names in ``__all__``; otherwise, re-exports all symbols in + ``Y`` that do not begin with an underscore. * ``from . import bar`` in an ``__init__`` module: re-exports ``bar`` if it does not begin with an underscore. * ``from .bar import Bar`` in an ``__init__`` module: re-exports ``Bar`` if it diff --git a/docs/spec/literal.rst b/docs/spec/literal.rst index 5cd50a443..f9447a276 100644 --- a/docs/spec/literal.rst +++ b/docs/spec/literal.rst @@ -93,6 +93,7 @@ what values may and may not be used as parameters. In short, a ``Literal[...]`` type may be parameterized by one or more literal expressions, and nothing else. +.. _literal-legal-parameters: Legal parameters for ``Literal`` at type check time """"""""""""""""""""""""""""""""""""""""""""""""""" From 61dafc4879f9d8d997d3de57e38d64f6589ce4c3 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Wed, 31 Jul 2024 00:04:53 -0700 Subject: [PATCH 30/41] Remove extra colon. --- docs/spec/distributing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 566757c93..3917e544d 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -65,7 +65,7 @@ in annotation contexts even before Python 3.9 reaches end-of-life. Supported Constructs ^^^^^^^^^^^^^^^^^^^^ -Type checkers should fully support these constructs:: +Type checkers should fully support these constructs: * All features from the ``typing`` module of the latest released Python version * Comments, including type declaration (``# type: X``) and error suppression From fe4c850363244ddf7e6a66e46c19c87eb3604d73 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Wed, 31 Jul 2024 00:44:56 -0700 Subject: [PATCH 31/41] Revert changes to wrting_stubs that should be made separately. --- docs/guides/writing_stubs.rst | 67 ----------------------------------- 1 file changed, 67 deletions(-) diff --git a/docs/guides/writing_stubs.rst b/docs/guides/writing_stubs.rst index 7000ca045..f59a2607a 100644 --- a/docs/guides/writing_stubs.rst +++ b/docs/guides/writing_stubs.rst @@ -337,46 +337,6 @@ No:: DAY_FLAG: int NIGHT_FLAG: int -Overloads ---------- - -All variants of overloaded functions and methods must have an ``@overload`` -decorator. Do not include the implementation's final non-`@overload`-decorated -definition. - -Yes:: - - @overload - def foo(x: str) -> str: ... - @overload - def foo(x: float) -> int: ... - -No:: - - @overload - def foo(x: str) -> str: ... - @overload - def foo(x: float) -> int: ... - def foo(x: str | float) -> Any: ... - -Decorators ----------- - -Include only the decorators listed :ref:`here `, whose effects -are understood by all of the major type checkers. The behavior of other -decorators should instead be incorporated into the types. For example, for the -following function:: - - import contextlib - @contextlib.contextmanager - def f(): - yield 42 - -the stub definition should be:: - - from contextlib import AbstractContextManager - def f() -> AbstractContextManager[int]: ... - Documentation or Implementation ------------------------------- @@ -621,30 +581,3 @@ No:: from typing import NamedTuple, TypedDict Point = NamedTuple("Point", [('x', float), ('y', float)]) Thing = TypedDict("Thing", {'stuff': str, 'index': int}) - -Built-in Generics ------------------ - -:pep:`585` built-in generics are supported and should be used instead -of the corresponding types from ``typing``:: - - from collections import defaultdict - - def foo(t: type[MyClass]) -> list[int]: ... - x: defaultdict[int] - -Using imports from ``collections.abc`` instead of ``typing`` is -generally possible and recommended:: - - from collections.abc import Iterable - - def foo(iter: Iterable[int]) -> None: ... - -Unions ------- - -Declaring unions with the shorthand `|` syntax is recommended and supported by -all type checkers:: - - def foo(x: int | str) -> int | None: ... # recommended - def foo(x: Union[int, str]) -> Optional[int]: ... # ok From 7bf4ce506357d5d9616bbba07a8ed6993a645c5e Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Wed, 31 Jul 2024 23:57:05 -0700 Subject: [PATCH 32/41] Delete unnecessary sentence. Co-authored-by: Carl Meyer --- docs/spec/distributing.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 3917e544d..2f60145b8 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -31,8 +31,8 @@ construct in a contradictory manner. However, type checkers are not required to implement checks for all of these constructs and can elect to ignore unsupported ones. Additionally, type checkers can support constructs not described here. -Type checkers should have a configurable search path for stub files. If a stub -file is found, the type checker should not read the corresponding "real" module. +If a stub +file is found for a module, the type checker should not read the corresponding "real" module. See :ref:`mro` for more information. .. _stub-file-syntax: From 6bbcff72b4ef087455c0d397bad73db8e0228485 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Thu, 1 Aug 2024 00:32:00 -0700 Subject: [PATCH 33/41] Clarify some language and terminology. --- docs/spec/distributing.rst | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 2f60145b8..ddea3d2c2 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -16,24 +16,25 @@ Python packages and modules. Stub files serve multiple purposes: * They are the only way to add type information to extension modules. * They can provide type information for packages that do not wish to add them inline. -* They can be distributed separately from the implementation. +* They can be distributed separately from the package or module that they + provide types for. The latter is referred to as the *implementation*. This allows stubs to be developed at a different pace or by different authors, which is especially useful when adding type annotations to existing packages. * They can act as documentation, succinctly explaining the external - API of a package, without including the implementation or private + API of a package, without including implementation details or private members. Stub files use a subset of the constructs used in Python source files, as described in :ref:`stub-file-supported-constructs` below. Type checkers should parse a stub that uses only such constructs without error and not interpret any -construct in a contradictory manner. However, type checkers are not required to -implement checks for all of these constructs and can elect to ignore unsupported -ones. Additionally, type checkers can support constructs not described here. +construct in a manner contradictory to this specification. However, type +checkers are not required to implement checks for all of these constructs and +can elect to ignore unsupported ones. Additionally, type checkers can support +constructs not described here. -If a stub -file is found for a module, the type checker should not read the corresponding "real" module. -See :ref:`mro` for more information. +If a stub file is found for a module, the type checker should not read the +corresponding "real" module. See :ref:`mro` for more information. .. _stub-file-syntax: From 98a72e7849d952bdd29b6b460bc2c39a1eeb5aa8 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Thu, 8 Aug 2024 22:12:51 -0700 Subject: [PATCH 34/41] Address reviewer feedback. --- docs/spec/distributing.rst | 45 ++++++++++++++------------------------ docs/spec/literal.rst | 2 ++ 2 files changed, 19 insertions(+), 28 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index ddea3d2c2..da480385a 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -45,21 +45,13 @@ Stub files are syntactically valid Python files in the earliest Python version that is not yet end-of-life. They use a ``.pyi`` suffix. The Python syntax used by stub files is independent from the Python versions supported by the implementation, and from the Python version the type checker runs under (if -any). Therefore, stubs may use the latest syntax features available in -the earliest supported version, even if the implementation supports older -versions. - -For example, Python 3.7 added the ``async`` keyword (see :pep:`492`). Stubs may -use it to mark coroutines, even if the implementation still uses the -``@coroutine`` decorator. On the other hand, the ``type`` soft keyword from -:pep:`695`, introduced in Python 3.12, is not valid in stubs until Python 3.11 -reaches end-of-life in October 2027. - -Stubs are treated as if ``from __future__ import annotations`` is enabled. In -particular, forward references do not need to be quoted, and syntax from newer -versions than otherwise supported may be used in annotation contexts. For -example, the pipe union syntax (``X | Y``) introduced in Python 3.10 may be used -in annotation contexts even before Python 3.9 reaches end-of-life. +any). + +Type checkers should treat stubs as if ``from __future__ import annotations`` is +enabled. In particular, forward references do not need to be quoted, and syntax +from newer versions than otherwise supported may be used in annotation contexts. +For example, the pipe union syntax (``X | Y``) introduced in Python 3.10 may be +used in annotation contexts even before Python 3.9 reaches end-of-life. .. _stub-file-supported-constructs: @@ -73,10 +65,7 @@ Type checkers should fully support these constructs: (``type: ignore``) comments * Import statements, including the standard :ref:`import-conventions` and cyclic imports -* Module-level type aliases (e.g., ``X: TypeAlias = int``, - ``Y: TypeAlias = dict[str, _V]``) -* Regular aliases (e.g., ``function_alias = some_function``) at both the module- - and class-level +* Aliases, including type aliases, at both the module and class level * :ref:`Simple version and platform checks ` The constructs in the following subsections may be supported in a more limited @@ -92,6 +81,8 @@ support the following expressions: * The ellipsis literal, ``...``, which can stand in for any value * Any value that is a :ref:`legal parameter for typing.Literal ` +* Floating point literals, such as `3.14` +* Complex literals, such as `1 + 2j` Module Level Attributes """"""""""""""""""""""" @@ -104,8 +95,8 @@ assignments:: x = 0 # type: int x = ... # type: int -A variable annotated as ``Final`` and assigned a literal value has the -corresponding ``Literal`` type:: +The :ref:`Literal shortcut using Final ` should be +supported:: x: Final = 0 # type is Literal[0] @@ -127,7 +118,7 @@ are expected to understand only the following constructs in class bodies: * Instance attributes follow the same rules as module level attributes (see above). * Method definitions (see below) and properties. -* Method aliases. +* Aliases. * Inner class definitions. Yes:: @@ -140,6 +131,7 @@ Yes:: def read_only(self) -> int: ... def do_stuff(self, y: str) -> None: ... doStuff = do_stuff + IntList: TypeAlias = list[int] class Inner: ... Functions and Methods @@ -164,6 +156,7 @@ in the ``typing`` module, plus these additional ones: * ``property`` (including ``.setter`` and ``.deleter``) * ``abc.abstractmethod`` * ``dataclasses.dataclass`` + * ``warnings.deprecated`` * functions decorated with ``@typing.dataclass_transform`` The Typeshed Project @@ -410,9 +403,5 @@ The following import forms re-export symbols: * ``import X as X`` (a redundant module alias): re-exports ``X``. * ``from Y import X as X`` (a redundant symbol alias): re-exports ``X``. * ``from Y import *``: if ``Y`` defines a module-level ``__all__`` list, - re-exports all names in ``__all__``; otherwise, re-exports all symbols in - ``Y`` that do not begin with an underscore. -* ``from . import bar`` in an ``__init__`` module: re-exports ``bar`` if it does - not begin with an underscore. -* ``from .bar import Bar`` in an ``__init__`` module: re-exports ``Bar`` if it - does not begin with an underscore. + re-exports all names in ``__all__``; otherwise, re-exports all public symbols + in ``Y``'s global scope. diff --git a/docs/spec/literal.rst b/docs/spec/literal.rst index f9447a276..564ddaa44 100644 --- a/docs/spec/literal.rst +++ b/docs/spec/literal.rst @@ -506,6 +506,8 @@ involving Literal bools. For example, we can combine ``Literal[True]``, else: scalar += "foo" # Type checks: type of 'scalar' is narrowed to 'str' +.. _literal-final-interactions: + Interactions with Final """"""""""""""""""""""" From a448b4ae53f87c40bb3a5c9767955f0389af7ddc Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Thu, 8 Aug 2024 22:19:40 -0700 Subject: [PATCH 35/41] Formatting --- docs/spec/distributing.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index da480385a..f9cfede13 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -81,8 +81,8 @@ support the following expressions: * The ellipsis literal, ``...``, which can stand in for any value * Any value that is a :ref:`legal parameter for typing.Literal ` -* Floating point literals, such as `3.14` -* Complex literals, such as `1 + 2j` +* Floating point literals, such as ``3.14`` +* Complex literals, such as ``1 + 2j`` Module Level Attributes """"""""""""""""""""""" From eaaade2d370a31963c55175210a01b581632566f Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Thu, 8 Aug 2024 22:43:55 -0700 Subject: [PATCH 36/41] Drop "annotation contexts." --- docs/spec/distributing.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index f9cfede13..613c828aa 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -49,9 +49,9 @@ any). Type checkers should treat stubs as if ``from __future__ import annotations`` is enabled. In particular, forward references do not need to be quoted, and syntax -from newer versions than otherwise supported may be used in annotation contexts. -For example, the pipe union syntax (``X | Y``) introduced in Python 3.10 may be -used in annotation contexts even before Python 3.9 reaches end-of-life. +from newer versions than otherwise supported may be used. For example, the pipe +union syntax (``X | Y``) introduced in Python 3.10 may be used even before +Python 3.9 reaches end-of-life. .. _stub-file-supported-constructs: From 8d18706847a0964f7fc9517e1a84da0917f5b2b8 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Fri, 9 Aug 2024 11:00:57 -0700 Subject: [PATCH 37/41] Add "in annotation expressions" to newer syntax explanation. --- docs/spec/distributing.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 613c828aa..b86f164e7 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -49,9 +49,9 @@ any). Type checkers should treat stubs as if ``from __future__ import annotations`` is enabled. In particular, forward references do not need to be quoted, and syntax -from newer versions than otherwise supported may be used. For example, the pipe -union syntax (``X | Y``) introduced in Python 3.10 may be used even before -Python 3.9 reaches end-of-life. +from newer versions than otherwise supported may be used in annotation +expressions. For example, the pipe union syntax (``X | Y``) introduced in Python +3.10 may be used even before Python 3.9 reaches end-of-life. .. _stub-file-supported-constructs: From 783b4357beb2816910f3e2d69e07b3b70b0928f9 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 12 Aug 2024 17:44:49 -0700 Subject: [PATCH 38/41] Update docs/spec/distributing.rst Co-authored-by: Jelle Zijlstra --- docs/spec/distributing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index b86f164e7..17b8c00fb 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -62,7 +62,7 @@ Type checkers should fully support these constructs: * All features from the ``typing`` module of the latest released Python version * Comments, including type declaration (``# type: X``) and error suppression - (``type: ignore``) comments + (``# type: ignore``) comments * Import statements, including the standard :ref:`import-conventions` and cyclic imports * Aliases, including type aliases, at both the module and class level From 6f030ad3b1491d456771f71063173f95b06a59b9 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Wed, 14 Aug 2024 19:50:11 -0700 Subject: [PATCH 39/41] Reword syntax section. --- docs/spec/distributing.rst | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 17b8c00fb..886a95f98 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -47,11 +47,12 @@ by stub files is independent from the Python versions supported by the implementation, and from the Python version the type checker runs under (if any). -Type checkers should treat stubs as if ``from __future__ import annotations`` is -enabled. In particular, forward references do not need to be quoted, and syntax -from newer versions than otherwise supported may be used in annotation -expressions. For example, the pipe union syntax (``X | Y``) introduced in Python -3.10 may be used even before Python 3.9 reaches end-of-life. +Type checkers should evaluate all annotation expressions as if they are quoted. +Consequently, forward references do not need to be quoted, and syntax from newer +versions than otherwise supported may be used in annotations. For example, the +pipe union syntax (``X | Y``) introduced in Python 3.10 may be used even before +Python 3.9 reaches end-of-life. Outside of annotations, type checkers may but +are not required to support type system features that use newer syntax. .. _stub-file-supported-constructs: @@ -61,6 +62,7 @@ Supported Constructs Type checkers should fully support these constructs: * All features from the ``typing`` module of the latest released Python version + that use :ref:`supported syntax ` * Comments, including type declaration (``# type: X``) and error suppression (``# type: ignore``) comments * Import statements, including the standard :ref:`import-conventions` and cyclic From fb8c354595cef58c28cd308f79427cad1e835c57 Mon Sep 17 00:00:00 2001 From: Rebecca Chen Date: Mon, 19 Aug 2024 20:31:27 -0700 Subject: [PATCH 40/41] Update docs/spec/distributing.rst Co-authored-by: Jelle Zijlstra --- docs/spec/distributing.rst | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 886a95f98..804653ffb 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -41,18 +41,26 @@ corresponding "real" module. See :ref:`mro` for more information. Syntax ^^^^^^ -Stub files are syntactically valid Python files in the earliest Python version -that is not yet end-of-life. They use a ``.pyi`` suffix. The Python syntax used -by stub files is independent from the Python versions supported by the -implementation, and from the Python version the type checker runs under (if -any). - -Type checkers should evaluate all annotation expressions as if they are quoted. -Consequently, forward references do not need to be quoted, and syntax from newer -versions than otherwise supported may be used in annotations. For example, the -pipe union syntax (``X | Y``) introduced in Python 3.10 may be used even before -Python 3.9 reaches end-of-life. Outside of annotations, type checkers may but -are not required to support type system features that use newer syntax. +Stub files are syntactically valid Python files with a ``.pyi`` suffix. They +should be parseable (e.g., with :py:func:`ast.parse`) in all Python versions +that are supported by the implementation and that are still supported +by the CPython project. For example, defining a type alias using the +``type`` keyword is only accepted by the Python parser in Python 3.12 and later, +so stubs supporting Python 3.11 or earlier versions should not use this syntax. +This allows type checkers implemented in Python to parse stub files using +functionality from the standard library. +Type checkers may choose to support syntactic features from newer Python versions +in stub files, but stubs that rely on such features may not be portable to all +type checkers. Type checkers may also choose to support Python versions that +are no longer supported by CPython; if so, they cannot rely on standard library +functionality to parse stub files. + +Type checkers should evaluate all :term:`annotation expressions ` as if they are quoted. +Consequently, forward references do not need to be quoted, and type system +features that do not depend on Python syntax changes are supported in stubs regardless +of the Python version supported. For example, the use of the ``|`` operator +to create unions (``X | Y``) was introduced in Python 3.10, but may be used +even in stubs that support Python 3.9 and older versions. .. _stub-file-supported-constructs: From 472397235df3925d716d4915d87968fe51d1df17 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 20 Aug 2024 03:31:33 +0000 Subject: [PATCH 41/41] [pre-commit.ci] auto fixes from pre-commit.com hooks --- docs/spec/distributing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/spec/distributing.rst b/docs/spec/distributing.rst index 804653ffb..57bcc66e2 100644 --- a/docs/spec/distributing.rst +++ b/docs/spec/distributing.rst @@ -60,7 +60,7 @@ Consequently, forward references do not need to be quoted, and type system features that do not depend on Python syntax changes are supported in stubs regardless of the Python version supported. For example, the use of the ``|`` operator to create unions (``X | Y``) was introduced in Python 3.10, but may be used -even in stubs that support Python 3.9 and older versions. +even in stubs that support Python 3.9 and older versions. .. _stub-file-supported-constructs: