From ad2785177fd72d02486e6972382c964b13711e75 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 13 Jun 2022 13:20:58 -0700 Subject: [PATCH 01/11] docs/userguide/ext_modules.rst: New --- docs/userguide/ext_modules.rst | 40 ++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 docs/userguide/ext_modules.rst diff --git a/docs/userguide/ext_modules.rst b/docs/userguide/ext_modules.rst new file mode 100644 index 0000000000..7819ed4bd6 --- /dev/null +++ b/docs/userguide/ext_modules.rst @@ -0,0 +1,40 @@ +========================== +Building Extension Modules +========================== + +Compiler and linker options +=========================== + +The command ``build_ext`` builds C/C++ extension modules. It creates +a command line for running the compiler and linker by combining +compiler and linker options from various sources, as specified by +`test_customize_compiler +`_: + + * the ``sysconfig`` variables ``CFLAGS`` and ``LDFLAGS``, + * the environment variables :envvar:`CC`, :envvar:`CPP`, + :envvar:`CXX`, :envvar:`LDSHARED` and :envvar:`LDFLAGS`, + :envvar:`CFLAGS`, :envvar:`CPPFLAGS`, :envvar:`LDFLAGS`, + * the :class:`Extension` options. + +.. Ignoring AR, ARFLAGS, RANLIB here because they are used by the (obsolete?) build_clib, not build_ext. + +The resulting command line is then processed by the compiler and linker. +According to the GCC manual sections on `directory options +`_ and +`environment variables +`` +directives in the following order: + + * first, in directories given by ``-I`` options (in left-to-right order), + * then, in directories given by the environment variable :envvar:`CPATH` (in left-to-right order), + * then, in directories given by ``-isystem`` options (in left-to-right order), + * then, in directories given by the environment variable :envvar:`C_INCLUDE_PATH` (for C) and :envvar:`CPLUS_INCLUDE_PATH` (for C++), + * then, in standard system directories, + * finally, in directories given by ``-idirafter`` options (in left-to-right order). + +The linker searches for libraries in the following order: + + * first, in directories given by ``-L`` options (in left-to-right order), + * then, in directories given by the environment variable :envvar:`LIBRARY_PATH` (in left-to-right order). From adfe9c3c54e81cfd14b59e31e43673d81cb8a055 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 13 Jun 2022 13:44:34 -0700 Subject: [PATCH 02/11] docs/userguide/ext_modules.rst: Add more specifics about Extension attributes --- docs/userguide/ext_modules.rst | 10 +++++++++- docs/userguide/index.rst | 1 + 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/userguide/ext_modules.rst b/docs/userguide/ext_modules.rst index 7819ed4bd6..fa06a8e990 100644 --- a/docs/userguide/ext_modules.rst +++ b/docs/userguide/ext_modules.rst @@ -2,6 +2,12 @@ Building Extension Modules ========================== +Setuptools can build C/C++ extension modules. The keyword argument +``ext_modules`` of :func:`setup` should be a list of instances of the +`Extension class +`_. + + Compiler and linker options =========================== @@ -15,7 +21,9 @@ compiler and linker options from various sources, as specified by * the environment variables :envvar:`CC`, :envvar:`CPP`, :envvar:`CXX`, :envvar:`LDSHARED` and :envvar:`LDFLAGS`, :envvar:`CFLAGS`, :envvar:`CPPFLAGS`, :envvar:`LDFLAGS`, - * the :class:`Extension` options. + * the :class:`Extension` attributes ``include_dirs``, + ``library_dirs``, ``extra_compile_args``, ``extra_link_args``, + ``runtime_library_dirs``. .. Ignoring AR, ARFLAGS, RANLIB here because they are used by the (obsolete?) build_clib, not build_ext. diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index d5d150af2e..7c20fe0404 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -28,6 +28,7 @@ Contents package_discovery entry_point dependency_management + ext_modules datafiles development_mode distribution From 78436032f9eed6f943074828b7c6ca926d23a280 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 13 Jun 2022 14:04:33 -0700 Subject: [PATCH 03/11] docs/userguide/ext_modules.rst: More sysconfig variables --- docs/userguide/ext_modules.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/userguide/ext_modules.rst b/docs/userguide/ext_modules.rst index fa06a8e990..129307f820 100644 --- a/docs/userguide/ext_modules.rst +++ b/docs/userguide/ext_modules.rst @@ -17,7 +17,8 @@ compiler and linker options from various sources, as specified by `test_customize_compiler `_: - * the ``sysconfig`` variables ``CFLAGS`` and ``LDFLAGS``, + * the ``sysconfig`` variables ``CC``, ``CXX``, ``CCSHARED``, + ``LDSHARED``, and ``CFLAGS``, * the environment variables :envvar:`CC`, :envvar:`CPP`, :envvar:`CXX`, :envvar:`LDSHARED` and :envvar:`LDFLAGS`, :envvar:`CFLAGS`, :envvar:`CPPFLAGS`, :envvar:`LDFLAGS`, From 6e4a1f394de2848f8adf99acc4ca4961b3cb009d Mon Sep 17 00:00:00 2001 From: Anderson Bravalheri Date: Tue, 14 Jun 2022 02:06:49 +0100 Subject: [PATCH 04/11] Add docstring documentation to setuptools.Extension --- setuptools/extension.py | 73 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 72 insertions(+), 1 deletion(-) diff --git a/setuptools/extension.py b/setuptools/extension.py index f696c9c1ac..31c3a86017 100644 --- a/setuptools/extension.py +++ b/setuptools/extension.py @@ -28,7 +28,78 @@ def _have_cython(): class Extension(_Extension): - """Extension that uses '.c' files in place of '.pyx' files""" + """ + Describes a single extension module. + + This means that all source files will be compiled into a single binary file + ``.`` (with ```` derived from ``name`` and + ```` defined by one of the values in + :mod:`importlib.machinery.EXTENSION_SUFFIXES`). + + In the case ``.pyx`` files are passed as ``sources and`` ``Cython`` is **not** + installed in the build environment, ``setuptools`` may also try to look for the + equivalent ``.cpp`` or ``.c`` files. + + :arg str name: + the full name of the extension, including any packages -- ie. + *not* a filename or pathname, but Python dotted name + :arg list[str] sources: + list of source filenames, relative to the distribution root + (where the setup script lives), in Unix form (slash-separated) + for portability. Source files may be C, C++, SWIG (.i), + platform-specific resource files, or whatever else is recognized + by the "build_ext" command as source for a Python extension. + + :keyword list[str] include_dirs: + list of directories to search for C/C++ header files (in Unix + form for portability) + :keyword list[tuple[str, str|None]] define_macros: + list of macros to define; each macro is defined using a 2-tuple: + the first item corresponding to the name of the macro and the second + item either a string with its value or None to + define it without a particular value (equivalent of "#define + FOO" in source or -DFOO on Unix C compiler command line) + :keyword list[str] undef_macros: + list of macros to undefine explicitly + :keyword list[str] library_dirs: + list of directories to search for C/C++ libraries at link time + :keyword list[str] libraries: + list of library names (not filenames or paths) to link against + :keyword list[str] runtime_library_dirs: + list of directories to search for C/C++ libraries at run time + (for shared extensions, this is when the extension is loaded) + :keyword list[str] extra_objects: + list of extra files to link with (eg. object files not implied + by 'sources', static library that must be explicitly specified, + binary resource files, etc.) + :keyword list[str] extra_compile_args: + any extra platform- and compiler-specific information to use + when compiling the source files in 'sources'. For platforms and + compilers where "command line" makes sense, this is typically a + list of command-line arguments, but for other platforms it could + be anything. + :keyword list[str] extra_link_args: + any extra platform- and compiler-specific information to use + when linking object files together to create the extension (or + to create a new static Python interpreter). Similar + interpretation as for 'extra_compile_args'. + :keyword list[str] export_symbols: + list of symbols to be exported from a shared extension. Not + used on all platforms, and not generally necessary for Python + extensions, which typically export exactly one symbol: "init" + + extension_name. + :keyword list[str] swig_opts: + any extra options to pass to SWIG if a source file has the .i + extension. + :keyword list[str] depends: + list of files that the extension depends on + :keyword str language: + extension language (i.e. "c", "c++", "objc"). Will be detected + from the source extensions if not provided. + :keyword bool optional: + specifies that a build failure in the extension should not abort the + build process, but simply not install the failing extension. + """ def __init__(self, name, sources, *args, **kw): # The *args is needed for compatibility as calls may use positional From 4029317682189f5e5110b25731adcdafb15e4bcd Mon Sep 17 00:00:00 2001 From: Anderson Bravalheri Date: Tue, 14 Jun 2022 02:08:38 +0100 Subject: [PATCH 05/11] Minor formatting/editorial improvements on ext_modules docs --- docs/userguide/ext_modules.rst | 70 +++++++++++++++++++--------------- 1 file changed, 40 insertions(+), 30 deletions(-) diff --git a/docs/userguide/ext_modules.rst b/docs/userguide/ext_modules.rst index 129307f820..a63ada82f8 100644 --- a/docs/userguide/ext_modules.rst +++ b/docs/userguide/ext_modules.rst @@ -3,9 +3,8 @@ Building Extension Modules ========================== Setuptools can build C/C++ extension modules. The keyword argument -``ext_modules`` of :func:`setup` should be a list of instances of the -`Extension class -`_. +``ext_modules`` of ``setup`` should be a list of instances of the +:class:`setuptools.Extension` class. Compiler and linker options @@ -13,37 +12,48 @@ Compiler and linker options The command ``build_ext`` builds C/C++ extension modules. It creates a command line for running the compiler and linker by combining -compiler and linker options from various sources, as specified by -`test_customize_compiler -`_: - - * the ``sysconfig`` variables ``CC``, ``CXX``, ``CCSHARED``, - ``LDSHARED``, and ``CFLAGS``, - * the environment variables :envvar:`CC`, :envvar:`CPP`, - :envvar:`CXX`, :envvar:`LDSHARED` and :envvar:`LDFLAGS`, - :envvar:`CFLAGS`, :envvar:`CPPFLAGS`, :envvar:`LDFLAGS`, - * the :class:`Extension` attributes ``include_dirs``, - ``library_dirs``, ``extra_compile_args``, ``extra_link_args``, - ``runtime_library_dirs``. +compiler and linker options from various sources: + +.. Reference: `test_customize_compiler` in distutils/tests/test_sysconfig.py + +* the ``sysconfig`` variables ``CC``, ``CXX``, ``CCSHARED``, + ``LDSHARED``, and ``CFLAGS``, +* the environment variables ``CC``, ``CPP``, + ``CXX``, ``LDSHARED`` and ``LDFLAGS``, + ``CFLAGS``, ``CPPFLAGS``, ``LDFLAGS``, +* the ``Extension`` attributes ``include_dirs``, + ``library_dirs``, ``extra_compile_args``, ``extra_link_args``, + ``runtime_library_dirs``. .. Ignoring AR, ARFLAGS, RANLIB here because they are used by the (obsolete?) build_clib, not build_ext. The resulting command line is then processed by the compiler and linker. -According to the GCC manual sections on `directory options -`_ and -`environment variables -`` -directives in the following order: - - * first, in directories given by ``-I`` options (in left-to-right order), - * then, in directories given by the environment variable :envvar:`CPATH` (in left-to-right order), - * then, in directories given by ``-isystem`` options (in left-to-right order), - * then, in directories given by the environment variable :envvar:`C_INCLUDE_PATH` (for C) and :envvar:`CPLUS_INCLUDE_PATH` (for C++), - * then, in standard system directories, - * finally, in directories given by ``-idirafter`` options (in left-to-right order). +According to the GCC manual sections on `directory options`_ and +`environment variables`_, the C/C++ compiler searches for files named in +``#include `` directives in the following order: + +* first, in directories given by ``-I`` options (in left-to-right order), +* then, in directories given by the environment variable ``CPATH`` (in left-to-right order), +* then, in directories given by ``-isystem`` options (in left-to-right order), +* then, in directories given by the environment variable ``C_INCLUDE_PATH`` (for C) and ``CPLUS_INCLUDE_PATH`` (for C++), +* then, in standard system directories, +* finally, in directories given by ``-idirafter`` options (in left-to-right order). The linker searches for libraries in the following order: - * first, in directories given by ``-L`` options (in left-to-right order), - * then, in directories given by the environment variable :envvar:`LIBRARY_PATH` (in left-to-right order). +* first, in directories given by ``-L`` options (in left-to-right order), +* then, in directories given by the environment variable ``LIBRARY_PATH`` (in left-to-right order). + + +---- + +API Reference +------------- + +.. autoclass:: setuptools.Extension + + +.. _Python docs about C/C++ extensions: https://docs.python.org/3/extending/extending.html +.. _Cython: https://cython.readthedocs.io/en/stable/index.html +.. _directory options: https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html +.. _environment variables: https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html> From 2a2ef46cdc6f082aa17cccfdfd2b71166e87a591 Mon Sep 17 00:00:00 2001 From: Anderson Bravalheri Date: Tue, 14 Jun 2022 02:09:13 +0100 Subject: [PATCH 06/11] Add small example --- docs/userguide/ext_modules.rst | 42 ++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/docs/userguide/ext_modules.rst b/docs/userguide/ext_modules.rst index a63ada82f8..66e74cbd7f 100644 --- a/docs/userguide/ext_modules.rst +++ b/docs/userguide/ext_modules.rst @@ -7,6 +7,48 @@ Setuptools can build C/C++ extension modules. The keyword argument :class:`setuptools.Extension` class. +For example, let's consider a simple project with only one extension module:: + + + ├── pyproject.toml + └── foo.c + +and all project metadata configuration in the ``pyproject.toml`` file: + +.. code-block:: toml + + # pyproject.toml + [build-system] + requires = ["setuptools"] + build-backend = "setuptools.build_meta" + + [project] + name = "mylib-foo" # as it would appear on PyPI + version = "0.42" + +To instruct setuptools to compile the ``foo.c`` file into the extension module +``mylib.foo``, we need to add a ``setup.py`` file similar to the following: + +.. code-block:: python + + from setuptools import Extension, setup + + setup( + ext_modules=[ + Extension( + name="mylib.foo", # as it would be imported + # may include packages/namespaces separated by `.` + + sources=["foo.c"], # all sources are compiled into a single binary file + ), + ] + ) + +.. seealso:: + You can find more information on the `Python docs about C/C++ extensions`_. + Alternatively, you might also be interested in learn about `Cython`_. + + Compiler and linker options =========================== From 0953ce60f3eceed8305158183fd480f8e4d82f2f Mon Sep 17 00:00:00 2001 From: Anderson Bravalheri Date: Tue, 14 Jun 2022 09:58:15 +0100 Subject: [PATCH 07/11] Fix missing intersphinx reference in Extension docstring --- setuptools/extension.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setuptools/extension.py b/setuptools/extension.py index 31c3a86017..2eed0631ab 100644 --- a/setuptools/extension.py +++ b/setuptools/extension.py @@ -34,7 +34,7 @@ class Extension(_Extension): This means that all source files will be compiled into a single binary file ``.`` (with ```` derived from ``name`` and ```` defined by one of the values in - :mod:`importlib.machinery.EXTENSION_SUFFIXES`). + ``importlib.machinery.EXTENSION_SUFFIXES``). In the case ``.pyx`` files are passed as ``sources and`` ``Cython`` is **not** installed in the build environment, ``setuptools`` may also try to look for the From 8ff090aacb9215893fd9af45343a0e0e1200f14e Mon Sep 17 00:00:00 2001 From: Anderson Bravalheri Date: Tue, 14 Jun 2022 10:16:36 +0100 Subject: [PATCH 08/11] Make it easier to read the docstring for setuptools/extension --- setuptools/extension.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/setuptools/extension.py b/setuptools/extension.py index 2eed0631ab..64baf1147b 100644 --- a/setuptools/extension.py +++ b/setuptools/extension.py @@ -43,6 +43,7 @@ class Extension(_Extension): :arg str name: the full name of the extension, including any packages -- ie. *not* a filename or pathname, but Python dotted name + :arg list[str] sources: list of source filenames, relative to the distribution root (where the setup script lives), in Unix form (slash-separated) @@ -53,49 +54,62 @@ class Extension(_Extension): :keyword list[str] include_dirs: list of directories to search for C/C++ header files (in Unix form for portability) + :keyword list[tuple[str, str|None]] define_macros: list of macros to define; each macro is defined using a 2-tuple: the first item corresponding to the name of the macro and the second item either a string with its value or None to define it without a particular value (equivalent of "#define FOO" in source or -DFOO on Unix C compiler command line) + :keyword list[str] undef_macros: list of macros to undefine explicitly + :keyword list[str] library_dirs: list of directories to search for C/C++ libraries at link time + :keyword list[str] libraries: list of library names (not filenames or paths) to link against + :keyword list[str] runtime_library_dirs: list of directories to search for C/C++ libraries at run time (for shared extensions, this is when the extension is loaded) + :keyword list[str] extra_objects: list of extra files to link with (eg. object files not implied by 'sources', static library that must be explicitly specified, binary resource files, etc.) + :keyword list[str] extra_compile_args: any extra platform- and compiler-specific information to use when compiling the source files in 'sources'. For platforms and compilers where "command line" makes sense, this is typically a list of command-line arguments, but for other platforms it could be anything. + :keyword list[str] extra_link_args: any extra platform- and compiler-specific information to use when linking object files together to create the extension (or to create a new static Python interpreter). Similar interpretation as for 'extra_compile_args'. + :keyword list[str] export_symbols: list of symbols to be exported from a shared extension. Not used on all platforms, and not generally necessary for Python extensions, which typically export exactly one symbol: "init" + extension_name. + :keyword list[str] swig_opts: any extra options to pass to SWIG if a source file has the .i extension. + :keyword list[str] depends: list of files that the extension depends on + :keyword str language: extension language (i.e. "c", "c++", "objc"). Will be detected from the source extensions if not provided. + :keyword bool optional: specifies that a build failure in the extension should not abort the build process, but simply not install the failing extension. From 875d771dcccc966b6f84c456a692515ac93956b5 Mon Sep 17 00:00:00 2001 From: Anderson Bravalheri Date: Tue, 14 Jun 2022 10:17:44 +0100 Subject: [PATCH 09/11] Add remarks to docs on extension modules --- docs/userguide/ext_modules.rst | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/userguide/ext_modules.rst b/docs/userguide/ext_modules.rst index 66e74cbd7f..4ed50a3c37 100644 --- a/docs/userguide/ext_modules.rst +++ b/docs/userguide/ext_modules.rst @@ -3,7 +3,7 @@ Building Extension Modules ========================== Setuptools can build C/C++ extension modules. The keyword argument -``ext_modules`` of ``setup`` should be a list of instances of the +``ext_modules`` of ``setup()`` should be a list of instances of the :class:`setuptools.Extension` class. @@ -48,6 +48,9 @@ To instruct setuptools to compile the ``foo.c`` file into the extension module You can find more information on the `Python docs about C/C++ extensions`_. Alternatively, you might also be interested in learn about `Cython`_. + If you plan to distribute a package that uses extensions across multiple + platforms, :pypi:`cibuildwheel` can also be helpful. + Compiler and linker options =========================== @@ -86,6 +89,17 @@ The linker searches for libraries in the following order: * first, in directories given by ``-L`` options (in left-to-right order), * then, in directories given by the environment variable ``LIBRARY_PATH`` (in left-to-right order). +.. important:: + All files used to compile your extension need to be available on the system + in the moment setuptools builds your project, so please make sure to include some + documentation on how users can obtain operating system level dependencies + (e.g. compilers and external binary libraries/artifacts). + + You will also need to make sure that all auxiliary files that are contained + inside your :term:`project` (e.g. C headers authored by you or your team) + are configured to be included in your :term:`sdist `. + Please have a look on our section on :ref:`Controlling files in the distribution`. + ---- From 7e855eca96a2e852d31aabfc0e444b3e7504bb46 Mon Sep 17 00:00:00 2001 From: Anderson Bravalheri Date: Tue, 14 Jun 2022 10:46:56 +0100 Subject: [PATCH 10/11] Improve remark about external dependencies --- docs/userguide/ext_modules.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/userguide/ext_modules.rst b/docs/userguide/ext_modules.rst index 4ed50a3c37..213e13c0c2 100644 --- a/docs/userguide/ext_modules.rst +++ b/docs/userguide/ext_modules.rst @@ -91,8 +91,9 @@ The linker searches for libraries in the following order: .. important:: All files used to compile your extension need to be available on the system - in the moment setuptools builds your project, so please make sure to include some - documentation on how users can obtain operating system level dependencies + when building the package, so please make sure to include some documentation + on how developers interested in building your package from source + can obtain operating system level dependencies (e.g. compilers and external binary libraries/artifacts). You will also need to make sure that all auxiliary files that are contained From 0540600a291b6c956babc4e29b89a2f17c4edf13 Mon Sep 17 00:00:00 2001 From: Anderson Bravalheri Date: Tue, 14 Jun 2022 17:23:33 +0100 Subject: [PATCH 11/11] Add news fragment --- changelog.d/3368.doc.rst | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelog.d/3368.doc.rst diff --git a/changelog.d/3368.doc.rst b/changelog.d/3368.doc.rst new file mode 100644 index 0000000000..229cf552eb --- /dev/null +++ b/changelog.d/3368.doc.rst @@ -0,0 +1 @@ +Added documentation page about extension modules -- by :user:`mkoeppe` \ No newline at end of file