Migrate to pyproject.toml

[JP-Ellis]

Following PEP 518 and PEP 621, we should migrate the older setup.cfg and setup.py build system to one making use of pyproject.toml. Furthermore, the use of setup.py is strongly discouraged and many aspects will be discontinued:

Setuptools offers first class support for setup.py files as a configuration mechanism.

It is important to remember, however, that running this file as a script (e.g. python setup.py sdist) is strongly discouraged, and that the majority of the command line interfaces are (or will be) deprecated (e.g. python setup.py install, python setup.py bdist_wininst, …).

We also recommend users to expose as much as possible configuration in a more declarative way via the pyproject.toml or setup.cfg, and keep the setup.py minimal with only the dynamic parts (or even omit it completely if applicable).

See Why you shouldn’t invoke setup.py directly for more background.

source

The existing setup.py build script also makes use of distutils which has been deprecated since 3.10 and is scheduled to removal in 3.12.

As part of this change, we also have an opportunity to change the underlying build system that handles the packaging of the Python package into a distributable bundle. The main ones to consider are:

• Setuptools
• Poetry
• Maturin
• Hatch
• PDM and PDM-backend
• Flit

An excellent write up by Jay Qi is available in their blog post The Basic of Python Packaging in Early 2023.

As pact-python will eventually make use of the Pact Rust core, any build system considered must be able to support so-called 'extensions' (which are typically used to build C extensions).

Deep Dive

Setuptools

This by far the most popular build system and is the current build system being used. Extensions are supported through setup.py.

Note that even if we keep this, closing this issue will require an extensive rework of setup.py due to the aforementioned deprecations.

A good example I have come across for this implementation would be following playwright-python's setup.py.

Poetry

Poetry is a very popular Python tool that predates PEP 621. Unfortunately, it does not yet support PEP 621 (though it is on the roadmap) and instead uses its own custom metadata section in pyproject.toml.

More concerning is the unstable support of extensions (python-poetry/poetry#2740) which effectively rules out Poetry as the build tool.

Poetry may still be useful to manage dependencies, as it does an excellent job of ensuring all dependencies installed within the virtual environment are mutually compatible. Furthermore, it integrates with dependabot. This of course depends on whether the build tool can work with Poetry for dependency management.

Maturin

Maturin is a project by the PyO3 team which maintains an ecosystem of tools that provide Rust binding for Python.

This may be an excellent option given that pact-python will eventually need to integrate with the Rust core and may be easier to use than Setuptools.

There is a concern that integrating too much with Rust might dissuade contributions from Python developers, and we must also check whether we can still rely on Ruby while migrating to the Rust core.

Hatch

Hatch is build tool and front-end. It appears to have extensive support for customisations through build hooks. These hooks are likely able to support both the Ruby and Rust implementations simultaneously, but this would need to be investigated properly.

Flit

Flit is a lightweight build manager, but unfortunately does not support build extensions and therefore is not appropriate here.

PDM

PDM describes itself as a "modern Python package and dependency manager supporting the latest PEP standards". It was created as an implementation of PEP 582 which provides an venv-less way of managing dependencies in a similar way to npm. While PEP 582 was rejected, PDM still support this.

PDM appears to have excellent build and publishing capabilities, and the PDM backend allows for customisations through hooks pdm_build.py.

[github-actions]

👋 Hi! The 'smartbear-supported' label has just been added to this issue, which will create an internal tracking ticket in PactFlow's Jira (PACT-1305). We will use this to prioritise and assign a team member to this task. All activity will be public on this ticket. For now, sit tight and we'll update this ticket once we have more information on the next steps.

See our documentation for more information.

[YOU54F]

I noted Ron had looked at Maturin at the first attempt of the Rust integration- https://github.com/pact-foundation/pact-python/blob/feat/V3/pact-python-v3/pyproject.toml for an example

[JP-Ellis]

Thanks, I'll take a look. I know of maturin mostly from the ruff project; but have not had a need to use it before.