build: Migrate from setup.py to pyproject.toml

[adrienverge]

Using setup.py is deprecated and the new recommanded way is to declare a pyproject.toml file (see PEP 517 ¹).

This commit proposes to use setuptools to achieve that, because setuptools is already used by yamllint, is standard and referenced by the official Python packaging documentation ², and seems to be the most lightweight solution. An alternative could have been to use Poetry, see the dedicated pull request and discussion ³.

For some period, the setup.py file will be kept (although almost empty), to allow old tools versions to keep working.

Closes #509.

---

Changes to RPM package

@@ -28,18 +28,19 @@ yamllint does not only check for syntax validity, but for weirdnesses lik>
 repetition and cosmetic problems such as lines length, trailing spaces,
 indentation, etc.
 
+%generate_buildrequires
+%pyproject_buildrequires
+
 %prep
 %autosetup -n %{pypi_name}-%{version} -p1
 # Remove bundled egg-info
 rm -rf %{pypi_name}.egg-info
 
 %build
-%py3_build
+%pyproject_wheel
 
 %install
-# Must do the subpackages' install first because the scripts in /usr/bin are
-# overwritten with every setup.py install.
-%py3_install
+%pyproject_install
 
 %check
 %{__python3} -m unittest discover

Footnotes

1. https://peps.python.org/pep-0517/ ↩

2. https://packaging.python.org/en/latest/tutorials/installing-packages/ ↩

3. https://github.com/adrienverge/yamllint/pull/557 ↩

[adrienverge]

@DimitriPapadopoulos @staticdev @ssbarnea @barrelful following your remarks from #509 and #557, could you test this pull request (at least python -m build, pip install and basic yamllint usage) and check whether everything works as you expect? Thanks in advance.

[coveralls]

Coverage: 99.384% (+0.2%) from 99.196% when pulling bb8300f on build/pyproject.toml into 16eae28 on master.

[coveralls]

Coverage: 99.381% (+0.2%) from 99.196% when pulling bb8300f on build/pyproject.toml into 16eae28 on master.

[ssbarnea]

You must remove setup.py or you really fail to implement PEP-621 which is what the title claims. Still, if you want to make the switch in several PRs, it is ok.

[adrienverge]

@DimitriPapadopoulos @staticdev @ssbarnea @barrelful following your remarks from #509 and #557, could you test this pull request (at least python -m build, pip install and basic yamllint usage) and check whether everything works as you expect? Thanks in advance.

No feedback on this after one week, so I'll merge now.

[staticdev]

@adrienverge as commented by @ssbarnea, you need to delete setup.py to get it working properly.

[adrienverge]

Have a look at the new setup.py of this pull request:

# This is only kept for backward-compatibility with older versions that don't
# support new packaging standards (e.g. PEP 517 or PEP 660):
setup()

If you know another way to build / install the package with setuptools < 61 and without setup.py and you've tested it to get a working .rpm or .deb on "stable" Linux distributions like CentOS 8 (setuptools == 50; poetry not packaged), CentOS 9 (setuptools == 53) or Ubuntu 2020.04 (setuptools == 45; poetry not packaged), please let me know. Distro packaging is a requirement listed in #557 (comment).

Also it's not too late if you want to try and answer #566 (comment) 👍

[staticdev]

@adrienverge I don't have a solution using setuptools, maybe why I don't use it. But I know this is not compliant with PEP-517. To be compliant with all packaging peps, I would say only solution I know is hatch (a middle ground between poetry and setuptools-scm). It was just merged on a lib I am also maintainer: sigmavirus24/github3.py#1142

[adrienverge]

But I know this is not compliant with PEP-517.

I didn't find anything stating this in PEP 517, could you give more details?

My reading of the PEP suggests that it's OK to keep setup.py as a fallback:

If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py ... Projects may still wish to include a setup.py for compatibility with tools that do not use this spec.

[staticdev]

My mistake then, but I remember I actually had problems having both when I was packaging. You might have unintended behaviour since it can use setup.py or pyproject.toml as source of truth.

[DimitriPapadopoulos]

But setup.py only contains setup() here. Having a vastly different setup.py or setup.cfg would indeed be an issue.

[staticdev]

Yes, best thing now since it is merged is observe if there is any problem. The alternative solution in this case could be hatch. I am playing with it right now, looks good.

[staticdev]

Have a look at the new setup.py of this pull request:

# This is only kept for backward-compatibility with older versions that don't
# support new packaging standards (e.g. PEP 517 or PEP 660):
setup()

If you know another way to build / install the package with setuptools < 61 and without setup.py and you've tested it to get a working .rpm or .deb on "stable" Linux distributions like CentOS 8 (setuptools == 50; poetry not packaged), CentOS 9 (setuptools == 53) or Ubuntu 2020.04 (setuptools == 45; poetry not packaged), please let me know. Distro packaging is a requirement listed in #557 (comment).

Also it's not too late if you want to try and answer #566 (comment) +1

Wait, also this shouldn't be needed by those distros to install the package, just to build. What do you need to build yamllint in them since Github Actions can do it with latest setuptools?

[DimitriPapadopoulos]

You're right about it @staticdev. A recent version of setuptools is required to build:

yamllint/pyproject.toml

Lines 43 to 45 in 98f2281

	[build-system]
	build-backend = "setuptools.build_meta"
	requires = ["setuptools >= 61"]

The compatibility setup.py is mostly there for other older tools. Removing it won't break builds on older systems like CentOS 8 because setuptools must be updated before building.

[adrienverge]

The compatibility setup.py is mostly there for other older tools. Removing it won't break builds on older systems like CentOS 8 because setuptools must be updated before building.

Actually it's not that simple; for example for CentOS 8/9 packages (which I maintain), the only setuptools versions available are 50/53 (and obviously you cannot use pip nor anything from the internet to update anything), so we need a setup.py file (while waiting for newer setuptools version to arrive there).

[DimitriPapadopoulos]

The setup.py file is an empty shell, which just redirects to setuptools which needs to be able to interpret pyproject.toml. A newer setuptools is required to build from pyproject.toml, isn't it?

[adrienverge]

A newer setuptools is required to build from pyproject.toml, isn't it?

Yes 👍

I already spent some hours on this legacy/stable Linux distro packaging. Please share if you have a fully working .spec file to build the .rpm package for CentOS 8 (for example) where only setuptools == 50 is available (you'll probably have to use the old macros %py3_build and %py3_install) without a setup.py/setup.cfg file, tested and working and with all the files installed at the right place (without UNKNOWN-0.0.0 instead of yamllint-1.31.0 for instance).

[DimitriPapadopoulos]

Indeed, UNKNOWN-0.0.0is typical of an old setuptools.

The following paragraph from the Python Packaging User Guide is about installing from sources, but clearly it has to apply to building as well:

Ensure pip, setuptools, and wheel are up to date

While pip alone is sufficient to install from pre-built binary archives, up to date copies of the setuptools and wheel projects are useful to ensure you can also install from source archives:

Unless you manage to update to a newer setuptools, you might have to add the old setup.cfg to the RPM spec, as a patch, to build on older platforms 😠

[adrienverge]

Agreed 👍

For info, the latest setuptools documentation about pyproject.toml recommends what I've done in this commit:

Important

If compatibility with legacy builds or versions of tools that don’t support certain packaging standards (e.g. PEP 517 or PEP 660), a simple setup.py script can be added to your project [1] (while keeping the configuration in pyproject.toml):

from setuptools import setup

setup()