Migrate kedro's setup.py to pyproject.toml

[idanov]

Description

The Python community is slowly moving to pyproject.toml as a package definition format and away from big and verbose setup.py files. Our Kedro package setup is still exclusively done in setup.py, with some parts in setup.cfg and pyproject.toml. We should move entirely to a pyproject.toml Python package definition for future-proofing our package setup.

Context

Adhering to the Python community standards is important in order to make it easier for future contributors to Kedro.

Possible Implementation

We can do that in one go and by creating a number of requirements.txt files as needed for the extras or dev dependencies. Here's a good starting documentation on how our pyproject.toml should look like: https://setuptools.pypa.io/en/latest/userguide/quickstart.html

Possible Alternatives

N/A

[merelcht]

We might be able to use this: https://github.com/asottile/setup-py-upgrade

[astrojuanlu]

Related: I spotted this while running kedro package on the spaceflights tutorial:

/Users/juan_cano/.micromamba/envs/kedro310/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.

We should get rid of setup.py everywhere.

[sbrugman]

We can do that in one go and by creating a number of requirements.txt files as needed for the extras or dev dependencies.

That works, but the idiomatic way to achieve this effect with pyproject.toml is to have an optional-dependencies section, obviating the need for separate files.

For hierarchical setting (e.g. different dependencies for distinct pipelines), one can have pyproject.toml files to extend/override in these subfolders.

[astrojuanlu]

Notice that this doesn't have to be done in one go: setuptools supports having pyproject.toml for static stuff, and setup.py or external files for dynamic stuff.

For example, pyproject.toml can refer to an external file for dependency/requirements.txt https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata so even if ideally we would be migrating it to be in pyproject.toml directly, it doesn't have to be done straight away, especially because that file is used in behave:

kedro/features/steps/cli_steps.py

Lines 397 to 403 in fd42a5a

	@given("I have updated kedro requirements")
	def update_kedro_req(context: behave.runner.Context):
	"""Replace kedro as a standalone requirement with a line
	that includes all of kedro's dependencies (-r kedro/requirements.txt)
	"""
	reqs_path = context.root_project_dir / "src" / "requirements.txt"
	kedro_reqs = f"-r {context.requirements_path.as_posix()}"

The part that can't be avoided for now then is extras_require, as explained above.

Hence a first step towards having mostly static metadata is fairly straightforward:

• Remove test_requires (it's deprecated)
• Replace python setup.py in Makefile by the appropriate command (pip install build && python -m build)
• Move everything except extras_require from setup.py to pyproject.toml (no more need to dynamically read README or version file, setuptools takes care of it)

[astrojuanlu]

Even though gh-2475 didn't technically obliterate setup.py, now that most of the project metadata is in pyproject.toml maybe we could just keep setup.py around for our extras mangling and close this issue. What do others think?

[merelcht]

@astrojuanlu It looks like all extra requirements are dataset related, so those will be removed anyway when datasets are removed from Kedro core in 0.19.0. So I'm guessing there isn't really anything we can do until then?

[astrojuanlu]

@merelcht good catch, yeah I agree!

[merelcht]

We have #2127 for that, so I think we can indeed close this issue as done 🙂