Clarity on installation from source

[jaraco]

Setuptools exists at a unique place in the Python ecosystem, and although this project aims to be a distributed third-party package like any other, because it is required to build many of the packages in the ecosystem, it has unique bootstrapping challenges, including:

• Attempt to unvendor dependencies (Move test dependencies to package metadata. #1981).
• Attempt to rely on PEP 517 for bootstrapping (Remove bootstrap script #2543).

I'd like to put together a story (documentation) about how to bootstrap a Python system to include Setuptools from source, including some integration tests that continuously validate these expectations and the associated constraints so that upstream platforms like Fedora and Debian can have a reliable means to supply Setuptools (only) when needed.

[hroncok]

pyproject.toml says setuptools is needed:

setuptools/pyproject.toml

Lines 2 to 5 in c121d28

	requires = [
	"setuptools >= 40.8; python_version > '3'",
	"wheel",
	]

With that, we cannot use any PEP 517/518 compliant builder if we don't already have setuptools.

My idea was to do something like:

export PYTHONPATH=/usr/lib64/python3.../ensurepip/_bundled/setuptools-...-...-none-any.whl

But that relies on the fact that for a certain part of our bootstrap process, we have the setuptools wheel bundled in Python. I can imagine others don't. Also, AFAIK ensurepip guarantees only pip, not setuptools.

[jaraco]

There is the backend-path feature which should allow setuptools to satisfy its own requirements, if only it had a means to bootstrap its own metadata in order to produce its own metadata:

setuptools/pyproject.toml

Line 7 in c121d28

backend-path = ["."]

[KOLANICH]

Let's assumme that

1. we have absolutely no other tools installed, including pip, build, its dep pep517, its deps, and flit_core required to build pep517;
2. we cannot use any wheels prebuilt by a third party in any form, i. e. from pypi and unpacked wheels in form of deb packages;
3. we won't use older versions of setuptools in order to build newer ones, it is a direct road to impractical bootstrap pipelines, like the ones needed to build Rust compiler and GCC.

So I tried to bootstrap setuptools the following way and it has failed:

git clone --depth=50 https://github.com/pypa/setuptools.git
Cloning into 'setuptools'...
+ git clone --depth=50 https://github.com/pypa/wheel.git
Cloning into 'wheel'...
+ cd ./setuptools
+ PYTHONPATH=../wheel/src python3 ./bootstrap.py
/tmp/setuptools/./bootstrap.py:7: UserWarning: bootstrap.py is no longer needed. Use a PEP-517-compatible builder instead.
  __name__ == '__main__' and warnings.warn(msg)
+ PYTHONPATH=../wheel/src python3 ./setup.py install
Traceback (most recent call last):
  File "/tmp/setuptools/./setup.py", line 162, in <module>
    dist = setuptools.setup(**setup_params)
  File "/tmp/setuptools/setuptools/__init__.py", line 152, in setup
    _install_setup_requires(attrs)
  File "/tmp/setuptools/setuptools/__init__.py", line 145, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/tmp/setuptools/setuptools/dist.py", line 680, in parse_config_files
    parse_configuration(self, self.command_options,
  File "/tmp/setuptools/setuptools/config.py", line 152, in parse_configuration
    options.parse()
  File "/tmp/setuptools/setuptools/config.py", line 463, in parse
    section_parser_method(section_options)
  File "/tmp/setuptools/setuptools/config.py", line 651, in parse_section_entry_points
    self['entry_points'] = parsed
  File "/tmp/setuptools/setuptools/config.py", line 210, in __setitem__
    raise KeyError(option_name)
KeyError: 'entry_points'

[jaraco]

#1644 is relevant here.

[jaraco]

* we have absolutely no other tools installed, including `pip`, `build`, its dep `pep517`, its deps, and `flit_core` required to build `pep517`;

This assumption is fine, but the way I'm thinking, the environment still needs a foundational tool that's not setuptools that implements PEP 517 and can resolve build dependencies and make them available for building. And ideally, that tool should support resolving these build dependencies even if those dependencies depend on Setuptools (as pip can through wheels).

My goal in this issue is to describe, demonstrate, and test how that would work. I don't yet have a great answer, but I'm working on it.

[KOLANICH]

I have solved the issue by just caching the egg-info. I think that we just need the bootstrap script back. Or maybe just to ship the info in the repo.

[jaraco]

Yep. I had a bit of a breakthrough today. In b968c01, I found one can add the minimal entry points as a separate set of metadata and they'll be sufficient when setuptools' own metadata has not been generated. It was working for me locally, but I see some CI failures, so it may require some more thinking.

[jaraco]

In #2582, I've managed to simplify the packaging, removing Setuptools build dependency on itself.

I've also started working on a technique to install from source following PEP 517 processes in this Dockerfile:

from ubuntu:focal

RUN apt update
RUN apt install -y python3 unzip wget git
RUN ln -s python3 /usr/bin/python

# install built copies of build-requires
RUN wget -P files https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl
RUN mkdir build-requires
RUN unzip -d build-requires files/wheel*

# download setuptools source from PyPI
RUN wget -P files https://files.pythonhosted.org/packages/8f/6b/0dcf95d95086ce459152e4c0ac306f2dbbcf984177a2b8b77b320ebfbf22/setuptools-53.1.0.tar.gz
RUN tar xfz files/setuptools*
RUN mkdir out

# Build Setuptools based on build-backend declaration
WORKDIR setuptools-53.1.0
ENV SETUPTOOLS_USE_DISTUTILS=local
# PYTHONPATH is '.' due to backend-path
RUN PYTHONPATH=.:/build-requires python -c "import setuptools.build_meta; setuptools.build_meta.build_wheel('/out')"
WORKDIR /

# Take the built wheel and install it
RUN mkdir /packages
RUN unzip -d packages /out/setuptools*.whl
CMD PYTHONPATH=/packages python

Unfortunately, this approach violates point 2

we cannot use any wheels prebuilt by a third party in any form, i. e. from pypi and unpacked wheels in form of deb packages;

And I've thought about it more and I don't think it's possible in a PEP 517 world. In PEP 517, there are two artifacts that can be produced, an sdist or a wheel. An sdist is just a source distribution, so isn't installable. A wheel is the only installable artifact produced by PEP 517.

The problem arises in that wheel has Setuptools as a build dependency and Setuptools has wheel as a build dependency. Best I can tell, you can't break this dependency in a PEP 517 world without involving a pre-built wheel.

I'll need to ponder this more.

[KOLANICH]

And I've thought about it more and I don't think it's possible in a PEP 517 world. In PEP 517, there are two artifacts that can be produced, an sdist or a wheel. An sdist is just a source distribution, so isn't installable. A wheel is the only installable artifact produced by PEP 517.

It is absolutely possible, given that python is an interpretable language (so packages not necessarily require compilation) and that a package is just a dir with files and that we can use packages that are not yet installed and that we can append the path to packages with any dir we like.

The problem arises in that wheel has Setuptools as a build dependency and Setuptools has wheel as a build dependency. Best I can tell, you can't break this dependency in a PEP 517 world without involving a pre-built wheel.

That is not quite a problem. I just clone the 3 repos, setuptools, wheel and pip, then build setuptools setting PYTHONPATH to recognize a non-installed wheel as a package, then install a self-built wheel of setuptools setting PYTHONPATH to recognize a non-installed pip as a package, then I build wheel and install it the same way, then I build pip and install it the same way with itself.

Here is my source: https://github.com/KOLANICH-GHActions/bootstrap-python-packaging/blob/master/action.sh

The problems appeared when setuptools bootstrap.py got broken an instead I was instructed to use a PEP 517 frontend. It and its deps wouldd likely also work without installation, but the installation script would have to be bloated with setting pythonpaths and it will be less modular. So I felt like it is easier to just cache the entry points to unbreak setuptools in its non-installed state.

[KOLANICH]

BTW, PEP 517 was itself a big mistake (and it was obvious since its introduction): now I have too install 3 tools (setuptools, flit_core, poetry-core) instead of just 1.

[jachymb]

So I tried to bootstrap setuptools the following way and it has failed:

I suffer the same error message when I attempt to build setuptools on my system (using python 3.8).

[KOLANICH]

@jachymb, there is a workaround in this issue. Clone KOLANICH-GHActions/bootstrap-python-packaging@a7f9b55 into egg-info dir

[jaraco]

That workaround shouldn't be necessary anymore with #2582 and the v54 release.