Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Python 3.11.1 breaks setuptools builds on Fedora #3761

Open
stratakis opened this issue Jan 9, 2023 · 21 comments
Open

[BUG] Python 3.11.1 breaks setuptools builds on Fedora #3761

stratakis opened this issue Jan 9, 2023 · 21 comments
Assignees
Labels
bug Needs Triage Issues that need to be evaluated for severity and status.

Comments

@stratakis
Copy link
Contributor

setuptools version

65.5.1-65.6.3

Python version

Python 3.11.1

OS

Fedora Linux

Additional environment information

This was observed in Fedora Linux as soon as we updated to 3.11.1

Description

Since python/cpython@07734a4 setuptools builds have started failing on Fedora.

It seems there are two issues I've discovered so far and bisecting the problem led me to the specific Python commit.

First when trying to build setuptools through the pypi tarball after an initial bootstrap build (meaning we delete the egg-info directory on the second build) and then our pyproject script tries to generate the build requirements we get a KeyError: 'entry_points' error.

Secondly if I add the build requirements manually and then create the wheel through python3 setup.py bdist_wheel after again removing the pre-generated egg-info directory I get an error: [Errno 2] No such file or directory: 'build/bdist.linux-x86_64/wheel/setuptools-65.6.3-py3.11.egg-info/PKG-INFO' error.

Tested it from 65.5.1 and on-wards. By removing that commit or by not removing the egg-info directory, everything works like before.

Expected behavior

Expected for the errors not to happen

How to Reproduce

For the first part:

On a Fedora 37 or Rawhide system
dnf install python3.11 which will bring you the latest Python 3.11.1 or compile CPython from the v3.11.1 tag
Get the setuptools tarball from pypi and unzip it
Delete the egg-info directory
dnf install '/usr/lib/rpm/redhat/pyproject_buildrequires.py'
And inside the dir do python3 -B /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3

By downgrading to 3.11.0 or skipping python/cpython@07734a4 the issue disappears.

For the second part:

A reproducer which should be non-fedora specific although it shows a different error.
Compile and/or install Python 3.11.1
In a system where setuptools is already installed.
Get the pypi tarball
Unzip it and delete the egg-info directory
Install wheel
python3 setup.py bdist_wheel

Same as before, if I downgrade to 3.11.0 or skip the specific commit the issue disappears.

Output

First part:

$ python3 -B /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3
/builddir/setuptools-65.6.3/setuptools/command/egg_info.py:643: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
Please extend command classes from setuptools instead of distutils.
  warnings.warn(
warning: no files found matching '*.py' under directory 'tests'
warning: no files found matching '*.txt' under directory 'docs'
warning: no files found matching '*.conf' under directory 'docs'
warning: no files found matching '*.css' under directory 'docs'
warning: no files found matching '*.css_t' under directory 'docs'
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
warning: no files found matching 'msvc-build-launcher.cmd'
HOOK STDOUT: running egg_info
HOOK STDOUT: creating setuptools.egg-info
HOOK STDOUT: writing manifest file 'setuptools.egg-info/SOURCES.txt'
HOOK STDOUT: reading manifest file 'setuptools.egg-info/SOURCES.txt'
HOOK STDOUT: reading manifest template 'MANIFEST.in'
HOOK STDOUT: adding license file 'LICENSE'
HOOK STDOUT: writing manifest file 'setuptools.egg-info/SOURCES.txt'
Handling wheel from get_requires_for_build_wheel
Requirement satisfied: wheel
   (installed: wheel 0.38.4)
python3dist(wheel)
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 508, in main
    generate_requires(
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 420, in generate_requires
    generate_run_requirements(backend, requirements, build_wheel=build_wheel, wheeldir=wheeldir)
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 330, in generate_run_requirements
    generate_run_requirements_hook(backend, requirements)
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 286, in generate_run_requirements_hook
    dir_basename = prepare_metadata('.')
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/setuptools-65.6.3/setuptools/build_meta.py", line 377, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/builddir/setuptools-65.6.3/setuptools/build_meta.py", line 335, in run_setup
    exec(code, locals())
  File "<string>", line 87, in <module>
  File "/builddir/setuptools-65.6.3/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/setuptools-65.6.3/setuptools/_distutils/core.py", line 159, in setup
    dist.parse_config_files()
  File "/builddir/setuptools-65.6.3/setuptools/dist.py", line 863, in parse_config_files
    setupcfg.parse_configuration(
  File "/builddir/setuptools-65.6.3/setuptools/config/setupcfg.py", line 162, in parse_configuration
    options.parse()
  File "/builddir/setuptools-65.6.3/setuptools/config/setupcfg.py", line 496, in parse
    section_parser_method(section_options)
  File "/builddir/setuptools-65.6.3/setuptools/config/setupcfg.py", line 724, in parse_section_entry_points
    self['entry_points'] = parsed
    ~~~~^^^^^^^^^^^^^^^^
  File "/builddir/setuptools-65.6.3/setuptools/config/setupcfg.py", line 268, in __setitem__
    raise KeyError(option_name)
KeyError: 'entry_points'

Second part:

$ python3 setup.py bdist_wheel
<snip>
running egg_info
creating setuptools.egg-info
writing manifest file 'setuptools.egg-info/SOURCES.txt'
reading manifest file 'setuptools.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.py' under directory 'tests'
warning: no files found matching '*.txt' under directory 'docs'
warning: no files found matching '*.conf' under directory 'docs'
warning: no files found matching '*.css' under directory 'docs'
warning: no files found matching '*.css_t' under directory 'docs'
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
warning: no files found matching 'msvc-build-launcher.cmd'
adding license file 'LICENSE'
writing manifest file 'setuptools.egg-info/SOURCES.txt'
Copying setuptools.egg-info to build/bdist.linux-x86_64/wheel/setuptools-65.6.3-py3.11.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/distutils-precedence.pth
error: [Errno 2] No such file or directory: 'build/bdist.linux-x86_64/wheel/setuptools-65.6.3-py3.11.egg-info/PKG-INFO'
@stratakis stratakis added bug Needs Triage Issues that need to be evaluated for severity and status. labels Jan 9, 2023
@stratakis
Copy link
Contributor Author

cc @hroncok

@pojoba02
Copy link

Also having the same issue with ubuntu

@jaraco
Copy link
Member

jaraco commented Mar 2, 2023

Thanks for the nudge. I'd forgotten about this issue and plan to look tomorrow.

@jaraco
Copy link
Member

jaraco commented Mar 3, 2023

The repro instructions aren't working for me. I created this Dockerfile:

from fedora:rawhide
run dnf install -y python3.11 python-launcher
run curl https://files.pythonhosted.org/packages/fe/31/2c88d37d4f8036c6648745e74ec51f73a6ba945a06e069601900f81fe00a/setuptools-67.4.0.tar.gz -o - | tar xz
run dnf install -y '/usr/lib/rpm/redhat/pyproject_buildrequires.py'
run ln -s setuptools* setuptools-src
workdir setuptools-src
run rm -r *.egg-info
cmd py -B /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3

And running it emits an error about packaging.

 draft $ docker run -it @$(docker build -q .)
Import error: No module named 'packaging'

Edit: Adding python-packaging to the installed packages works around that error and the KeyError reproduces.

@hroncok
Copy link
Contributor

hroncok commented Mar 3, 2023

run dnf install -y '/usr/lib/rpm/redhat/pyproject_buildrequires.py' python3-packaging

@jaraco
Copy link
Member

jaraco commented Mar 3, 2023

I tried entering a debugger where the issue occurs, but something about the pyproject_buildrequires.py script is interfering.

 draft $ docker run -it @$(docker build -q .) py -B -m pdb /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3
> /usr/lib/rpm/redhat/pyproject_buildrequires.py(1)<module>()
-> import glob
(Pdb) c
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 532, in main
    generate_requires(
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 439, in generate_requires
    generate_build_requirements(backend, requirements)
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 289, in generate_build_requirements
    new_reqs = get_requires()
               ^^^^^^^^^^^^^^
  File "/setuptools-67.4.0/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/setuptools-67.4.0/setuptools/build_meta.py", line 320, in _get_build_requires
    self.run_setup()
  File "/setuptools-67.4.0/setuptools/build_meta.py", line 335, in run_setup
    exec(code, locals())
  File "<string>", line 87, in <module>
  File "/setuptools-67.4.0/setuptools/__init__.py", line 107, in setup
    _install_setup_requires(attrs)
  File "/setuptools-67.4.0/setuptools/__init__.py", line 78, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/setuptools-67.4.0/setuptools/dist.py", line 881, in parse_config_files
    setupcfg.parse_configuration(
  File "/setuptools-67.4.0/setuptools/config/setupcfg.py", line 163, in parse_configuration
    options.parse()
  File "/setuptools-67.4.0/setuptools/config/setupcfg.py", line 508, in parse
    section_parser_method(section_options)
  File "/setuptools-67.4.0/setuptools/config/setupcfg.py", line 736, in parse_section_entry_points
    self['entry_points'] = parsed
    ~~~~^^^^^^^^^^^^^^^^
  File "/setuptools-67.4.0/setuptools/config/setupcfg.py", line 280, in __setitem__
    raise KeyError(option_name)
KeyError: 'entry_points'
The program exited via sys.exit(). Exit status: 1
> /usr/lib/rpm/redhat/pyproject_buildrequires.py(1)<module>()
-> import glob
(Pdb) Traceback (most recent call last):
  File "/usr/lib64/python3.11/pdb.py", line 1774, in main
    pdb._run(target)
  File "/usr/lib64/python3.11/pdb.py", line 1652, in _run
    self.run(target.code)
  File "/usr/lib64/python3.11/bdb.py", line 597, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 1, in <module>
    import glob
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 1, in <module>
    import glob
  File "/usr/lib64/python3.11/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/bdb.py", line 114, in dispatch_line
    self.user_line(frame)
  File "/usr/lib64/python3.11/pdb.py", line 340, in user_line
    self.interaction(frame, None)
  File "/usr/lib64/python3.11/pdb.py", line 435, in interaction
    self._cmdloop()
  File "/usr/lib64/python3.11/pdb.py", line 400, in _cmdloop
    self.cmdloop()
  File "/usr/lib64/python3.11/cmd.py", line 126, in cmdloop
    line = input(self.prompt)
           ^^^^^^^^^^^^^^^^^^
ValueError: I/O operation on closed file.
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /usr/lib64/python3.11/cmd.py(126)cmdloop()
-> line = input(self.prompt)
(Pdb) Traceback (most recent call last):
  File "/usr/lib64/python3.11/pdb.py", line 1774, in main
    pdb._run(target)
  File "/usr/lib64/python3.11/pdb.py", line 1652, in _run
    self.run(target.code)
  File "/usr/lib64/python3.11/bdb.py", line 597, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 1, in <module>
    import glob
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 1, in <module>
    import glob
  File "/usr/lib64/python3.11/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/bdb.py", line 114, in dispatch_line
    self.user_line(frame)
  File "/usr/lib64/python3.11/pdb.py", line 340, in user_line
    self.interaction(frame, None)
  File "/usr/lib64/python3.11/pdb.py", line 435, in interaction
    self._cmdloop()
  File "/usr/lib64/python3.11/pdb.py", line 400, in _cmdloop
    self.cmdloop()
  File "/usr/lib64/python3.11/cmd.py", line 126, in cmdloop
    line = input(self.prompt)
           ^^^^^^^^^^^^^^^^^^
ValueError: I/O operation on closed file.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib64/python3.11/pdb.py", line 1801, in <module>
    pdb.main()
  File "/usr/lib64/python3.11/pdb.py", line 1793, in main
    pdb.interaction(None, t)
  File "/usr/lib64/python3.11/pdb.py", line 435, in interaction
    self._cmdloop()
  File "/usr/lib64/python3.11/pdb.py", line 400, in _cmdloop
    self.cmdloop()
  File "/usr/lib64/python3.11/cmd.py", line 126, in cmdloop
    line = input(self.prompt)
           ^^^^^^^^^^^^^^^^^^
ValueError: I/O operation on closed file.

@jaraco
Copy link
Member

jaraco commented Mar 3, 2023

Looking at the traceback for the KeyError, there's nothing in the traceback that interacts with importlib.metadata. The failure occurs when entry_points doesn't exist in the ConfigOptionsHandler. I'm still trying to determine what factors affect that state.

I added this statement to the Dockerfile to enable debugging:

run sed -i -e 's/except Exception:/except Exception:\n        raise/' /usr/lib/rpm/redhat/pyproject_buildrequires.py

@jaraco
Copy link
Member

jaraco commented Mar 3, 2023

Aha. So part of the issue is that when parsing the configuration, a MinimalDistribution is present and it's that object that has no entry_points attribute.

@hroncok
Copy link
Contributor

hroncok commented Mar 3, 2023

I think we saw something similar in pypa/virtualenv#2493 (comment)

Since the problem only occurred at test-time, we solved that by deleting the prepare_metadata_for_build_wheel-produced .dist-info directory after reading the data from it. I wonder if pyproject_buildrequires.py could do that sooner, but since this fails right when the hook is invoked, I don't think so.

@jaraco
Copy link
Member

jaraco commented Mar 3, 2023

Good news. I've been able to replicate the failure without involving pyproject_buildrequires. Simply invoking egg_info on a tarball with the metadata removed triggers the failure:

from fedora:rawhide
run dnf install -y python3.11 python-launcher python-packaging
run curl https://files.pythonhosted.org/packages/fe/31/2c88d37d4f8036c6648745e74ec51f73a6ba945a06e069601900f81fe00a/setuptools-67.4.0.tar.gz -o - | tar xz
run ln -s setuptools* setuptools-src
workdir setuptools-src
run rm -r *.egg-info
cmd py setup.py egg_info

This repro should make debugging easier because it (a) leaves . on sys.path and (b) doesn't interfere with stdio, allowing for one to set breakpoints.

@jaraco
Copy link
Member

jaraco commented Mar 3, 2023

The place that entry_points gets set based on the metadata is here:

for ep in metadata.entry_points(group='distutils.setup_keywords'):
vars(self).setdefault(ep.name, None)

When the .egg-info directory is missing, entry_points never gets set on the MinimalDistribution and the KeyError occurs. Next to figure out why the importlib.metadata changes are relevant.

@jaraco
Copy link
Member

jaraco commented Mar 3, 2023

How confident are you that the importlib.metadata change is implicated? I downgraded importlib.metadata (using parent commit python/cpython@46a493e) using the steps in this Dockerfile:

from fedora:rawhide
run dnf install -y python3.11 python-launcher python-packaging
run curl https://files.pythonhosted.org/packages/fe/31/2c88d37d4f8036c6648745e74ec51f73a6ba945a06e069601900f81fe00a/setuptools-67.4.0.tar.gz -o - | tar xz
run ln -s setuptools* setuptools-src
# downgrade importlib.metadata
run curl https://raw.githubusercontent.com/python/cpython/46a493e2db942cada2dd1abd468d52725ceba992/Lib/importlib/metadata/__init__.py -o /usr/lib64/python3.11/importlib/metadata/__init__.py
workdir setuptools-src
run rm -r *.egg-info

cmd py setup.py egg_info

But the error still occurs.

Is there a way to easily get a Python 3.11.0 install?

@hroncok
Copy link
Contributor

hroncok commented Mar 3, 2023

Is there a way to easily get a Python 3.11.0 install?

dnf -y install https://kojipkgs.fedoraproject.org/packages/python3.11/3.11.0/1.fc38/x86_64/python3{,-libs}-3.11.0-1.fc38.x86_64.rpm

@hroncok
Copy link
Contributor

hroncok commented Mar 3, 2023

Is still get the KeyError with Python 3.11.0.

@hroncok
Copy link
Contributor

hroncok commented Mar 3, 2023

When Python was updated from 3.11.0 to 3.11.1 we started to see this failure. At that point, we had setuptools 65.5.1. Will try with that one.

@hroncok
Copy link
Contributor

hroncok commented Mar 3, 2023

Even with 65.5.1 and Python 3.11.0 I still get the KeyError. It has been more than a month since we tested this with @stratakis and I am quite sure it started on 3.11.1, but I really cannot get 3.11.0 to work now :(

@hroncok
Copy link
Contributor

hroncok commented Mar 3, 2023

I don't know what's happening here. I clearly see the point in history where this started in here: https://koschei.fedoraproject.org/package/python-setuptools?last_seen_ts=1671377122&collection=f39

but the logs are unfortunately already garbage-collected.

@stratakis
Copy link
Contributor Author

Also when we updated Python 3.10 in Fedora to a version which contained those change we observed the same failure again. I bisected the issue with my own reproducer at the time and it pointed to the importlib.metadata commit, so I would say fairly confident back then.

@jaraco
Copy link
Member

jaraco commented Mar 3, 2023

The importlib metadata commit fixed some bugs in metadata discovery, so potentially there was some corrupted or nonstandard metadata lying around on sys.path, maybe leaked from elsewhere. Since we’ve excluded it as a factor, I’m left thinking we should focus on the undersirable behavior directly and not as a regression.

To some extent, the failure is working as intended. Setuptools expects itself to be present as a build dependency… including its metadata. That’s why there’s a bootstrap.egg-info in the github repo—to enable the first build in a clean checkout. The fact that it’s failing to find entry_points during configure parsing is just an early indication that key functionality is missing.

You may have said it in the report already but I didn’t get it. Can you explain why you’re deleting the metadata?

@hroncok
Copy link
Contributor

hroncok commented Mar 29, 2023

Can you explain why you’re deleting the metadata?

In Fedora, we have a policy to not ship pre-generated things -- and we always deleted this, to ensure it's generated during the build. Until recently, it worked.

@stratakis
Copy link
Contributor Author

To add to this, I was testing setuptools 62.6.0 version which we have on Fedora 37 and the same issue occurs, however even by deleting the egg-info dir the problem is not solved. And of course it's the same drill, fails with Python 3.11.1 but succeeds with Python 3.11.0

liuyangxy pushed a commit to fedora-riscv/numpy that referenced this issue Apr 28, 2023
This leaks from setuptools: pypa/setuptools#3761

Resolves: rhbz#2183387
JasenChao pushed a commit to fedora-riscv/python-setuptools that referenced this issue Nov 4, 2023
Since Python 3.11.1, the setuptools builds are broken if the pre-
generated egg-info directory is removed.

We work around the issue by extracting the bootstrap.egg-info directory
from the github tarball.

Note that the bootstrap build doesn't work, however on F37 we won't bootstrap
anyway, so this is considered a workaround.

See also: pypa/setuptools#3761
leplatrem added a commit to Kinto/kinto-attachment that referenced this issue Jan 19, 2024
leplatrem added a commit to Kinto/kinto-attachment that referenced this issue Jan 23, 2024
* Modernize repo

* Run 'make format'

* Add changelog configuration

* Update releasing instructions

* Fix tests location and run moto on CI

* Remove readonly

* Run http server in background in functional test

* Install kinto_http for functional tests

* Enable setuptools-scm and remove useless MANIFEST.in

* Downgrade to python 3.11.0

See pypa/setuptools#3761 (comment)

* Use psycopg2-binary (only in tests)

* Get rid of tox

* Use kinto optional deps

* Try without extra dev deps

* Install dev deps in two steps

* Try @grahamalama's fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Needs Triage Issues that need to be evaluated for severity and status.
Projects
None yet
Development

No branches or pull requests

4 participants