Backport #702 for 5.4: Pin Cython build constraint to < 3.0

[idanmiara]

This is a backport of #702 + release of 5.4.2
Also closes #724

I'm not sure if release/5.4.1 is the correct branch to rebase this, but I didn't find a v5.4 branch or similar.

[idanmiara]

@nitzmahone please review😁

[nitzmahone]

After the fire-drill required to rebuild all the 6.0 wheels, I'm not planning to touch the 5.x codebases or packages. The packaging metadata is the easy part- the hard part is rebuilding and re-releasing all the same wheels without risking breakage (esp since a number of the worker images have been discontinued). If we release a new 5.x sdist, we have to re-release all the wheels or we've broken everyone else that was still working fine with those old versions.

For folks that need to build 5.x from sdist, I'd suggest using some combo of PIP_CONSTRAINTS or the git clone + sed hackery someone else mentioned to get the existing 5.x sdists to build while constraining the Cython version.

[idanmiara]

@nitzmahone thanks for looking into this and for your detailed response.
I'm thinking that since the binaries are unchanged maybe another solution could be to download them all, unpack them, change the version to 5.4.2 and repack. I can suggest a python script to do so.

[MrMino]

@nitzmahone IIUC re-releasing the sdist to PyPI wouldn't require re-releasing wheels. Am I missing something?

The issue with PIP_CONSTRAINTS and building from checkouts is that it doesn't really work for people who don't have control over transitive dependencies.

[idanmiara]

@nitzmahone IIUC re-releasing the sdist to PyPI wouldn't require re-releasing wheels. Am I missing something?

I also think so, assuming one didn't change their requirements, if pip fails to install 5.4.2 from source it should backtrack to 5.4.1 and install without a problem.

[MrMino]

What I'm talking about is re-releasing 5.4.1 sdist with the single Cython < 3.0 change added to pyproject.toml and build/macos.sh.

Right now the 5.4.1 sdist is basically unusable, but the rest of the wheels are fine.

[idanmiara]

What I'm talking about is re-releasing 5.4.1 sdist with the single Cython < 3.0 change added to pyproject.toml and build/macos.sh.

Right now the 5.4.1 sdist is basically unusable, but the rest of the wheels are fine.

Not possible. pypi refuses to reuse a version name.

[chlndh]

but e.g. docker-compose is requiring PyYAML < 6 and existing 5.4.1 is broken for me

[etomkova]

It's a bit unfortunate that this issue got closed because the problem with building the package essentially means that we're unable to build our application the usual way (poetry with a lock file - because the build dependencies of transitive dependencies are not locked).

EDIT: I now have an ugly workaround. As part of the dockerfile:

RUN \
  poetry run pip install "cython<3" && \
  poetry run pip install --no-build-isolation "pyyaml==5.4.1" # the exact version from poetry.lock 

[etomkova]

Would it be possible to release pre-built wheels for Python 3.10? That would solve the issue for us because then we wouldn't have to build the wheel. Thank you kindly for considering this.

[mrossi2023]

Hi guys,
we use 5.4.1 in our codebase and we have a PROD release early next week. We cannot upgrade to 6.0.1 . When will this version 5.4.1 be fixed?

[idanmiara]

Hi @nitzmahone
I see that many people are effected by this bug, see the amount of reaction to all the related issues and PRs.
Would you please reconsider my proposal #726 (comment) or otherwise explain what's the problem with just releasing a new sdist with or without re-releasing the same wheels with with a new version ?

[idanmiara]

Another idea is to release 5.4.2 as Python3.10+ sdist package so it wouldn't effect Python<=3.9 users and will fix the installation for Python 3.10+ users.

[Uzume]

@idanmiara: I think a 5.4.2 Python3.10+ sdist is a great idea but I would prefer a #731 (formerly #602) style fix instead of a #702 workaround port.

[idanmiara]

@Uzume
Any fix that that is agreed by the maintainers is fine by me, I just think that in order for this to be relevant we should seek a quick fix first.
I think that maybe a minimal fix i.e. pin Cython could be used for 5.4.2 and if needed, 5.5.0 or 5.4.3 could propose a proper longer term fix.
Are there any other maintainers for this project who might want to share their opinion on that matter?

[Uzume]

@idanmiara: By "other maintainers" I assume you are referring to @ingydotnet or @perlpunk? Those are the only other users listed as maintainers at PyPI (and mapping usernames to GH).

[perlpunk]

For the record, no opinion from me, because I'm not familiar enough with python releases and especially not cython. I concentrate mainly on YAML related stuff.

[Uzume]

@perlpunk: Thanks for the response. What you said may be true, however, you three are the only ones at PyPI that seem to have access to releases there (despite the greater number of people here in the GH org) and thus are the only ones capable to affecting changes there (besides PyPI admins themselves taking a direct hand in something).