Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

version conflict in setuptools-rust when compiling cryptography #6671

Closed
tseeling opened this issue Dec 1, 2021 · 26 comments
Closed

version conflict in setuptools-rust when compiling cryptography #6671

tseeling opened this issue Dec 1, 2021 · 26 comments

Comments

@tseeling
Copy link

tseeling commented Dec 1, 2021

Linux s390x (zLinux on IBM Mainframe)
Redhat 7.9
Python 3.6.12
rustc 1.52.1 (Red Hat 1.52.1-2.el7_9)
cryptography 36.0.0
cffi 1.15.0
setuptools 59.4.0
pip 21.3.1
wheel 0.37.0

ERROR: Some build dependencies for setuptools-rust>=0.11.4 from https://files.pythonhosted.org/packages/22/fa/ac431f0571bd84d8da5b55d9c9a00d510ae8682324405eabd96a752d090c/setuptools-rust-1.0.0.tar.gz#sha256=eb6e6d41f01e63d042288e5c609684adf93426c10ea745ec324263f3fcbf27e9 conflict with the backend dependencies: setuptools_scm==6.0.1 is incompatible with setuptools_scm>=6.3.2.

The installer then tries to fall back and finally succeeds for cryptography 3.3.2.

@alex
Copy link
Member

alex commented Dec 1, 2021 via email

@reaperhulk
Copy link
Member

setuptools_scm is a dep of setuptools_rust. There's a setuptools-rust 1.1.1 now though, so the first step here is probably just to see what happens in a fresh virtualenv with the latest packages?

@tseeling
Copy link
Author

tseeling commented Dec 1, 2021

The final goal is building synapse, a matrix server. The relevant part of my build script is this:

# (sourcing lots of redhat SCL stuff for gcc-10, rust, python 3 etc.)
  python -m venv ${pyenv}
  cd ${pyenv}
  source bin/activate

# clean gcc environment
  unset CPPFLAGS LDFLAGS

  msg "# build in ${pyenv}"
  python -m pip install --upgrade pip
  python -m pip install --upgrade wheel
  python -m pip install --upgrade urllib3
  python -m pip install --upgrade setuptools

# we need openssl 1.1.1 for cryptography
# so we use a local install

  source ~/openssl-home.sh
  CFLAGS="-I${ssldir}/include" LDFLAGS="-L${ssldir}/lib" pip wheel --no-binary :all: cryptography

@tseeling
Copy link
Author

tseeling commented Dec 1, 2021

upgrading the base tools already reports this:

Successfully installed pip-21.3.1 semantic-version-2.8.5 setuptools-59.4.0 **setuptools-rust-1.1.1** tomli-1.2.2 typing-extensions-4.0.1 urllib3-1.26.7 wheel-0.37.0

@reaperhulk
Copy link
Member

You're going to need to figure out where in your dependency chain that a setuptools_scm pin is occurring. The failure is because setuptools_scm==6.0.1 is incompatible with setuptools_scm>=6.3.2 and cryptography and its own dependencies do not pin the version. Something else in your environment, apparently, does.

Since this is purely a setup requirement you could also build a cryptography wheel separately and then install that wheel in the env you need it in.

@ddormer
Copy link

ddormer commented Dec 2, 2021

I think this is actually a problem with cyclic build dependencies between toml and flit with the setup tools error being a red herring. I can reproduce this by running pip install cryptography --no-binary ":all:" and the first error is about toml, I've added an edited output at the end of this comment.

I don't know how to bootstrap a pip build environment to include these build dependencies, the best I could find at the moment is to have toml and flit installed (via wheel) and then build cryptography with pip's --no-build-isolation flag.


Looking in indexes: https://pypi.python.org/simple/
Collecting cryptography
  Using cached cryptography-36.0.0.tar.gz (571 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/darren/.virtualenvs/cryptography-test-build/bin/python /private/var/folders/f9/h7rjr99x5pl_rs33yt2w22800000gn/T/pip-standalone-pip-z342c1_7/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/f9/h7rjr99x5pl_rs33yt2w22800000gn/T/pip-build-env-gmn476qs/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.python.org/simple/ -- 'setuptools>=40.6.0' wheel 'cffi>=1.12; platform_python_implementation != '"'"'PyPy'"'"'' 'setuptools-rust>=0.11.4'
       cwd: None
  Complete output (177 lines):
  
 ..<SNIP>..

  Collecting tomli>=1.2.1
    Using cached tomli-1.2.2.tar.gz (15 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 2:
     command: /Users/darren/.virtualenvs/cryptography-test-build/bin/python /private/var/folders/f9/h7rjr99x5pl_rs33yt2w22800000gn/T/pip-standalone-pip-z342c1_7/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/f9/h7rjr99x5pl_rs33yt2w22800000gn/T/pip-build-env-w205m29d/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.python.org/simple/ -- 'flit_core>=3.2.0,<4'
         cwd: None
    Complete output (60 lines):
    Looking in indexes: https://pypi.python.org/simple/
    Collecting flit_core<4,>=3.2.0
      Using cached flit_core-3.5.1.tar.gz (27 kB)
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
      Preparing metadata (pyproject.toml): started
      Preparing metadata (pyproject.toml): finished with status 'done'
    Collecting tomli
      Using cached tomli-1.2.2.tar.gz (15 kB)
    ERROR: Exception:
    Traceback (most recent call last):
      File "/private/var/folders/f9/h7rjr99x5pl_rs33yt2w22800000gn/T/pip-standalone-pip-z342c1_7/__env_pip__.zip/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
       
    ...<SNIP>...

      File "/private/var/folders/f9/h7rjr99x5pl_rs33yt2w22800000gn/T/pip-standalone-pip-z342c1_7/__env_pip__.zip/pip/_internal/req/req_tracker.py", line 92, in add
        raise LookupError(message)
    LookupError: https://files.pythonhosted.org/packages/aa/5b/62165da80cbc6e1779f342234c7ddc6c6bc9e64cef149046a9c0456f912b/tomli-1.2.2.tar.gz#sha256=c6ce0015eb38820eaf32b5db832dbc26deb3dd427bd5f6556cf0acac2c214fee (from https://pypi.org/simple/tomli/) (requires-python:>=3.6) is already being built: tomli>=1.2.1 from https://files.pythonhosted.org/packages/aa/5b/62165da80cbc6e1779f342234c7ddc6c6bc9e64cef149046a9c0456f912b/tomli-1.2.2.tar.gz#sha256=c6ce0015eb38820eaf32b5db832dbc26deb3dd427bd5f6556cf0acac2c214fee (from setuptools-rust>=0.11.4)
    ----------------------------------------
  WARNING: Discarding https://files.pythonhosted.org/packages/aa/5b/62165da80cbc6e1779f342234c7ddc6c6bc9e64cef149046a9c0456f912b/tomli-1.2.2.tar.gz#sha256=c6ce0015eb38820eaf32b5db832dbc26deb3dd427bd5f6556cf0acac2c214fee (from https://pypi.org/simple/tomli/) (requires-python:>=3.6). Command errored out with exit status 2: /Users/darren/.virtualenvs/cryptography-test-build/bin/python /private/var/folders/f9/h7rjr99x5pl_rs33yt2w22800000gn/T/pip-standalone-pip-z342c1_7/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/f9/h7rjr99x5pl_rs33yt2w22800000gn/T/pip-build-env-w205m29d/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.python.org/simple/ -- 'flit_core>=3.2.0,<4' Check the logs for full command output.

@reaperhulk
Copy link
Member

@ddormer I can also reproduce this on my machine. tomli depends on flit-core and flit-core depends on tomli and this appears to defeat the setuptools build backend. I see that this is somewhat documented by flit (https://github.com/pypa/flit/blob/353f920b7a1a95c28ba1234d275324c351d6f4a3/doc/bootstrap.rst) so I'm not sure what the right path is here. It may be that users that want to build purely from source and use no wheels will have to handle this themselves.

It's plausible that this is the same error as the original reporter, but I'd like confirmation of that.

@tseeling
Copy link
Author

tseeling commented Dec 2, 2021

My environment contains tomli and flit, so I'm lost at what I could do to have a successfull compile.
I admit that my build platform is rather exotic but I'm stuck here.

You're going to need to figure out where in your dependency chain that a setuptools_scm pin is occurring. The failure is because setuptools_scm==6.0.1 is incompatible with setuptools_scm>=6.3.2 and cryptography and its own dependencies do not pin the version. Something else in your environment, apparently, does.

Since this is purely a setup requirement you could also build a cryptography wheel separately and then install that wheel in the env you need it in.

I already try to build cryptography by itself and not during the synapse compile.
How would I find out the file in which that == version pin is used? Could I edit that out (i.e. make it >= instead) and compile locally?

@alex
Copy link
Member

alex commented Dec 3, 2021

Well. This is a headscratcher. I'm not even sure who to file a bug against about this, for all I know this is working as intended.

I think your best bet is probably to just use the wheel package of tomli. It's a universal wheel, so it'll work on any platform. If this is a security concern around using binary packages, I'd probably write a small script that verifies that a pure-python wheel contains the same source as the tarball.

@tseeling
Copy link
Author

tseeling commented Dec 4, 2021

I tried to build flit_core per the instructions on bootstrapping there but I have to admit I'm not an expert at python packaging. I still have no idea where this ==6.0.1 comes from and I don't really have a clue how to inject one local file into my build process.
Can you please give me a little hands-on how I should continue?
Is it possible to find out the offending package which breaks the build process?
Why would someone force an exact version of an intermediate dependency package?

alex added a commit to alex/setuptools-rust that referenced this issue Dec 4, 2021
Depending on tomli produces circular dependencies when attempting to install from sdists only: pyca/cryptography#6671
@alex
Copy link
Member

alex commented Dec 4, 2021

I've submitted PyO3/setuptools-rust#200 to setuptools-rust to drop their dependency on tomli to avoid this whole mess.

alex added a commit to alex/setuptools-rust that referenced this issue Dec 4, 2021
Depending on tomli produces circular dependencies when attempting to install from sdists only: pyca/cryptography#6671
alex added a commit to alex/setuptools-rust that referenced this issue Dec 4, 2021
Depending on tomli produces circular dependencies when attempting to install from sdists only: pyca/cryptography#6671
@alex
Copy link
Member

alex commented Dec 5, 2021

Can you confirm that this works with the latest setuptools-rust release?

@alex
Copy link
Member

alex commented Dec 7, 2021

This appears to have fixed the same issue for homebrew. On the presumption that that carries over, I'm going to close. If you still experience the issue please levae a comment and we'll reopen.

@alex alex closed this as completed Dec 7, 2021
@tseeling
Copy link
Author

tseeling commented Dec 8, 2021

Can you confirm that this works with the latest setuptools-rust release?

Doesn't work for me, still falls back to cryptography-3.3.2.
Still contains the hard dependency on the exact version number 6.0.1.

Collecting cryptography
  Using cached cryptography-36.0.0.tar.gz (571 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /homes/DPAORNET/seelinth-admin/cryptography/bin/python /homes/DPAORNET/seelinth-admin/cryptography/lib64/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dnb2brdn/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.12; platform_python_implementation != '"'"'PyPy'"'"'' 'setuptools-rust>=0.11.4'
       cwd: None
  Complete output (24 lines):
  Collecting setuptools>=40.6.0
    Downloading setuptools-59.5.0.tar.gz (2.3 MB)
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting wheel
    Using cached wheel-0.37.0.tar.gz (65 kB)
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Collecting cffi>=1.12
    Using cached cffi-1.15.0.tar.gz (484 kB)
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Collecting setuptools-rust>=0.11.4
    Downloading setuptools-rust-1.1.2.tar.gz (285 kB)
    Installing build dependencies: started
    Installing build dependencies: still running...
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
ERROR: Some build dependencies for setuptools-rust>=0.11.4 from https://files.pythonhosted.org/packages/67/08/e1aa2c582c62ac76e4d60f8e454bd3bba933781a06a88b4e38797445822a/setuptools-rust-1.1.2.tar.gz#sha256=a0adb9b503c0ffc4e8fe80b7c617898cefa78049983aaaea7f747e153a3e65d1 conflict with the backend dependencies: setuptools_scm==6.0.1 is incompatible with setuptools_scm>=6.3.2

@alex
Copy link
Member

alex commented Dec 8, 2021

Are any new steps required to reproduce this? Following your original instructions I was able to successfully install cryptography 36.

@tseeling
Copy link
Author

tseeling commented Dec 9, 2021

Are any new steps required to reproduce this? Following your original instructions I was able to successfully install cryptography 36.

Same script to compile, removed explicit flit and toml install.

rm -fr ~/cryptography
python -m venv ~/cryptography
cd ~/cryptography
source bin/activate

pip install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org pytest-xdist
python -m pip install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org --upgrade pip

python -m pip install --upgrade pip wheel urllib3 setuptools setuptools-rust
openssl version
export ssldir=/homes/DPAORNET/seelinth-admin/opt/synapse/openssl
CFLAGS="-I${ssldir}/include" LDFLAGS="-L${ssldir}/lib" pip wheel --no-binary :all: cryptography

Complains all the way down until it reaches 3.3.2 which it then compiles.
I did a find + grep in the cryptography directory and there was no ==6.0.1 version pinning to be found. How could I identify the location of this dreaded setting?

@tseeling
Copy link
Author

tseeling commented Dec 9, 2021

Updated to node.js 16.13.1, npm 8.2.0. Still the same error. Is there some cleanup I can do to make sure I really do a fresh start from all previous problems?
ERROR: Some build dependencies for setuptools-rust>=0.11.4 from https://files.pythonhosted.org/packages/67/08/e1aa2c582c62ac76e4d60f8e454bd3bba933781a06a88b4e38797445822a/setuptools-rust-1.1.2.tar.gz#sha256=a0adb9b503c0ffc4e8fe80b7c617898cefa78049983aaaea7f747e153a3e65d1 conflict with the backend dependencies: setuptools_scm==6.0.1 is incompatible with setuptools_scm>=6.3.2.

@alex alex reopened this Dec 9, 2021
@alex
Copy link
Member

alex commented Dec 9, 2021

FWIW, here's what I get:

/t/y ❯❯❯ python3 -m venv .venv
/t/y ❯❯❯ source .venv/bin/activate.fish
(.venv) /t/y ❯❯❯ python -m pip install -U pip
Requirement already satisfied: pip in ./.venv/lib/python3.9/site-packages (21.3.1)
(.venv) /t/y ❯❯❯ CRYPTOGRAPHY_SUPPRESS_LINK_FLAGS="1" LDFLAGS="/opt/homebrew/opt/openssl@1.1/lib/libcrypto.a /opt/homebrew/opt/openssl@1.1/lib/libssl.a" CFLAGS="-I/opt/homebrew/opt/openssl@1.1/include" pip install -v --no-binary :all: cryptography==36.0.0
Using pip 21.3.1 from /private/tmp/y/.venv/lib/python3.9/site-packages/pip (python 3.9)
Collecting cryptography==36.0.0
[...]
Successfully installed cffi-1.15.0 cryptography-36.0.0 pycparser-2.21

@tseeling
Copy link
Author

tseeling commented Dec 9, 2021

Your usage of LDFLAGS looks strange, but be it. I notice you use pip install and it still doesn't work for me on s390x with your command line.

  Collecting setuptools-rust>=0.11.4
    Using cached setuptools-rust-1.1.2.tar.gz (285 kB)
    Installing build dependencies: started
    Installing build dependencies: still running...
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
  ERROR: Some build dependencies for setuptools-rust>=0.11.4 from https://files.pythonhosted.org/packages/67/08/e1aa2c582c62ac76e4d60f8e454bd3bba933781a06a88b4e38797445822a/setuptools-rust-1.1.2.tar.gz#sha256=a0adb9b503c0ffc4e8fe80b7c617898cefa78049983aaaea7f747e153a3e65d1 conflict with the backend dependencies: setuptools_scm==6.0.1 is incompatible with setuptools_scm>=6.3.2.
  Installing build dependencies ... error
WARNING: Discarding https://files.pythonhosted.org/packages/60/06/d9109aba62c0b42466195e5b9b30dded26621a675b73998218070d8cc637/cryptography-36.0.0.tar.gz#sha256=52f769ecb4ef39865719aedc67b4b7eae167bafa48dbc2a26dd36fa56460507f (from https://pypi.org/simple/cryptography/) (requires-python:>=3.6). Command errored out with exit status 1: /homes/DPAORNET/seelinth-admin/cryptography/bin/python /homes/DPAORNET/seelinth-admin/cryptography/lib64/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-3tvez9ox/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.12; platform_python_implementation != '"'"'PyPy'"'"'' 'setuptools-rust>=0.11.4' Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement cryptography==36.0.0 (from versions: 0.1, 0.2, 0.2.1, 0.2.2, 0.3, 0.4, 0.5, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6, 0.6.1, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.9, 0.9.1, 0.9.2, 0.9.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.1.1, 1.1.2, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.4, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.6, 1.7, 1.7.1, 1.7.2, 1.8, 1.8.1, 1.8.2, 1.9, 2.0, 2.0.1, 2.0.2, 2.0.3, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2, 2.2.1, 2.2.2, 2.3, 2.3.1, 2.4, 2.4.1, 2.4.2, 2.5, 2.6, 2.6.1, 2.7, 2.8, 2.9, 2.9.1, 2.9.2, 3.0, 3.1, 3.1.1, 3.2, 3.2.1, 3.3, 3.3.1, 3.3.2, 3.4, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.4.5, 3.4.6, 3.4.7, 3.4.8, 35.0.0, 36.0.0)
ERROR: No matching distribution found for cryptography==36.0.0

@alex
Copy link
Member

alex commented Dec 9, 2021 via email

@tseeling
Copy link
Author

tseeling commented Dec 9, 2021

How can I locate the package and file where this ==6.0.1 pinning takes place? The error message is very detailed but it's missing who tries to enforce this.
I also tried to export the compiler flags before calling python -m pip install but it didn't make a difference.

@ddormer
Copy link

ddormer commented Dec 9, 2021

@alex The setuptools-rust change fixed my cryptography build issues but while trying to reproduce @tseeling's issue, I noticed that there's still technically a dependency problem between flit and tomli (pulled in by cryptography: setuptools_scm[toml]>=3.4.3 -> setuptools_scm[toml]: setuptools>=42 -> setuptools: tomli>=1.0)
Pip successfully resolves the problem by downgrading tomli to version 1.0.1 which is using poetry and not flit; Perhaps this is related to @tseeling's issue?

@tseeling Perhaps you could add the pip argument: -vvv and provide the full output of the failed cryptography build?

@tiran
Copy link
Contributor

tiran commented Dec 9, 2021

The blog posts https://iscinumpy.dev/post/bound-version-constraints/ sheds some light on the general issue with version pins. Poetry strongly recommends upper version pins.

@tseeling
Copy link
Author

Now trying to compile on x86_64 with CentOS 8. Same issue.

ERROR: Some build dependencies for setuptools-rust>=0.11.4 from https://files.pythonhosted.org/packages/67/08/e1aa2c582c62ac76e4d60f8e454bd3bba933781a06a88b4e38797445822a/setuptools-rust-1.1.2.tar.gz#sha256=a0adb9b503c0ffc4e8fe80b7c617898cefa78049983aaaea7f747e153a3e65d1 conflict with the backend dependencies: setuptools_scm==6.0.1 is incompatible with setuptools_scm>=6.3.2.

@alex
Copy link
Member

alex commented Feb 13, 2022

I've just attempted to reproduce this again with: pip install -vv --no-binary :all: cryptography==36.0.1 which ran fine.

Are folks still experiencing this issue?

@alex
Copy link
Member

alex commented Mar 8, 2022

No answer, since I couldn't reproduce I'm going to close. Please leave a comment if this is still an issue.

@alex alex closed this as completed Mar 8, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 7, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests

5 participants