Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nspkg dependency issues during pip install with python 3 #3447

Closed
grahamegee opened this issue Sep 27, 2018 · 8 comments
Closed

nspkg dependency issues during pip install with python 3 #3447

grahamegee opened this issue Sep 27, 2018 · 8 comments

Comments

@grahamegee
Copy link

We have been experiencing very strange behaviour which I believe is related to #3412

some context.

We have a local pypi proxy in nexus, the weird behaviour seems to be limited to when we point to this private pypi.

Behaviour with Python 3.5

Nexus pypi

<redacted>@<redacted> ✗ pip install azure-storage
Looking in indexes: https://nexus.<redacted>/repository/pypi-all/simple
Collecting azure-storage
  Using cached https://nexus.<redacted>/repository/pypi-all/packages/76/26/482c033f8f3a48d16cf75aad91c3e1256856719f4117fabb4696d33aa5da/azure_storage-0.36.0-py2.py3-none-any.whl
Collecting cryptography (from azure-storage)
  Using cached https://nexus.<redacted>/repository/pypi-all/packages/59/32/92cade62c645756a83598edf56289e9b19aae5370642a7ce690cd06bc72f/cryptography-2.3.1-cp34-abi3-manylinux1_x86_64.whl
Collecting requests (from azure-storage)
  Using cached https://nexus.<redacted>/repository/pypi-all/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl
Collecting azure-nspkg>=2.0.0 (from azure-storage)
  Using cached https://nexus.<redacted>/repository/pypi-all/packages/1e/a4/84eb1251c2f3d2392bae60c7999f070ce1c68412880dfc80e644a854354e/azure_nspkg-3.0.0-py2.py3-none-any.whl
azure-nspkg requires Python '<3' but the running Python is 3.5.6

for some reason an attempt to pull azure-nspkg==3.0.0 happens even though we're running python 3. The install fails.

Public pypi

<redacted>@<redacted> ✓ pip install azure-storage
Collecting azure-storage
  Using cached https://files.pythonhosted.org/packages/76/26/482c033f8f3a48d16cf75aad91c3e1256856719f4117fabb4696d33aa5da/azure_storage-0.36.0-py2.py3-none-any.whl
Collecting requests (from azure-storage)
  Using cached https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl
Collecting azure-nspkg>=2.0.0 (from azure-storage)
  Using cached https://files.pythonhosted.org/packages/cd/a0/76ca6659ae9afd7567fdbb5b9c85e9c9a0b48238cfcacd92525484408f66/azure_nspkg-2.0.0-py2.py3-none-any.whl
Collecting azure-common>=1.1.5 (from azure-storage)
  Using cached https://files.pythonhosted.org/packages/ac/d3/055ce7ad06459a415ff9ca210e04c6cbb51bd6564815b7c8ac34bf5a1c39/azure_common-1.1.16-py2.py3-none-any.whl
Collecting python-dateutil (from azure-storage)
  Using cached https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl
Collecting cryptography (from azure-storage)
  Using cached https://files.pythonhosted.org/packages/59/32/92cade62c645756a83598edf56289e9b19aae5370642a7ce690cd06bc72f/cryptography-2.3.1-cp34-abi3-manylinux1_x86_64.whl
Collecting urllib3<1.24,>=1.21.1 (from requests->azure-storage)
  Using cached https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->azure-storage)
  Using cached https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->azure-storage)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.8,>=2.5 (from requests->azure-storage)
  Using cached https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil->azure-storage)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography->azure-storage)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.7 (from cryptography->azure-storage)
  Using cached https://files.pythonhosted.org/packages/59/cc/0e1635b4951021ef35f5c92b32c865ae605fac2a19d724fb6ff99d745c81/cffi-1.11.5-cp35-cp35m-manylinux1_x86_64.whl
Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography->azure-storage)
Installing collected packages: urllib3, certifi, chardet, idna, requests, azure-nspkg, azure-common, six, python-dateutil, asn1crypto, pycparser, cffi, cryptography, azure-storage
Successfully installed asn1crypto-0.24.0 azure-common-1.1.16 azure-nspkg-2.0.0 azure-storage-0.36.0 certifi-2018.8.24 cffi-1.11.5 chardet-3.0.4 cryptography-2.3.1 idna-2.7 pycparser-2.19 python-dateutil-2.7.3 re
quests-2.19.1 six-1.11.0 urllib3-1.23

azure_nspkg==2.0.0 is collected. The install works.

In #3412 the *_nspkg modules are added as requires_extra with a python<3 tag so I'm confused as to why they are being fetched at all during a pip install on python 3.

Current fix

Currently I have explicitly installed all required *_nspkg modules pinned at version 2.0.0 in the my respective setup.py files as a work around.

Help!

  • I have tried removing azure_nspkg==3.0.0 from nexus but it keeps coming back
  • I've tried invalidating the nexus cache
  • I've tried reindexing the nexus pypi repo

This was all working until 2 days ago which coincides with the changes introduced during #3412.

@lmazuel
Copy link
Member

lmazuel commented Sep 27, 2018

Hi @grahamegee

It turns out some "not-PyPI" implementation of PyPI-like server does not respect the expected protocol of communication. I got the same isuue with JFrog from Artifactory:
https://www.jfrog.com/jira/browse/RTFACT-15433

If you look at the source of this page:
https://pypi.org/simple/azure-nspkg/

You will see this:

<!DOCTYPE html>
<html>
  <head>
    <title>Links for azure-nspkg</title>
  </head>
  <body>
    <h1>Links for azure-nspkg</h1>
    <a href="https://files.pythonhosted.org/packages/3d/90/c9d3608ccbef4e95736b5a52ddb8cb67f4841e5d362ee1fa29e547e797ec/azure-nspkg-1.0.0rc1.zip#sha256=a1c2f54c484090ad58f7ca1612acbd357ee8844221f44e3fc72253a3d6aa32c6">azure-nspkg-1.0.0rc1.zip</a><br/>
    <a href="https://files.pythonhosted.org/packages/6c/e3/519e130fb967dfe2f6a07a4ca3065504d753a8aadb2f82dd1b224d654dfc/azure_nspkg-1.0.0rc1-py2.py3-none-any.whl#sha256=ec3eb0e8fedee778480de4a1beab601bf4ffa3ab523b9b341f2ff84961061717">azure_nspkg-1.0.0rc1-py2.py3-none-any.whl</a><br/>
    <a href="https://files.pythonhosted.org/packages/b7/38/bde8f56bd8d9db17eaa54c6b8dfd2f8b06368567a71b11c6d62c136c06b4/azure-nspkg-1.0.0rc2.zip#sha256=68429643e3bbfb02a14139bbaa409d15de317dd170690b9f16ab404534473baa">azure-nspkg-1.0.0rc2.zip</a><br/>
    <a href="https://files.pythonhosted.org/packages/a9/3d/c6a4a73f0a706ee22af083ac4bad9ebafd728ee06be480d6b0a575d3705e/azure_nspkg-1.0.0rc2-py2.py3-none-any.whl#sha256=6a45bef98a7d53196263f96c2cd8138005b56a43bd4bcc82960ea56b0ce889dd">azure_nspkg-1.0.0rc2-py2.py3-none-any.whl</a><br/>
    <a href="https://files.pythonhosted.org/packages/6c/bc/16e85022bef01d024cac48ad2d1bfe41279ca9a369faab30138e72d0ee1b/azure-nspkg-1.0.0.zip#sha256=293f286c15ea123761f30f5b1cb5adebe5f1e5009efade923c6dd1e017621bf7">azure-nspkg-1.0.0.zip</a><br/>
    <a href="https://files.pythonhosted.org/packages/48/bc/f89ff99a5ae18adfeba019a04c09e42eb8c66d792826a642d2d940919c50/azure_nspkg-1.0.0-py2.py3-none-any.whl#sha256=084804005cd14ae340c8f8d99895f8cd88d8b119ecf433b9e69d2b802911d8f7">azure_nspkg-1.0.0-py2.py3-none-any.whl</a><br/>
    <a href="https://files.pythonhosted.org/packages/06/a2/77820fa07ec4657d6456b67edfa78856b4789ada42d1bb8e8485df19824e/azure-nspkg-2.0.0.zip#sha256=fe19ee5d8c66ee8ef62557fc7310f59cffb7230f0a94701eef79f6e3191fdc7b">azure-nspkg-2.0.0.zip</a><br/>
    <a href="https://files.pythonhosted.org/packages/cd/a0/76ca6659ae9afd7567fdbb5b9c85e9c9a0b48238cfcacd92525484408f66/azure_nspkg-2.0.0-py2.py3-none-any.whl#sha256=4bd758e649f57cc188db4f3c64becaca16195e057e4362b6caad56fe1e7934e9">azure_nspkg-2.0.0-py2.py3-none-any.whl</a><br/>
    <a href="https://files.pythonhosted.org/packages/1c/db/7a5e6d37f90107e3c58de06394d421dc08e0f90fe0d40cc464e81911e9a8/azure-nspkg-3.0.0.zip#sha256=08c0c8a236616323ca72d566119e1614f4fa2f6ada420c8f5a5afb2fcaec18f5" data-requires-python="&lt;3">azure-nspkg-3.0.0.zip</a><br/>
    <a href="https://files.pythonhosted.org/packages/1e/a4/84eb1251c2f3d2392bae60c7999f070ce1c68412880dfc80e644a854354e/azure_nspkg-3.0.0-py2.py3-none-any.whl#sha256=413def61ecf6789388b4c0b2178661aa0b8f93257c9760cd691214942338045b" data-requires-python="&lt;3">azure_nspkg-3.0.0-py2.py3-none-any.whl</a><br/>
    </body>
</html>
<!--SERIAL 4309349-->

See that the last link have a special metadata: data-requires-python="&lt;3" that allows pip to filter this release.

Result is that pip will automatically select azure-nspkg 3.0 on Python 2, and azure-nspkg 2.0 on Python 3.

My guess is that nexus has the same bug as JFrog Artifactory, and does not expose these metadata correctly...

Note that you can expect more and more packages to come with that syntax. Django for instance, already switch automatically from 1.x versions in Python 2 to 2.x on Python 3. aiohttp users, can use 2.x in 3.4 and 3.x in 3.5 and up and declare it using the same syntax.

I'm not exactly sure what I can do myself, I respect all the setuptools protocol, and I use features released 2 years ago (November 2016...), which is not really cutting edge technology... :(

@lmazuel
Copy link
Member

lmazuel commented Sep 27, 2018

Side note @grahamegee , azure-storage is a deprecated package.

@lmazuel
Copy link
Member

lmazuel commented Sep 27, 2018

As workaround, you use the pip flag --ignore-requires-python, to tell pip that you're not concerned about this package.
There might be an impact on initial loading performance, and it's why I added the metadata to not install it on Python3. But it won't break anything.

@grahamegee
Copy link
Author

grahamegee commented Sep 28, 2018

@lmazuel interesting... I had a feeling that this might have been the wrong place to raise an issue.

I've done some digging in nexus and had a look at the attributes on the v3 package.

nexus pypi

You can see right down the bottom that requires_python < 3 so the metadata does exist in some form, of course it doesn't necessarily mean that:

  1. It's correctly formed.
  2. It's actually used.

I guess it's worth mentioning that I'm running Nexus OSS 3.2.0-01. Maybe this is fixed in a newer version.
I'll post back here if I found out more. Thanks for your help!

@grahamegee
Copy link
Author

I've raised an issue with sonatype https://issues.sonatype.org/browse/NEXUS-18117.

@johnarnold
Copy link
Member

FYI, this broke us too, using devpi for package server.

@lmazuel
Copy link
Member

lmazuel commented Sep 28, 2018

@johnarnold I see the fix in the release of 4.7.0 of devpi
https://devpi.net/docs/devpi/devpi/stable/+d/changelog.html#id44

fix #511: support PEP508 requires-python attribute on links. Thanks to Sergey Kolosov for the parsing parts.

devpi/devpi#511

Could you confirm your version of devpi?

@lmazuel
Copy link
Member

lmazuel commented Oct 2, 2018

Released azure-nspkg 3.0.1 and azure-mgmt-nspkg 3.0.1 that does not use python_requires, but install an empty Python 3 package with no file instead.

There is too many people impacted, even if I did nothing wrong, and it's third-party issues, to just do nothing. Hopefully this fixes all issues, so closing here.

@lmazuel lmazuel closed this as completed Oct 2, 2018
@github-actions github-actions bot locked and limited conversation to collaborators Apr 13, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants