Load build backend in isolated build environment with --no-index given

[cyrixsimon]

• Pip version: 10.0.1
• Python version: 3.6.5
• Operating system: openSUSE

Description:

Since version 10 pip raises an exception during installation for some python packages.

pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools

Although the setuptools are installed.

setuptools          39.1.0     
setuptools-scm      2.0.0  

With pip version 9 everything works fine.

What I've run:

One python package which fails with pip installation is dateutil.

# /usr/local/python/bin/pip3.6 install --verbose --no-index .
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-titakidg
Created temporary directory: /tmp/pip-install-idx4_e_v
Processing /tmp/workspace/python-dateutil-2.7.2
  Created temporary directory: /tmp/pip-req-build-h_jueb1s
  Created temporary directory: /tmp/pip-build-env-lz7qvsgc
  0 location(s) to search for versions of setuptools:
  Could not find a version that satisfies the requirement setuptools (from versions: )
Cleaning up...
  Removing source in /tmp/pip-req-build-h_jueb1s
No matching distribution found for setuptools
Exception information:
Traceback (most recent call last):
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
    self.require_hashes
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/operations/prepare.py", line 324, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/operations/prepare.py", line 150, in prep_for_dist
    build_requirements)
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/operations/prepare.py", line 59, in _install_build_reqs
    for r in build_requirements
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/operations/prepare.py", line 59, in <listcomp>
    for r in build_requirements
  File "/usr/local/python/lib/python3.6/site-packages/pip-18.0.dev0-py3.6.egg/pip/_internal/index.py", line 533, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools

[jaimguer]

I'm seeing the same with dateutil, --no-index, and --find-links on Amazon Linux 2017.03 and Python 3.4.3. I have setuptools-39.1.0.zip downloaded in the --find-links directory as well.

sudo /usr/local/bin/pip3 install -vvv -r requirements.txt --no-index --find-links /tmp/deps/                                                                                                                  
Collecting python-dateutil==2.7.3 (from -r /tmp/requirements.txt (line 58))                                                                                                                                   
  0 location(s) to search for versions of python-dateutil:                                                                                                                                                    
  Found link file:///mnt/tmp/deps/python-dateutil-2.7.3.tar.gz, version: 2.7.3                                                                                                                                
  Local files found: /mnt/tmp/deps/python-dateutil-2.7.3.tar.gz                                                                                                                                               
  Using version 2.7.3 (newest of versions: 2.7.3)                                                                                                                                                             
  Created temporary directory: /mnt/tmp/pip-build-env-pm8ysmpg                                                                                                                                                
  0 location(s) to search for versions of setuptools:                                                                                                                                                         
  Could not find a version that satisfies the requirement setuptools (from versions: )                                                                                                                        
Cleaning up...                                                                                                                                                                                                
  Removing source in <dirs>...                                                                                                                                                                                
No matching distribution found for setuptools                                                                                                                                                                 
Exception information:                                                                                                                                                                                        
Traceback (most recent call last):                                                                                                                                                                            
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/basecommand.py", line 228, in main                                                                                                               
    status = self.run(options, args)                                                                                                                                                                          
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/commands/install.py", line 291, in run                                                                                                           
    resolver.resolve(requirement_set)                                                                                                                                                                         
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/resolve.py", line 103, in resolve                                                                                                                
    self._resolve_one(requirement_set, req)                                                                                                                                                                   
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one                                                                                                           
    abstract_dist = self._get_abstract_dist_for(req_to_install)                                                                                                                                               
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for                                                                                                 
    self.require_hashes                                                                                                                                                                                       
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 324, in prepare_linked_requirement                                                                                  
    abstract_dist.prep_for_dist(finder, self.build_isolation)                                                                                                                                                 
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 150, in prep_for_dist                                                                                               
    build_requirements)                                                                                                                                                                                       
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 59, in _install_build_reqs                                                                                          
    for r in build_requirements                                                                                                                                                                               
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/operations/prepare.py", line 59, in <listcomp>                                                                                                   
    for r in build_requirements                                                                                                                                                                               
  File "/usr/local/lib/python3.4/site-packages/pip/_internal/index.py", line 533, in find_requirement                                                                                                         
    'No matching distribution found for %s' % req                                                                                                                                                             
pip._internal.exceptions.DistributionNotFound: No matching distribution found for setuptools

[hynek]

Just as a data point, I ran into this error by adding a pyproject.toml file that only contained configuration for black. Pinning pip to <10 fixed it and sadly I can't investigate any further.

[dstufft]

I think if you add a pyproject.toml, you need to add the build dependencies too, so adding

[build-system]
requires = ["setuptools", "wheel"]

should fix it.

[benoit-pierre]

The presence of pyproject.toml triggers PEP 518 support, and build isolation. You won't be able to install such a project with pip if wheels for setuptools/wheel are not available: no index and not in cache, or forcing source installs. You can disable build isolation with --no-build-isolation.

[hynek]

Oh I think @benoit-pierre has it. All of us seem to be running --no-index and that can’t work anymore. So if the others have a pyproject.toml too, this is rather about docs and/or refusing no-index in in build isolation mode?

[benoit-pierre]

Using --no-index does not necessarily mean it will fails: if you use --find-links with a directory providing wheels for setuptools/wheel. Next version of pip should also support installing build dependencies from source (see #5336).

[pradyunsg]

Indeed. This is exactly the case that makes me think #5416 would be a good idea.

[pfmoore]

I don't think that #5416 is relevant here. If you have a pyproject.toml you need to be able to install your build tool, and --no-index precludes that, unless you have a local copy of the build tool wheels. It's an inconvenient (and may be unexpected) side effect of PEP 518 plus build isolation, but it's essentially "by design".

It may be that we can find some way of mitigating the impact (try again with build isolation disabled if the build tools aren't available for install, for example, or allow locally installed copies of the requested build tools "leak through" into the isolated build environment) but ultimately we have to bite the bullet and accept that PEP 517 and 518 are moving us into a world where we don't assume build tools are available, and that making sure they are available becomes the responsibility of the user running the build (whether by having local wheels or by manually specifying --no-build-isolation).

[asottile]

At least with python-dateutil 2.7.1 (which does not specify PEP518 information in pyproject.toml) this is an issue (python-dateutil has since added PEP518 information to pyproject.toml, perhaps making this issue more confusing)

[pfmoore]

Yeah, if projects include invalid pyproject.toml files (the build-system.requires key is mandatory according to PEP 518) then that's a different issue (and one that I argue in #5416 that we should just reject).

[benoit-pierre]

Should the download command include build dependencies, so pip download -d dir project && pip wheel/install --no-index -f dir project work?

[asottile]

hmmm, I was under the impression that pyproject.toml was "the replacement for setup.cfg" -- a generic place to put configuration information. Seems odd to require pip build information to be there?

[benoit-pierre]

It's not pip build information, it's the project build requirements, a (working) replacement for setup_requires support when using setuptools. (As well as a way to support other build systems with PEP 517).

[asottile]

I'm sorry I was imprecise, the point stands though -- if it's not a "generic place to put configuration" then that's very very unfortunate.

[pfmoore]

@asottile It's defined in PEP 518. that's the relevant standard. The PEP does allow general configuration in there, but only for projects that use pyproject.toml (for its given purpose of specifying build requirements). I'm not quite sure where you got the impression that it was a generic place to put configuration - it's defined in PEP 518 which is clearly titled "Specifying Minimum Build System Requirements for Python Projects" (and not something like "A common configuration location for Python projects")

If that's "very, very unfortunate", then all I can say is that the problem you're seeing didn't come up in the (extensive) discussions on distutils-sig. It's certainly true that the spec could need change based on real-life feedback as it comes into general use. The PEP can be updated, and the pip implementation updated to follow (I guess the PEP gets altered to make build-system.requires optional, and state that installers like pip should simply ignore pyproject.toml if it's not there). But I doubt it's something that will happen unless someone champions such a change - the participants in the original discussions are mostly focused on actually implementing PEPs 517 and 518 now, not revising them.

[benoit-pierre]

I have seen several projects including a pyproject.toml for configuring towncrier (and not defining build requirements).

[pfmoore]

Yes, I think black might do the same. I'm not sure where the idea that this was acceptable came from, is what I'm saying.

[dstufft]

Eh, the config file was explicitly designed to allow other tools to use it as configuration. I think their use is perfectly fine. The only real question is whether using it for those tools should also opt you into the build isolation and such. I think either answer is OK here.

[pfmoore]

Agreed either answer is OK. But the current answer is that if the mandatory key is missing, we reject pyproject.toml. At the moment, we don't do so very cleanly - #5512 helps by providing a clean error message - but we do require that key.

I'm fine if the consensus is to make that key optional, but I would hope that we update the PEP in line with that decision. The worst case for me would be if the PEP says one thing, but pip does something else.

Personally, I prefer the current behaviour, where I can tell if pip will use the new PEP 517/518 processing simply by checking if pyproject.toml is present. Having to actually check the contents of that file is just a bit more of a nuisance.

[pradyunsg]

Right now, pip opts-in packages to isolation, based on the presence of pyproject.toml but does not require the key to be specified. If it's not there, it just assumes setuptools + wheel are the dependencies and continues with the isolation.

This means that if you have a pyproject.toml, you need that run of pip to also able to install setuptools+wheel; with pip 10.0, it means including wheels for setuptools and wheel, if you're using --no-index.

---

The current behaviour just seems sort of weird to me: the package get opted into isolation because they included configuration for a tool, pip's still sometimes assuming what the build-time requirements are (setuptools+wheel) even in isolation and no one's following the PEP's language.

#5512 makes pip compliant with the PEP's language.

I'm personally fine with making the build-system.requires key optional; updating the PEP and then updating pip as per that. In that case, it's presence should be used as a pointer on whether to enable PEP 518.

[pradyunsg]

We're having a discussion about this in 3 places. Can we please move the discussion about dealing with non-compliant pyproject.toml files to #5416?

[pradyunsg]

This means that if you have a pyproject.toml, you need that run of pip to also able to install setuptools+wheel; with pip 10.0, it means including wheels for setuptools and wheel, if you're using --no-index.

To be clear, the solution to OP's issue is providing setuptools and wheel wheels to pip when using --no-index. This is just another constraint of pip 10's incomplete PEP 518 support.

[pfmoore]

This is just another constraint of pip 10's incomplete PEP 518 support.

I'm not sure it's related to "incomplete" support, is it? If you pass --no-index, you'll need to make setuptools and wheel available somewhere. The only difference caused by the current limited support is that you have to provide wheels, not sdists. Or am I missing something?

[pradyunsg]

@pfmoore Indeed. My bad.

[pfmoore]

No problem - I genuinely wasn't sure whether there was a provision elsewhere in the PEPs that I might have missed. The idea that if you're not using PyPI, it's your responsibility to ensure that your build tools are available to install is something that probably merits a discussion in the PEPs (but quite likely wasn't something we thought about at the time).

[pradyunsg]

I don't think there's anything actionable here for pip; is there? Is this a case where pip could try printing a better error message?

[RonnyPfannschmidt]

yes, instead of a exception, it would be really helpful to suggest that when using no-index, one also must provide the dependency in the location where one puts all the elements

[jaraco]

I've run into this issue also in a somewhat different, though presumably common, use-case. I'd like to run tox when the index is not available, such as when on the airplane or when internet has crashed or when not connected to a private network with the configured private index. I've previously installed the package under test and run the tests, so all dependencies have been met. However, because of the way tox works, it runs pip install --no-deps -e . before each test run, which invokes the pyproject.toml build isolation behavior, which fails if the index can't be reached.

It's a bit of a high burden to require the project to be able to reach its configured index even when all dependencies are satisfied and it's already installed and the libs are cached. Is there something that a tool like tox could/should do to allow the 'install' step to proceed on a project that declares build requirements? Should tox set PIP_NO_BUILD_ISOLATION=0 (equivalent to --no-build-isolation) and supply those dependencies itself? Perhaps tox will need to be less aggressive about attempting an install on each test run.

[pradyunsg]

Noting here that #7158 happened. We realized a few interesting things about PEP 440 and I came up with an idea to make that happen, in #7132 (comment).

[blakehawkins]

This is still an issue for me with pip 22.3 from /usr/local/lib/python3.10/site-packages/pip (python 3.10) when using pip-compile with kserve 0.9.0.

  ERROR: Could not find a version that satisfies the requirement ray[serve]==1.10.0 (from kserve==0.9.0->-r requirements-dev.in (line 36)) (from versions: 1.13.0, 2.0.0rc0, 2.0.0rc1, 2.0.0, 2.0.1)
pip._internal.exceptions.DistributionNotFound: No matching distribution found for ray[serve]==1.10.0 (from kserve==0.9.0->-r requirements-dev.in (line 36))

As you can see, it pins ray[serve]==1.10.0 https://github.com/kserve/kserve/blob/v0.9.0/python/kserve/requirements.txt

Which is not yanked in pypi: https://pypi.org/project/ray/1.10.0/

Setting build-system.requires in pyproject.toml doesn't help:

[tool.black]
line-length = 120
skip-string-normalization = true

[tool.isort]
profile = "black"
line_length = 120
group_by_package = true
skip_gitignore = true

[build-system]
requires = ["setuptools", "wheel"]

[uranusjr]

If you use pip-tools, the exception is likely on them. Note that this issue is specifically about no matching distribution found for setuptools.

Actually, I think this is no longer an open issue?

[pradyunsg]

Actually, I think this is no longer an open issue?

Indeed. pip can't install your package's build tools if they're not available from any of the sources it's set up to use for that run. This means, --no-index is basically not a useful/meaningful choice, not unless you configure pip to look in different places.

So, your tractable options are:

• Pass more values via --find-links or change the index server instead.
• Not use --no-index and instead specify a custom/curated index that you operate/maintain/use that has only the packages that you want to permit.
• Use --no-build-isolation and install the build-time dependencies for all the packages globally -- this is possibly a useful "immediate patchwork" for this, since it'll likely work more often than not, but it'll break horribly if/when your packages have conflicting build requirements (that's part of the point of build isolation :P).

This isn't something that pip can provide for you, sadly. It's the user's workflows that'll need to change to adapt for the way that build isolation works (and will work in the future, within pip).