we need a way to specify the use of a custom package index for a build dependency

[con-f-use]

Given the following pyproject.toml

[build-system]
requires = [
    "setuptools",
    "custom_buildpackage"
]
build-backend = "setuptools.build_meta"

how do I specify that custom_buildpackage (and only it) is to be installed from a package-index other than the one at pypi.org?

---

I know that these are options:
1.) Set environment variables for pip to use an --extra-index-url or --index-url
2.) Set the (extra-)index in ~/.config/pip.conf
3.) Install the package beforehand from the other index

[takluyver]

Broadly, that's up to the build frontend, e.g. pip. I don't think pip has a way to specify which packages should come from a given index - you just give it a list of indices to look at, and it finds packages in any of them, I think.

See also Donald Stufft's blog post about abstract and concrete dependencies - this list is describing what it needs, not how to get that.

[con-f-use]

Not sure that distinction is wise for or even applies to build dependencies.

The way I understand the post-PEP517 workflow (and correct me if I'm wrong), is that packages are build by the build-backend in their own separate environment and build dependencies are installed there (also by the build backend). How does one control pip inside that environment and e.g. set pip environment-variables such as PIP_INDEX_URL? (Btw., even if that question is answered, PIP_INDEX_URL still effects all packages and not just a single one). It's by design a closed system. Since the build-backend installs the build-dependencies via the build-frontent, the build-backend needs to tell the frontend, where to get these build-dependencies from. There is really no way around it. IMHO, they are very concrete dependencies.

[pfmoore]

PIP_INDEX_URL still effects all packages and not just a single one

That shouldn't be an issue - packages are uniquely specified by their name and version, so it doesn't matter which index they are retrieved from, as long as they exist somewhere. If that's not the case, and it makes a difference whether you get foo-1.0 from PyPI or your local index, then your index is likely set up incorrectly.

Since the build-backend installs the build-dependencies via the build-frontent, the build-backend needs to tell the frontend, where to get these build-dependencies from.

That's correct, and as @takluyver said, this is a matter for the frontend to communicate with the backend about. As a library, the pep517 package doesn't get involved in that communication, it just installs build dependencies as required.

What frontend are you using in this case? Are you using pep517 directly as the frontend? Because if so, it's a relatively limited one (I'd be inclined to refer to it as a "proof of concept", but that may be unfair - I don't really know the longer term goals for the higher level parts of the package).

I suspect that in this instance, you are trying to directly use the higher level build APIs in the library (rather than the lower level backend wrappers). You should be aware that these are not as extensively used as the low-level code - I'm not at all sure there are any frontends using them (pip doesn't, and I don't know what other build frontends exist). Using the pep517.envbuild high level wrappers doesn't really allow for fine control over how pip gets invoked in the build environment (and it doesn't even offer the option of using an alternative frontend rather than pip) so you may simply be expecting more from those wrappers than they (currently) offer.

[takluyver]

Are you using pep517 directly as the frontend? Because if so, it's a relatively limited one (I'd be inclined to refer to it as a "proof of concept", but that may be unfair

I'd say that's fair. My main aim with pep517 was to provide a library that could be used to implement PEP 517 frontends, not to provide a frontend in itself. The envbuild level is almost like working pseudocode - it should work for the simplest cases, but probably not for more complex things.

[con-f-use]

That shouldn't be an issue - packages are uniquely specified by their name and version, so it doesn't matter which index they are retrieved from, as long as they exist somewhere.

Not across indexes. On a private index like, for instance, a company-internal devpi, one can have a local package with a common name, say pyapi at version 0.0.1, while a different one of the same version is up on pypi.org. That even is a known attack vector for private indexes that also mirror pypi, called higher version attack. And even if there is no name collision, one might have a private build dependency that is not on pypi. A package in practice is not uniquely identified by name and version (and target architecture in case of binary wheels).

That's correct, and as @takluyver said, this is a matter for the frontend to communicate with the backend about. As a library, the pep517 package doesn't get involved in that communication, it just installs build dependencies as required.

So you are saying I should take this up with the pip or setuptools folks?

[pfmoore]

On a private index like devpi, one can have a local package with a common name, say pyapi at version 0.0.1, while a different one of the same version is up on pypi.org.

That is not a valid use. Packaging tools are entitled to assume that all packages with the same name/version are identical (that's sdists or wheels - unpackaged source trees are treated specially).

[pfmoore]

So you are saying I should take this up with the pip or setuptools folks?

I was asking what frontend you were using to do your build. You haven't yet answered that question. If you're using pip, then the pip tracker would be the place to ask. But the answer would be that pip doesn't have a means to specify a per-package index (as it should never be necessary)...

[con-f-use]

Yes it is pip. The concrete problem is that in my organization we have a private plugin to setuptools that is needed for installing/building our packages. That it hosted on a private index, which also caches pypi. In the meantime someone uploaded a package with the same name as our plugin to pypi. There are several options, but the easiest would just be to specify where to get the plugin from.

That is not a valid use. Packaging tools are entitled to assume that all packages with the same name/version are identical (that's sdists or wheels - unpackaged source trees are treated specially).

From a more theoretical standpoint, with private indexes, there's really no way to completely avoid name collisions, because the name+version+platform+interpreter combinations simply are not unique. It matters not, if it's "not a valid use". The only reasonably unique identifier of a distribution is its hash. It should be valid use, because if it's not, there seem to be unavoidable security implications.

And thanks for you help. I appreciate you taking the time to argue. I mean no harm, python packaging is just a beast sometimes. I suppose there is a reason why pep517 is still provisional (as of writing).

[takluyver]

If your index also caches PyPI, then the obvious answer is to tell pip to use that for all dependencies.

[con-f-use]

If only the obvious answer would survive in our real world. Thanks again, if nothing else, I have a firmer grasp on the issue now.

[pfmoore]

That it hosted on a private index, which also caches pypi. In the meantime someone uploaded a package with the same name as our plugin to pypi

That is an issue, and I can see why existing solutions aren't viable. I think that this use case would make a good argument for a feature request to pip. I think I would be against a per-package index type of solution, but I can imagine allowing some sort of index prioritisation. It's not something I'd want to try to define out of nothing, though - there is presumably a lot of prior art in this area (conda channels, Linux distro package repositories, etc). It may even be something that should be standardised, so that there's a common way for all frontends to specify what indexes to use.

I'm fine if you think that's more work than you'd be happy to get involved in, though.

[pfmoore]

One other thought, it wouldn't be that hard to write a Python index proxy (basically like devpi, but with no storage of its own) that presents an index API while rewriting or redirecting links to order things however you'd like.

I've made suggestions like this a few times in various contexts, but it always seems to be more effort than the actual problem warrants. I do think that a "python package index proxy" might be a useful, if niche, piece of middleware. I don't know if it's something I'd ever have time to build myself, though (particularly as it solves an issue that doesn't affect me directly).

[xavfernandez]

That it hosted on a private index, which also caches pypi. In the meantime someone uploaded a package with the same name as our plugin to pypi

FWIW we have the same case with one of our packages at work but it isn't really an issue with devpi which has the concept of whitelist

[elucify]

@con-f-use did you ever find a solution to this problem? I'm up against precisely the same thing. requires= doesn't seem to have a way to tell pip to use the local index server (or define a chain of them, like .pypirc does for twine).