How can I choose a source on a per-dependency basis?

[beaugunderson]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Question

I would like to specify a source for a dependency on a per-dependency basis.

I would like to do this for a number of reasons:

• PyPi is much faster than my private repository, so it makes sense to install all of my non-private dependencies from PyPi
• I don't understand the semantics of specifying another source... Will that source be used for all dependencies, or just ones that are not found in PyPi? (If the latter, that is a security issue as someone could register our package's name on PyPi and have their code included in our app)

Therefore it seems to me that the way to have understandable semantics and the best installation speed is to support something like:

[[tool.poetry.source]]
name = "my-private-repo"
url = "http://<hostname>/simple/"

[tool.poetry.dependencies]
python = "^3.6"
my-private-package = { version = "^x.y.z", source = "my-private-repo }

[beaugunderson]

For comparison's sake, the above is possible in Pipenv with this syntax:

[[source]]
url = "http://pypi.home.kennethreitz.org/simple"
name = "home"

[packages]
requests = {version="*", index="home"}
records = "*"

(Edit: as @fridex pointed out, PyPi will also be queried for this dependency and PyPi's version will be used if it is higher than the one in the home index, leading to a security risk!)

[beaugunderson]

There was a prior request for this here as well.

[fridex]

I don't understand the semantics of specifying another source... Will that source be used for all dependencies, or just ones that are not found in PyPi? (If the latter, that is a security issue as someone could register our package's name on PyPi and have their code included in our app)

This is somehow by design in Python ecosystem. Indeed, this can be a security reason especially with the fact pip can silently fallback to the secondary index - it treats them all as mirrors.

We have done some work around this - see our provenance checks in the Thoth project. Currently we support Pipfile and Pipfile.lock style checks. In the future there is a plan for adding PEP-518 support.

See also Pipenv discussion on this in pypa/pipenv#2159

[beaugunderson]

@fridex awesome, thank you for the links--it appears the security issues remains in Pipenv even when specifying a specific index, which I had not realized.

It seems like currently the maximally safe way to use another source is to use that source instead of PyPi, not in addition to it. (In this case the source I want to use is pypicloud with passthrough to pypi; we're coming from pip-tools with --index-url at the top of the file so I think it's an easy migration path to poetry.)

[beaugunderson]

Hmm... Though I don't see a way to use another repository instead of PyPi, only in addition to it; from the docs, emphasis mine:

From now on, Poetry will also look for packages in your private repository.

[jared-mackey]

I'd like to be able to specify a default source and then override it on a per-dependency basis. The use case is that for most of my packages I am use the public PyPI index but also have a couple private packages on a private repo.

Something like this would be perfect.

[[tool.poetry.source]]
name = "my-private-repo"
url = "http://<hostname>/simple/"
default = false  # Disable using it by default

[tool.poetry.dependencies]
python = "^3.6"
my-private-package = { version = "^x.y.z", source = "my-private-repo }
requests = "*"  # Use pypi and never look at my-private-repo

[sdispater]

This is now available in the latest beta release (see #908 for more information).

[ShedPlant]

https://python-poetry.org/docs/dependency-specification/

@sdispater the documentation doesn't mention the source argument on dependencies.

Although, to be fair, --source is mentioned under poetry add --help:
https://github.com/python-poetry/poetry/blob/a3aafa840de950f81e99553e52190a54c94d6bce/docs/cli.md

I found that poetry update was being very slow especially when I defined a secondary source.

I only wanted to use the secondary source for my company's proprietary packages.
I think (not sure) that poetry is calling out to secondary source when not necessary.

Explicitly defining source per package seemed to have a big help!
Although it's unfortunately verbose.

[tool.poetry.dependencies]
python = "^3.9"
pint = { version = "^0.17", source = "PyPI" }
acme-private-decimals = { version = "^0.0.6", source="acme-private" }

[[tool.poetry.source]]
name = "acme-private"
url = "http://www.acme-private.com/pypi/"
secondary = true

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.