Using private repos in addition to the official PyPI

[madig]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

I think this question has been skimmed in other issues, but not dealt with explicitly.

Question

What is the official way to source dependencies both from the public PyPI and from a private repository? I have:

[[tool.poetry.source]]
name = "pypi_"
url = "https://pypi.org/simple/"

[[tool.poetry.source]]
name = "private"
url = "https://pypi.private.com/simple/"

and it seems to work, but is there are "more official" way to say that I use the official PyPI but if a package is missing there, it should be looked up on "private"? Like pip install --index-url official_pypi --extra-index-url private ...?

[madig]

Ugh. Just updated to from 0.12.5 to 0.12.6 and now get

> poetry update
Updating dependencies
Resolving dependencies... (10.2s)

[KeyError]
'summary'

update [--no-dev] [--dry-run] [--lock] [--] [<packages>]...

The error goes away when I remove

[[tool.poetry.source]]
name = "pypi_"
url = "https://pypi.org/simple/"

Edit: Fixed by installing poetry git master.

[madig]

Ugh! Specifying the official PyPI is messing with dependency resolution. Try poetry update with the following pyproject.toml:

[tool.poetry]
name = "testpoetry"
version = "0.1.0"
description = ""
authors = ["..."]

[tool.poetry.dependencies]
python = "^3.6"
defcon = "^0.6"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Now poetry update again with the following file:

[tool.poetry]
name = "testpoetry"
version = "0.1.0"
description = ""
authors = ["..."]

[tool.poetry.dependencies]
python = "^3.6"
defcon = "^0.6"

[tool.poetry.dev-dependencies]

[[tool.poetry.source]]
name = "pypi_"
url = "https://pypi.org/simple/"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

The first file will pull in the packages fs, pytz and appdirs. The second will remove them again! This breaks defcon.

[kbakk]

@madig Did you work around this?

[madig]

Yes, with the last snippet on my last comment. This messes with dependency resolving unfortunately, i.e. optional dependencies specified by [] in your dependencies might not get picked up.

[hongquan]

Please also support extra-index-url from command line (not pyproject.toml file).

One use case is that, we need to use peotry install in an Alpine-based Docker container, where musl lib C is used in placed of glibc.
The wheel file built for glibc is not always compatible with musl, so we setup a separate repo for musl-based packages. This repo is only used inside Docker container, to run unit test. For production deployment, we use normal glibc environment, that's why we cannot add this "index URL" to pyproject.toml file (or pip will mistakenly download package from it).

If poetry supports extra-index-url parameter, we just need to modify the installer command in Docker container and things will work well.

[Yanpas]

Official docs as well require http auth poetry config http-basic, but my private repo is read-only for me.

I need one package from it, how can I achieve it? Something like this wiould be cool (same as pip -i):

packagename = 1.0 {repo=myprivaterepo.com}

[thejohnfreeman]

@sdispater sorry to poke you, but can you provide some insight into this?

The problem as I see it is that we need a precedence order for source repositories, with the capability to override the order per-dependency.

I uploaded my own package to Test PyPI that transitively depends on lazy-object-proxy (through conan, but it is also depended on by the very popular pylint package). All of the latest versions of lazy-object-proxy uploaded to Test PyPI are pre-release versions (1.4.2.dev{1,2,3}). You can reproduce this problem yourself by adding the Test PyPI source to pyproject.toml and calling poetry add lazy-object-proxy. Poetry uses pip to deduce that the latest version available is 1.4.2, but refuses to install one of the pre-release versions without the --allow-prereleases flag. I just want it to use non-Test PyPI for all dependencies except my package.

Related issues:

• Failure to detect dependencies of scikit-learn from private source #1160
• Dependency resolution problem with non-pypi repository #706

[thejohnfreeman]

If I update to Poetry 1.0.0a3, then I get this error trying to install lazy-object-proxy with the Test PyPI source in my pyproject.toml:

# in an empty directory
$ python --version
Python 3.7.1
$ poetry --version
Poetry version 1.0.0a3
$ poetry init
...
$ cat >>pyproject.toml <<EOS
[[tool.poetry.source]]
name = "test-pypi"
url = "https://test.pypi.org/simple"
EOS
$ poetry add lazy-object-proxy
Creating virtualenv $venv in $PWD/.venv
Using version ^1.4 for lazy-object-proxy

Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  - Installing lazy-object-proxy (1.4.2)

[AttributeError]
'Pool' object has no attribute 'default'

[areeh]

@thejohnfreeman See #1163 which seems to be caused by 694bef2

[DomWeldon]

Did you have any luck with this? Causing a big issue with my packaging at the moment. Poetry passes -i my-pypi.example.org to pip as well as --extra-index-url pypi.example.org which means none of
my package's dependencies can be resolved. This was in 0.12.17 and 1.0.0b2.

[mathiesonsterling]

I'm seeing the same issue I believe with 0.12.17 - basically if I specify a source, it no longer can seem to find any packages from the public repository, and fails on updates.

[sdispater]

There has been a lot of improvement in private indices management (see #908).

This is available in the latest beta release.

[madig]

It werks! Thanks for working on this to everyone involved 😃

[nicktimko]

If anyone comes across this later: here are the docs for controlling package repos. (permalink in the repo if they move)

[madig]

Hrm, I have to take my last post back. If I use a private repository and list a package from there as a dependency, transient dependencies such as toml for isort[pyproject] get purged on poetry update. I'll have to see if I can replicate this with test.pypi.org.

[mawillcockson]

Official docs as well require http auth poetry config http-basic, but my private repo is read-only for me.

I need one package from it, how can I achieve it? Something like this wiould be cool (same as pip -i):

packagename = 1.0 {repo=myprivaterepo.com}

This specific issue can be addressed if a direct download link for the package is available. For example, at the time of this writing poetry is at 1.1.4. This version can be downloaded directly from PyPI (even if this version is yanked):

poetry add "https://files.pythonhosted.org/packages/f9/dd/b6ba82de8c075d1a795670ae16952a4c1cb21398b4330d8cfb51241e32d7/poetry-1.1.4-py2.py3-none-any.whl"

Currently, this results in a pyproject.toml that has this:

[tool.poetry.dependencies]
python = "^3.9"
poetry = {url = "https://files.pythonhosted.org/packages/f9/dd/b6ba82de8c075d1a795670ae16952a4c1cb21398b4330d8cfb51241e32d7/poetry-1.1.4-py2.py3-none-any.whl"}

This is mentioned in the docs. In case that changes, here is where that documentation exists in this repository.

Unfortunately, if instead the correct version needs to be selected automatically, poetry cannot currently do this as of 1.1.4. pip has the --find-links option, and poetry currently does not have an equivalent (see #1391).

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.