Can't install packages from private static repository #5548

lecardozo · 2022-05-05T20:36:40Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS Monterey (12.2.1)
Poetry version: 1.1.13
Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

Hey, everyone! :D

I'm trying to use Poetry as the dependency manager for one of our internal applications. Here we have an internal private PyPI repository that is essentially a static index (served on Github Pages) following the https://<DOMAIN>/python-package-server/<PACKAGE_NAME>/ URL pattern. For each package, we have a list of links to specific repositories where the artifacts live in. Here is an example of what a page looks like for a package in this repository. 👇

<!DOCTYPE html>
<html>
  <head>
      <title>Links for package</title>
  </head>
  <body>
    <h1>Links for package</h1>
    <!-- package-server-links-start -->
    <a href="git+https://github.com/<redacted>/package@2.0.0#egg=package-2.0.0">package-2.0.0</a><br/>
    <a href="git+https://github.com/<redacted>/package@1.1.0#egg=package-1.1.0">package-1.1.0</a><br/>
    <a href="git+https://github.com/<redacted>/package@1.0.0#egg=package-1.0.0">package-1.0.0</a><br/>
  </body>
</html>

After adding the private source to pyproject.toml, the resolver was unable to find matching versions for that package

Loading configuration file /Users/me/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/me/projects/my-project/poetry.toml
Adding repository private (https://<DOMAIN>/python-package-server)
Using virtualenv: /Users/me/projects/my-project/.venv
[urllib3.connectionpool] Starting new HTTPS connection (1): <DOMAIN>:443
[urllib3.connectionpool] https://<DOMAIN>:443 "GET /python-package-server/package/ HTTP/1.1" 304 0
private: 0 packages found for package * # <---- at least it tried :) 
private: 0 packages found for package * # <---- once again? maybe 
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/package/json/ HTTP/1.1" 404 1988
PyPI: No packages found for package *

I guess this is happening because this format doesn't seem to be fully PEP 503 complaint, although we never had issues with PIP or PDM before.

After checking for previous discussions, I ended up finding this PR which seems to be related but I'm not completely sure. I've tested that specific version (from 5517) to see if it would work on my use case, but, unfortunately, it didn't.

Any clues on what could be the issue here? 🤔

Thanks!

The text was updated successfully, but these errors were encountered:

abn · 2022-05-05T20:54:58Z

Hi @lecardozo #5517 is aimed at single page PEP 503 repositories only. This won't work as it is referring to VCS sources. The pip and PDM use cases for these sources are different to that of Poetry. You might have better luck either pointing them to release artifacts - ie. valid sdist fiels or wheels instead of a git repository.

lecardozo · 2022-05-05T21:05:53Z

Thanks for the quick reply @abn ! 😃

The pip and PDM use cases for these sources are different to that of Poetry. You might have better luck either pointing them to release artifacts - ie. valid sdist fiels or wheels instead of a git repository.

I see. So the limitation is in the fact that Poetry can't handle links to a specific tag in a repository (to be cloned and installed), only direct links to artifacts (sdists or wheels). Is that correct? If so, is it something that poetry might support in the future?

abn · 2022-05-05T21:21:44Z

Not sure if it is a limitation, but rather a lack of standards around this. Poetry's source support is based on the PEP 503 standards, which iirc, does not define the distribution of direct link or vcs sources.

Also, since Poetry will need to download (in your case clone) all the package versions separately and inspect the metadata to generate the universal lock file, such a mechanism of consuming packages might be rather inefficient. Additionally, this can also lead to a lot of ambuity. If a package distributed via such a mecahnism is locked, do we identify it as a VCS sourced dependency or a simple API repository sourced dependency? And what happens when a user consumes this lockfile? Does Poetry go over these again and inspect on their local machine? Or do we treat it as VCS dependency and manage it so?

As things are today, I doubt this will be something we will support. But maybe if there are enough folks out there who want something similar, you could potentially create a plugin.

lecardozo · 2022-05-06T12:00:32Z

Thanks for the clarification!

If a package distributed via such a mecahnism is locked, do we identify it as a VCS sourced dependency or a simple API repository sourced dependency? And what happens when a user consumes this lockfile? Does Poetry go over these again and inspect on their local machine? Or do we treat it as VCS dependency and manage it so?

I tend to think that this should be identified as a VCS sourced dependency instead of Simple API (even though the API is the intermediate here). The API ends up being just a "shortcut" index, so the user doesn't need to explicitly pass the git dependency.

With the link-to-VCS API

[tool.poetry.dependencies]
python = "^3.8"
package = "2.0.0" # still comes from VCS, but has an intermediate API that links to the actual repo

Directly to VCS

[tool.poetry.dependencies]
python = "^3.8"
package = { git = "https://github.com/company/package-repo.git", tag = "2.0.0" } #  points directly to the repo

As things are today, I doubt this will be something we will support. But maybe if there are enough folks out there who want something similar, you could potentially create a plugin.

Yeah, that makes sense! In our case, linking explicitly to the VCS seems to be enough. Although this is a little less convenient from the user's point-of-view, this seems a better choice. 😃

abn · 2022-05-06T12:15:07Z

@lecardozo there is general interest in supporting something similar to "find-links" in pip. Might be a good discussion topic. Although what you have is more similar to a constraints file.

But all of this seems to be a way to avoid having a proper package index. Having one will also save developers wheel build times too among other things.

lecardozo · 2022-05-06T13:12:48Z

@lecardozo there is general interest in supporting something similar to "find-links" in pip.

It seems like my use case falls between the find-links and the PEP 503 index approach. I guess a find-links functionality would probably work for this case but doesn't seem ideal.

But all of this seems to be a way to avoid having a proper package index. Having one will also save developers wheel build times too among other things.

Agreed. Although I feel that, in some cases, setting up a proper private package index might be prohibitive/not worth the effort.

abn · 2022-05-06T13:39:49Z

GitHub packages supporting Python packages might be helpful in your case. Alternatively, I could also recommend that you use a github release action, similar to

poetry/.github/workflows/release.yml

Lines 36 to 42 in 0d418fc

  - name: Create Release 

 uses: ncipollo/release-action@v1 

 with: 

 artifacts: "dist/*" 

 token: ${{ secrets.GITHUB_TOKEN }} 

 draft: false 

 prerelease: steps.check-version.outputs.prerelease == 'true'

And then in your html file link to the release assets, like in https://github.com/python-poetry/poetry/releases/download/1.2.0b1/poetry-1.2.0b1-py3-none-any.whl.

github-actions · 2024-03-01T23:06:10Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

lecardozo added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels May 5, 2022

lecardozo closed this as completed May 6, 2022

mkniewallner removed the status/triage This issue needs to be triaged label Jun 11, 2022

acrogenesis mentioned this issue Jun 3, 2023

Doesn't work with Poetry astariul/github-hosted-pypi#68

Closed

astariul mentioned this issue Sep 14, 2023

Poetry fails to find and install packages astariul/github-hosted-pypi#69

Open

github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't install packages from private static repository #5548

Can't install packages from private static repository #5548

lecardozo commented May 5, 2022

abn commented May 5, 2022

lecardozo commented May 5, 2022

abn commented May 5, 2022

lecardozo commented May 6, 2022

abn commented May 6, 2022

lecardozo commented May 6, 2022 •

edited

Loading

abn commented May 6, 2022 •

edited

Loading

github-actions bot commented Mar 1, 2024

Can't install packages from private static repository #5548

Can't install packages from private static repository #5548

Comments

lecardozo commented May 5, 2022

Issue

abn commented May 5, 2022

lecardozo commented May 5, 2022

abn commented May 5, 2022

lecardozo commented May 6, 2022

With the link-to-VCS API

Directly to VCS

abn commented May 6, 2022

lecardozo commented May 6, 2022 • edited Loading

abn commented May 6, 2022 • edited Loading

github-actions bot commented Mar 1, 2024

lecardozo commented May 6, 2022 •

edited

Loading

abn commented May 6, 2022 •

edited

Loading