Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pip & poetry backend does not honor poetry's "secondary" repo setting #4400

Closed
maksbotan opened this issue Nov 10, 2021 · 9 comments
Closed

pip & poetry backend does not honor poetry's "secondary" repo setting #4400

maksbotan opened this issue Nov 10, 2021 · 9 comments
Labels
L: python:poetry Python packages via poetry T: bug 🐞 Something isn't working

Comments

@maksbotan
Copy link

Package ecosystem
pip + poetry
Package manager version
Poetry version 1.1.11
Language version
Python 3.8
Manifest location and content prior to update
https://github.com/maksbotan/dependabot-poetry-demo/blob/master/pyproject.toml
https://github.com/maksbotan/dependabot-poetry-demo/blob/master/poetry.lock
dependabot.yml content
https://github.com/maksbotan/dependabot-poetry-demo/blob/master/.github/dependabot.yml
Updated dependency
Bump docxtpl from 0.14.1 to 0.14.2
What you expected to see, versus what you actually saw

I have a private registry with secondary = true in my pyproject.toml, i.e. I intend to use it only for my private packages, not overriding the global pypi. poetry correctly locks only private package to this repo (see https://github.com/maksbotan/dependabot-poetry-demo/blob/7c539a42a3e2ff108140827142be6bd19e8c5767/poetry.lock#L299-L302).

But dependabot's update locks all packages to use this private repo, including global pypi's ones. See this diff: https://github.com/maksbotan/dependabot-poetry-demo/pull/7/files#diff-f53a023eedfa3fbf2925ec7dc76eecdc954ea94b7e47065393dbad519613dc89

Note that all packages got a [[package.source]] section, with a bogus reference at that. And the one with correct reference got rewritten:

https://github.com/maksbotan/dependabot-poetry-demo/pull/7/files#diff-f53a023eedfa3fbf2925ec7dc76eecdc954ea94b7e47065393dbad519613dc89L302

Native package manager behavior

poetry will not try to get a dependency from a private repo with secondary = true unless explicitly told so.

By the way, this exact bug was present in poetry 1.1.4 and was only fixed in 1.1.10. See the relevant issue: python-poetry/poetry#3306. Maybe it would be enough to update poetry version used by dependabot?

Images of the diff or a link to the PR, issue or logs

https://github.com/maksbotan/dependabot-poetry-demo/pull/7/files

updater | INFO <job_231638073> Checking if boto3 1.20.0 needs updating
  proxy | 2021/11/10 20:25:06 [164] GET https://pypi.org:443/simple/boto3/
  proxy | 2021/11/10 20:25:06 [164] 200 https://pypi.org:443/simple/boto3/
  proxy | 2021/11/10 20:25:08 [166] GET https://gitlab.math.bio:443/api/v4/groups/biocad/-/packages/pypi/simple/boto3/
  proxy | 2021/11/10 20:25:08 [166] * authenticating python index request (host: gitlab.math.bio)
  proxy | 2021/11/10 20:25:09 [166] 302 https://gitlab.math.bio:443/api/v4/groups/biocad/-/packages/pypi/simple/boto3/
  proxy | 2021/11/10 20:25:09 [168] GET https://pypi.org:443/simple/boto3/
  proxy | 2021/11/10 20:25:09 [168] 200 https://pypi.org:443/simple/boto3/
updater | INFO <job_231638073> Latest version is 1.20.3

Here it can be seen that dependabot tries to check this package in private repo (even though lock files states that it should not happen) and finds it there (since the repo redirects unknown packages to global pypi).

🕹 Bonus points: Smallest manifest that reproduces the issue

https://github.com/maksbotan/dependabot-poetry-demo/
@maksbotan maksbotan added the T: bug 🐞 Something isn't working label Nov 10, 2021
@brrygrdn brrygrdn added the L: python:poetry Python packages via poetry label Nov 26, 2021
@maksbotan
Copy link
Author

Hi! Is there a timeline for reaction to this? I would like to see the issue resolved.

I've tried disabling proxying to global PyPI on our private registry and it almost fixed the problem, however dependabot still messes up reference fields in packages from secondary repos. See here:

https://github.com/maksbotan/dependabot-poetry-demo/blob/52c3f301f0c59f339e088c0e494316e6aad0fc95/poetry.lock#L302

image

This is plain wrong :(

@andreasfedermannkm
Copy link

Hi there. A timeline would be great! I have the same issue and unfortunately I can't use dependabot.

@maksbotan
Copy link
Author

@brrygrdn can we have your attention please? This issue is blocking us from using dependabot for projects within our organization.

@jakecoffman
Copy link
Member

I made some changes in how Poetry operates in #5222 which fixes the reference change issue. It may also address honoring the secondary setting. Let me know if it's working for you now, thanks for your patience!

@pliniodng
Copy link

Hey @jakecoffman, it seems that did not fix the issue at hand:

After adding the following to pyproject.toml:

  [[tool.poetry.source]]
  name = "aws"
  url = "https://dgx-shared-codeartifact-REDACTED.d.codeartifact.REDACTED.amazonaws.com/pypi/dgx-shared-pypi-repo/simple/"
  secondary = true

I've went ahead and retried a Dependabot update that was pending PR creation on the same repo. Here are the dependabot logs for that.

updater | INFO <job_391852703> Starting job processing
updater | INFO <job_391852703> Starting update job for REDACTED/REDACTED
updater | INFO <job_391852703> Checking if pyjwt 1.7.1 needs updating
  proxy | 2022/06/14 14:30:54 [054] GET https://pypi.org:443/simple/pyjwt/
  proxy | 2022/06/14 14:30:54 [054] 200 https://pypi.org:443/simple/pyjwt/
  proxy | 2022/06/14 14:30:54 [056] GET https://dgx-shared-codeartifact-REDACTED.d.codeartifact.REDACTED.amazonaws.com:443/pypi/dgx-shared-pypi-repo/simple/pyjwt/
  proxy | 2022/06/14 14:30:54 [056] 401 https://dgx-shared-codeartifact-REDACTED.d.codeartifact.REDACTED.amazonaws.com:443/pypi/dgx-shared-pypi-repo/simple/pyjwt/
  proxy | 2022/06/14 14:30:54 [058] GET https://dgx-shared-codeartifact-REDACTED.d.codeartifact.REDACTED.amazonaws.com:443/pypi/dgx-shared-pypi-repo/simple/
  proxy | 2022/06/14 14:30:54 [058] 404 https://dgx-shared-codeartifact-REDACTED.d.codeartifact.REDACTED.amazonaws.com:443/pypi/dgx-shared-pypi-repo/simple/
updater | INFO <job_391852703> Latest version is 2.4.0
  proxy | 2022/06/14 14:31:00 [060] GET https://pypi.org:443/pypi/smart-open/json
  proxy | 2022/06/14 14:31:00 [060] 200 https://pypi.org:443/pypi/smart-open/json
  proxy | 2022/06/14 14:31:00 [062] GET https://dgx-shared-codeartifact-REDACTED.d.codeartifact.REDACTED.amazonaws.com:443/pypi/dgx-shared-pypi-repo/simple/smart-open/
  proxy | 2022/06/14 14:31:00 [062] 401 https://dgx-shared-codeartifact-REDACTED.d.codeartifact.REDACTED.amazonaws.com:443/pypi/dgx-shared-pypi-repo/simple/smart-open/
updater | I, [2022-06-14T14:31:00.489768 #7]  INFO -- sentry: ** [Raven] Sending event 7092f57cd4844c899fd4d7a171150b1b to Sentry
  proxy | 2022/06/14 14:31:00 [064] POST https://sentry.io:443/api/1451818/store/
  proxy | 2022/06/14 14:31:00 [064] 200 https://sentry.io:443/api/1451818/store/
updater | ERROR <job_391852703> Error processing pyjwt (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | ERROR <job_391852703> Creating virtualenv REDACTED-d02FRkIa-py3.10 in /home/dependabot/.cache/pypoetry/virtualenvs
updater | <job_391852703> Updating dependencies
updater | <job_391852703> Resolving dependencies...
updater | <job_391852703> 
updater | <job_391852703>   RepositoryError
updater | <job_391852703> 
updater | <job_391852703>   401 Client Error: Unauthorized for url: https://dgx-shared-codeartifact-REDACTED.d.codeartifact.REDACTED.amazonaws.com/pypi/dgx-shared-pypi-repo/simple/smart-open/
updater | <job_391852703> 
updater | <job_391852703>   at /usr/local/.pyenv/versions/3.10.4/lib/python3.10/site-packages/poetry/repositories/legacy_repository.py:393 in _get
updater | <job_391852703>       389│             if response.status_code == 404:
updater | <job_391852703>       390│                 return
updater | <job_391852703>       391│             response.raise_for_status()
updater | <job_391852703>       392│         except requests.HTTPError as e:
updater | <job_391852703>     → 393│             raise RepositoryError(e)
updater | <job_391852703>       394│ 
updater | <job_391852703>       395│         if response.status_code in (401, 403):
updater | <job_391852703>       396│             self._log(
updater | <job_391852703>       397│                 "Authorization error accessing {url}".format(url=response.url),
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.191.1/lib/dependabot/python/update_checker/poetry_version_resolver.rb:325:in `run_poetry_command'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.191.1/lib/dependabot/python/update_checker/poetry_version_resolver.rb:91:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.191.1/lib/dependabot/shared_helpers.rb:168:in `with_git_configured'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.191.1/lib/dependabot/python/update_checker/poetry_version_resolver.rb:77:in `block in fetch_latest_resolvable_version_string'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.191.1/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.191.1/lib/dependabot/shared_helpers.rb:49:in `chdir'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.191.1/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.191.1/lib/dependabot/python/update_checker/poetry_version_resolver.rb:76:in `fetch_latest_resolvable_version_string'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.191.1/lib/dependabot/python/update_checker/poetry_version_resolver.rb:58:in `resolvable?'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.191.1/lib/dependabot/python/update_checker.rb:135:in `fetch_lowest_resolvable_security_fix_version'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.191.1/lib/dependabot/python/update_checker.rb:87:in `lowest_resolvable_security_fix_version'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.191.1/lib/dependabot/update_checkers/base.rb:72:in `preferred_resolvable_version'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.191.1/lib/dependabot/update_checkers/base.rb:257:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.191.1/lib/dependabot/update_checkers/base.rb:249:in `numeric_version_can_update?'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.191.1/lib/dependabot/update_checkers/base.rb:199:in `version_can_update?'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.191.1/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:454:in `requirements_to_unlock'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:232:in `check_and_create_pull_request'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `block in run'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `each'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `run'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_391852703> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:35:in `run'
updater | ERROR <job_391852703> bin/update_files.rb:22:in `<main>'
updater | INFO <job_391852703> Finished job processing
updater | INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | time="2022-06-14T14:31:01Z" level=info msg="task complete" container_id=job-391852703-updater exit_code=0 job_id=391852703 step=updater

Once I remove the CodeArtifact source, Dependabot is able to create the PR without issues.

@maksbotan
Copy link
Author

@pliniodng did you configure credentials for that repository in dependabot.yml and in github repo secrets?

@maksbotan
Copy link
Author

@jakecoffman the fix seems to work for us! There is one minor nuisance I noticed, I will look into that and then close the issue.

@pliniodng
Copy link

@pliniodng did you configure credentials for that repository in dependabot.yml and in github repo secrets?

Since credentials for CodeArtifact expires every 12 hours, setting it is not a viable option for the company I'm working.

The ideia was that if DependaBot honored the secondary=True flag, it would first try to fetch the packages from PyPI, and fallback to the private repository, as per Poetry default behavior. (https://python-poetry.org/docs/repositories/#install-dependencies-from-a-private-repository).

@jeffwidman
Copy link
Member

@pliniodng as per the poetry docs:

All package sources (including secondary sources) will be searched during the package lookup process. These network requests will occur for all sources, regardless of if the package is found at one or more sources.

In order to limit the search for a specific package to a particular package repository, you can specify the source explicitly. This is strongly suggested for all private packages to avoid dependency confusion attacks.

I suspect if you fire up tcpdump / wireshark and watch native poetry, you'll see it making the same calls as Dependabot even though CodeArtifact is configured with secondary=True.

I'm going to close as the original issue here is resolved, and the remaining edge case unfortunately sounds like an upstream issue with how poetry works. Dependabot shells out to native poetry as part of the update process, so there is a decent chance that if they change their behavior, that we might get it for free when we update to that version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
L: python:poetry Python packages via poetry T: bug 🐞 Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@maksbotan @jeffwidman @jakecoffman @brrygrdn @andreasfedermannkm @pliniodng