Using: keyrings.google-artifactregistry-auth pipenv update -d fails hard. #4706
Comments
|
I experienced the same issue and noticed that This will update the lockfile, but also modify your Pipfile to point towards local archives. Undo the changes to the Pipfile and it will work as expected. I'm trying to figure out how to properly fix this, I'll reach out if I find anything. |
|
I'm experiencing a similar issue and am also trying to find a solution. This workaround of downloading packages first and then installing them from local works, but is not so ideal |
|
Its my dream to have pipenv+artifact-registry work nicely together (leveraging keychain for secure credential management). Therin lies a world of rainbows and unicorns. +1 on fixing this problem. I unfortunately lack the time to dig into this but if somebody could gat a fix in they would have my eternal gratitude. |
|
This also fails for Azure DevOps Atifact Feeds with the I'm going to call it a night for today, but I wanted to share what I discovered so far. |
|
I did not get as much sleep as I hoped, but at least I came up with a possible long term fix while lying awake: add a Any thoughts? |
|
@Darsstar I see three options:
you have to replace |
ghost
mentioned this issue
|
Any idea when this fix will be released? We're blocked from moving over to GCP artifact registry because of it. |
|
@savitha-atg and cc: @allcloud-jonathan I was hoping we could get some documentation added as part of the PR: https://github.com/pypa/pipenv/pull/5036/files Have you verified that PR is working for you? Planning to do a pipenv release tonight, and so its possible we could merge it later today but like I said about the documentation. Also since there are no tests for this change, that should be considered if its possible to add a test that would prevent future regression, even if it was a mock.patch test. If we have to punt on the documentation and/or test for it, which seems less than ideal, lets at least ticket it before merging the other PR. |
|
No, I haven't. I'm very new to |
|
@savitha-atg Its relatively simpler than that with pip. First you would If you do the preferred user installs of pipenv (which requires having that location on your PATH) then you would just add the |
|
So apparently I have not linked the PR I created for the Pip issue I did link. Let me rectify that: pypa/pip#11029. In the unlikely(?) situation that patching the vendored Pip with those changes and adding a |
|
I ran this slightly different command to install the fix: I've confirmed that I've the right version: Then, I modified my Pipfile like this: Is that right? I'm still running into failure to lock: |
|
Thanks @Darsstar -- My quick reaction is that is too big of a change for a pip vendoring patch when you already have a good level of buy in that this will be accepted into pip, which we would get through re-vendoring at that time. This raises the question of does the current outstanding PR satisfy patching this issue in the interim? |
|
@savitha-atg I think that |
Assuming that it will indeed gets accepted into pip, would you want pipenv to have a |
|
Ok, I've made the above change. I'm still running into locking issue: I've confirmed that |
|
@savitha-atg You are running into the new features of package index restrictions. Long story short, because pypi is your second source its non default. You'll have to pin a package one way or the other to an index with your setup, depending on the number of packages you need to pin is how I would pick my default. So for example: For more details there are some recently closed issues and the documentation was recently updated to be more helpful: https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes |
|
@Darsstar Feel free to open a PR about it, I am admittedly a bit rusty on how the pip arguments get supported in pipenv, and I seem recall discussions on old issues about it being challenging to support all the arguments. Perhaps there is a way though with environment variables to specify pip arguments such that we don't need to add an explicit flag within pipenv for it? Sorry, I am a bit tied up at the moment to really research this further today but definitely see what you can figure out and it will get us thinking more about it. |
|
@matteius That worked! Would you be able to release this fix today? I'd rather install an official |
|
@matteius I'm having a similar issue to what savitha-atg is seeing. We have packages that are only available in a private Google Artifact Registry server. However, we're currently specifying our dependencies in a with all the dependencies for |
|
@kylebluenote There is no current way to specify the index in the setup.cfg or pyproject.toml or setup.py -- as far as I know, it is only a pip and pipenv feature today to specify the indexes. For your case the recommendation would be to mirror the pypi packages to your local pypi server and have that be the default index. All packages that are unspecified try to resolve only to the default index. We are working on a feature that would allow installation from multiple sources again as it was before since they have the hashes properly sources in the lock file already, but for locking resolution to work it needs to not guess what index to pull the package from. |
|
After some trial and error, I found that you can specify dependencies packaged through
|
|
Sharing an extra observation - |
|
pipenv 2023.4.20 vendors pip 23.1. pip 23.1 contains pypa/pip#11698. While I hope this is helpfull to people. |
|
@Darsstar's comment nudged me in the right direction (thanks !). As I mentionned in this other issue I face the same situation with Google Artifact and the username must be set to |
|
Just to make it easier for future readers. This kind of a Pipfile works for me on a private Google artifact repository. You also need to have your authentication set up correctly. So basically two features are needed:
I have pipenv version |
|
I'm trying to get a better understanding of how My setup:
I have a project with a Pipfile that looks like this: A few things I've noticed:
The two configurations where the keyring packages aren't installed in the project virtualenv result in the following error when I attempt to run Additionally, I've tested this on the following
When I run on I'm not totally sure if this is a pipenv bug, but I'm a little bit confused about where to start understanding this, since this is the keyring library version in my project virtualenv. Lastly, I'm not entirely sure what Darsstar meant above, but does seem to still be necessary for me, in the configurations that work at all. I think my overall question is: is there an "expected" usage pattern for keyring integration with pipenv for private package registries? e.g. should the keyring packages be installed adjacent to pipenv, or inside of the project venv? Should I think with a better understanding of how I "should" use these, I can better hone in on what's actually going wrong in my environment. Thanks! |
What does Since you used Pipx to install pipenv Pipx's Pip version should now be 23.1 or higher, but tools like pyprojectx probably won't work yet untill a Python version comes out that ships a .whl in the ensurepip module of version 23.1 or higher and you start using that version exclusively.
Depends on the Configure Pip so that a venv created with
Configure Pip so that a venv created with |
|
Thanks @justin-yan your comment was really insightful:
That was a missing piece for me to get up an running to debug these issues fruther. It seems the next issue is that in Which has the effect of causing the pip requests to be the first one that is imported, and it does not have a |
|
Sorry for the slow response - I'm not great at checking my notifications on github! From your comment @matteius, it sounds like the intended behavior and usage pattern is that the keyring packages be installed inside of the project venv, and pipenv leverages those within-venv packages to perform installs of other packages? I think that explains my original confusion as to how pipenv was using my project's dependencies to generate the lockfile, and I also understand your concern about "something in the virtualenv that could be older than whats in the vendor packages could take precedence and cause a different side effect in pipenv" -> that does seem fairly problematic. I wonder if it makes sense to have something like an |
That is the only way Pip, and thus Pipenv, was able to use keyring. This is no longer the only way.
With the above you configure Pip. So improving the documentation seems like the better way to go about this. PS. I contributed the |
|
Following up after I had a moment to test things out! I just spent some time testing out various configurations, and finally figured out what I had to do to get this working - Darsstar's comment was extremely helpful in giving me the thread to start pulling on to figure some of this out!
So to sum up:
and I can run |
|
@justin-yan thanks for those nice notes. i was able to finally configure the Pipfile to use the keyring providers. The trick was the
|
Be sure to check the existing issues (both open and closed!), and make sure you are running the latest version of Pipenv.
Check the diagnose documentation for common issues before posting! We may close your issue if it is very similar to one of them. Please be considerate, or be on your way.
Make sure to mention your debugging experience if the documented solution failed.
Issue description
Google Artifact Repositories use keyrings with their own addon to make git and twine work perfectly, but when using pipenv it fails hard to a point where we cant use pipenv.
Note I also tried with 20.11.15 with same result
Expected result
pipenv update -d should update dependencies and make an update Pipfile.lock
Actual result
(applied_mapper) klaus@new-server ~/src/applied_mapper master> pipenv update -d -v
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Building requirements...
Resolving dependencies...
⠧ Locking...
ROUND 1
✘ Locking Failed!
Current constraints:
aadomain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 10))
applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
applied-domain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 9))
geojson (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 13))
keyring (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 7))
keyrings.google-artifactregistry-auth (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 15))
pbr (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 5))
pylint (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 4))
pytest (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 8))
pytest-flake8 (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 6))
python-dateutil (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 12))
strenum (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 11))
Finding the best candidates:
found candidate aadomain==1.4.42 (constraint was )
Traceback (most recent call last):
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 808, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 260, in _resolve_one_round
best_matches = {self.get_best_match(ireq) for ireq in constraints}
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 260, in
best_matches = {self.get_best_match(ireq) for ireq in constraints}
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 319, in get_best_match
best_match = self.repository.find_best_match(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 202, in find_best_match
raise NoCandidateFound(ireq, all_candidates, self.finder)
pipenv.patched.piptools.exceptions.NoCandidateFound: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
No versions found
Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in
main()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
results, resolver = resolve(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
return resolve_deps(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1093, in actually_resolve_deps
resolver.resolve()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 818, in resolve
raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
No versions found
Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable?
ROUND 1
Current constraints:
aadomain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 10))
applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
applied-domain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 9))
geojson (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 13))
keyring (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 7))
keyrings.google-artifactregistry-auth (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 15))
pbr (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 5))
pylint (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 4))
pytest (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 8))
pytest-flake8 (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 6))
python-dateutil (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 12))
strenum (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 11))
Finding the best candidates:
found candidate aadomain==1.4.42 (constraint was )
Traceback (most recent call last):
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 808, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 260, in _resolve_one_round
best_matches = {self.get_best_match(ireq) for ireq in constraints}
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 260, in
best_matches = {self.get_best_match(ireq) for ireq in constraints}
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 319, in get_best_match
best_match = self.repository.find_best_match(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 202, in find_best_match
raise NoCandidateFound(ireq, all_candidates, self.finder)
pipenv.patched.piptools.exceptions.NoCandidateFound: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
No versions found
Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in
main()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
results, resolver = resolve(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
return resolve_deps(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1093, in actually_resolve_deps
resolver.resolve()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 818, in resolve
raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
No versions found
Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable?
Steps to replicate
Provide the steps to replicate (which usually at least includes the commands and the Pipfile).
Its hard to replicate as you need alpha access to artifact repo pypi, but using a Pipfile like this:
[[source]]
name = "applied-backend"
url = "https://europe-north1-pypi.pkg.dev/new-server/pypi/simple"
verify_ssl = true
[[source]]
name = "applied"
url = "https://legacy-pypi.appliedautonomy.no:8443/simple"
verify_ssl = true
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
keyring = ""
"keyrings.google-artifactregistry-auth" = ""
pytest = ""
pytest-flake8 = ""
pbr = ""
pylint = ""
StrEnum = ""
python-dateutil = ""
geojson = ""
aadomain = ""
applied-core = ""
applied-domain = ""
[packages]
[requires]
python_version = "3.9"
then running pipenv update -d you will get the hard fail listed above. I do believe its connected to the keyring as it works as a charm in pip without any drama.
Please run
$ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.$ pipenv --support
Pipenv version:
'2020.8.13'Pipenv location:
'/home/klaus/.local/lib/python3.8/site-packages/pipenv'Python location:
'/usr/bin/python3'Python installations found:
3.9.0:/home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3.93.9.0:/home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python33.9.0:/home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3.93.9.0:/home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python33.9.0:/usr/bin/python3.93.9.0:/bin/python3.93.8.5:/usr/bin/python3.83.8.5:/usr/bin/python33.8.5:/bin/python3.83.8.5:/bin/python32.7.18:/usr/bin/python22.7.18:/usr/bin/python2.72.7.18:/bin/python22.7.18:/bin/python2.7PEP 508 Information:
System environment variables:
GJS_DEBUG_TOPICSSSH_AUTH_SOCKSESSION_MANAGERGNOME_TERMINAL_SCREENSSH_AGENT_PIDXDG_CURRENT_DESKTOPLANGLC_IDENTIFICATIONDEFAULTS_PATHXDG_SESSION_CLASSCOLORTERMLIBVIRT_DEFAULT_URIGPG_AGENT_INFODESKTOP_SESSIONGJS_DEBUG_OUTPUTXDG_MENU_PREFIXUSERQT_IM_MODULELC_MEASUREMENTVTE_VERSIONDBUS_SESSION_BUS_ADDRESSPWDLC_NUMERICGTK_MODULES_WINDOWPATHXDG_SESSION_DESKTOPJOURNAL_STREAMQT_ACCESSIBILITYHOMEGNOME_DESKTOP_SESSION_IDMANAGERPIDLC_TIMEXDG_DATA_DIRSGNOME_TERMINAL_SERVICELC_PAPERLOGNAMEMANDATORY_PATHXDG_RUNTIME_DIRXDG_CONFIG_DIRSXDG_SESSION_TYPEXMODIFIERSPATHLC_TELEPHONELC_MONETARYSHELLGNOME_SHELL_SESSION_MODEUSERNAMEINVOCATION_IDSHLVLXAUTHORITYLC_NAMEIM_CONFIG_PHASETERMLC_ADDRESSDISPLAYGDMSESSIONZSHPAGERLESSLSCOLORSLS_COLORSCLOUDSDK_HOMEAPPLIED_SCRIPT_HOMEGOOGLE_APPLICATION_CREDENTIALSGOOGLE_ACCOUNT_SERVICE_FILENVM_DIRNVM_CD_FLAGSNVM_BINNVM_INCpipfile_dirPIP_DISABLE_PIP_VERSION_CHECKPYTHONDONTWRITEBYTECODEPIP_PYTHON_PATHPIPENV_ACTIVEVIRTUAL_ENVPS1PIP_SHIMS_BASE_MODULEPYTHONFINDER_IGNORE_UNSUPPORTEDPipenv–specific environment variables:
PIPENV_ACTIVE:1Debug–specific environment variables:
PATH:/home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin:/home/klaus/bin:/opt/ghc/bin:/home/klaus/.nvm/versions/node/v15.1.0/bin:/home/klaus/bin:/opt/ghc/bin:/home/klaus/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/binSHELL:/usr/bin/zshLANG:en_US.UTF-8PWD:/home/klaus/src/applied_mapperVIRTUAL_ENV:/home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4Contents of
Pipfile('/home/klaus/src/applied_mapper/Pipfile'):The text was updated successfully, but these errors were encountered: