Status code 403 with pip 21.1.x and keyrings.google-artifactregistry-auth 0.0.2

[looztra]

• WSL2/Fedora 33
• Python 3.9.2
• works with pip 21.0.1
• cannot install a package located in my GAR with pip 21.1 or 21.1.1.

Getting page https://europe-pypi.pkg.dev/xxxxxx-xix9/python/simple/cpe-common/
Found index url https://europe-pypi.pkg.dev/xxxxxx-xix9/python/simple/
Looking up "https://europe-pypi.pkg.dev/xxxxxx-xix9/python/simple/cpe-common/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): europe-pypi.pkg.dev:443
https://europe-pypi.pkg.dev:443 "GET /xxxxxx-xix9/python/simple/cpe-common/ HTTP/1.1" 403 0
Status code 403 not in (200, 203, 300, 301)
Could not fetch URL https://europe-pypi.pkg.dev/xxxxxx-xix9/python/simple/cpe-common/: 403 Client Error: Forbidden for url: https://europe-pypi.pkg.dev/xxxxxx-xix9/python/simple/cpe-common/ - skipping

╰(.venv)─» pip freeze
appdirs==1.4.4
bleach==3.3.0
cachetools==4.2.2
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
colorama==0.4.4
cryptography==3.4.7
distlib==0.3.1
docutils==0.17.1
filelock==3.0.12
google-auth==1.30.0
idna==2.10
importlib-metadata==4.0.1
jeepney==0.6.0
keyring==23.0.1
keyrings.google-artifactregistry-auth==0.0.2
packaging==20.9
pkginfo==1.7.0
pluggy==0.13.1
py==1.10.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
Pygments==2.9.0
pyparsing==2.4.7
readme-renderer==29.0
requests==2.25.1
requests-toolbelt==0.9.1
rfc3986==1.4.0
rsa==4.7.2
SecretStorage==3.3.1
six==1.15.0
toml==0.10.2
tox==3.23.0
tqdm==4.60.0
twine==3.4.1
urllib3==1.26.4
virtualenv==20.4.4
webencodings==0.5.1
zipp==3.4.1

╰(.venv)─» keyring --list-backends
keyrings.gauth.GooglePyPIAuth (priority: 9)
keyring.backends.chainer.ChainerBackend (priority: -1)
keyring.backends.fail.Keyring (priority: 0)

[looztra]

It looks like the keyring is not called at all with pip 21.1.x, because using pip 21.0.1, I can see the logs messages from the keyring that I don't see with pip 21.1.x

Getting page https://europe-pypi.pkg.dev/xxxxxx-xix9/python/simple/cpe-common/
Found index url https://europe-pypi.pkg.dev/xxxxxx-xix9/python/simple/
Getting credentials from keyring for https://europe-pypi.pkg.dev/xxxxxx-xix9/python/simple/
Checking None for explicit credentials as part of auth process...
Checking Cloud SDK credentials as part of auth process...
Making request: POST https://oauth2.googleapis.com/token
Starting new HTTPS connection (1): oauth2.googleapis.com:443
https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
Found credentials in keyring for europe-pypi.pkg.dev
Looking up "https://europe-pypi.pkg.dev/xxxxxx-xix9/python/simple/cpe-common/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): europe-pypi.pkg.dev:443

[looztra]

maybe linked to pypa/pip#9870

[looztra]

@megan-kuo @Gridasov : can you confirm something changed very recently and now the GAR returns a 401 instead of a 403 when not authenticated?
This generated another kind of issue on our side, cf pypa/pip#9870 (comment)

[megan-kuo]

Yes, Artifact Registry now returns a 401 instead of a 403 when not authenticated.

Can you elaborate a bit more on your issue? It is the intended behavior that users should have to set up authentication by installing keyrings.google-artifactregistry-auth before installing packages from an Artifact Registry Repository (see this quickstart guide here: [https://cloud.google.com/artifact-registry/docs/python/quickstart]

Also, you can use ar-contact@google.com to reach our team :)

[looztra]

• Given I have a pip.conf that refers to a GAR url as an extra-url

[global]
timeout = 5
extra-index-url = https://europe-pypi.pkg.dev/my-gcp-project/python/simple/

• Given I am in a new virtual env that does not have the keyring packages installed yet (keyring==23.0.1 keyrings.google-artifactregistry-auth==0.0.2)
• When I want to install the packages keyring and keyrings.google-artifactregistry-auth==0.0.2
• Then I am prompted for a Username/Password when pip tries to reach the package repository hosted on the GAR

If the expected mandatory behaviour is to install the keyrings packages before configuring the pip.conf, what is the way to do it inside tox venv? Using a custom install_command:

[testenv]
install_command = {toxinidir}/path/to/tox_custom_install_command.sh {opts} {packages}

• tox_custom_install_command.sh content:

#!/usr/bin/env bash

PIP_CONFIG_FILE=/dev/null pip install --upgrade pip==21.1.1
PIP_CONFIG_FILE=/dev/null pip install 'keyring>=23.0.1'
PIP_CONFIG_FILE=/dev/null pip install 'keyrings.google-artifactregistry-auth>=0.0.2'
pip install "$@"

?
This feels more like a hack to me, but I may be wrong :)

[gaborbernat]

Hello, I'm both tox and virtualenv maintainer. @di summoned me here.

If the expected mandatory behaviour is to install the keyrings packages before configuring the pip.conf, what is the way to do it inside tox venv?

tox under the hood uses virtualenv, not venv. The problem described to me sounds like a valid virtual environment on these machines is one that doesn't just have pip+setuptools+wheel in it but also keyring and keyrings.google-artifactregistry-auth. Therefore my recommendation is to alter the virtual environment creation accordingly, you can extend virtualenv package seed logic, see https://virtualenv.pypa.io/en/latest/extend.html#seed-mechanism. Likely your best option is to extend the current seeder and seed these extra packages afterward. You can then install this tox plugin within the machine manually. Sadly you can't use https://tox.readthedocs.io/en/latest/config.html#conf-requires because that creates a virtual environment without this plugin being already active (chicken-egg problem).

Your hacky solution works too, but is kinda hacky 😮

[di]

Hi @looztra, once #9 is released, this should be resolved by configuring your project as described in https://github.com/GoogleCloudPlatform/artifact-registry-python-tools#usage-with-tox instead. Will be interested to hear if that works for you!

[looztra]

@di this doesn't solve my problem, because I have an existing pip.conf file that references the GAR, so when trying to create the venv, all the pip commands that try to install packages on pypi fail

[di]

@looztra Instead of globally configuring pip with a pip.conf file here, you should add the following to the top of any requirements.txt file that needs the extra index instead:

--extra-index-url https://europe-pypi.pkg.dev/my-gcp-project/python/simple/

[looztra]

I'm experimenting this way to do it, so far so good.
thx for the hint @di