pip freeze prints out username and password from URLs with them

[sbhattathiri]

Description

When I do pip freeze, it prints out the username and password that I included while cloning the local modules \ repositories

Similar to : #4746

Expected behavior

Passwords should'nt be shown

pip version

21.1.1

Python version

3.8.10

OS

Centos7

How to Reproduce

1. Clone a repository to local using username and password :
   git clone -b my_branch https://$gusr:$gpwd@gitlab.com/my-repo.git
2. Do a local install of the repo
   cd my-repo
pip install -e .
3. Do pip freeze

Output

prints out my user credentials

-e git+https://gusr:gpwd@gitlab.com/my-repo#egg=repo_name

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

A pull request fixing this would be welcome!

We have a redact_auth_from_url helper, that likely needs to be called on the corresponding string here, before printing it.

[uranusjr]

Personally I’m not actually sure this is undesired. The freeze file is supposed to reflect the installation (different from stdout as in the linked issue, which is merely informational), and redacting the password means a part of the installation information is lost in the process. For this instance specifically, the user passed in the password in plain text (from pip’s perspective; the password was in plain text in the environment varialbe when pip was invoked and was resolved outside of pip by the shell), so I would expect the user to accept the password to be spit out in plain text, and modify the file afterward to accomodate the fact.

[pfmoore]

I agree. Presumably the password is also stored in the direct_url.json file from pep 610, so it’s already exposed to anyone who can run pip freeze.

[sbidoul]

When doing pip install git+https://user:pwd@github.com/pypa/pip-test-package, the credentials are stripped from the URL in direct_url.json and therefore not printed by pip freeze. That behaviour is actually specified in PEP 610.

In the scenario of the OP, the credentials are stored in .git/config and also revealed with git remote -v.
That is where pip freeze gets them from, as it tries to be smart when it detects editable local directories from a VCS.

So we have a minor discrepancy between editables and non-editable VCS installs.

[connesy]

@sbidoul What is the reasoning for stripping the credentials from a VCS URL in direct_url.json when specified as environment variables?

Specifically, I'm referring to this section of the specification in PEP 610:

url MUST be stripped of any sensitive authentication information, for security reasons.
The user:password section of the URL MAY however be composed of environment variables [...]

[sbidoul]

@connesy I had lost track of this.

What is the reasoning for stripping the credentials from a VCS URL in direct_url.json when specified as environment variables?

I don't understand your question, actually. When the credentials are specified as environment variables, they should not be stripped. Di you suspect a bug in that area?

[connesy]

@sbidoul I have a requirements.txt file that contains the line

ml-framework @ git+https://${USERNAME}:${PASSWORD}@gitlab.com/ds-api/team-ai-projects/team-ai-internal-projects/ml_framework.git@0.10.2

When installed, this results in the following ml_framework-0.10.2.dist-info/direct_url.json:

{
  "url": "https://gitlab.com/ds-api/team-ai-projects/team-ai-internal-projects/ml_framework.git",
  "vcs_info": {
    "commit_id": "fcff07aa9c2d4eec7277b704f0433cf07157910d",
    "requested_revision": "fcff07aa9c2d4eec7277b704f0433cf07157910d",
    "vcs": "git"
  }
}

[connesy]

Here's a reproducible example:

$ mkdir pip-test; cd pip-test
$ echo "resolvelib @ git+https://${TEST_USER}:${TEST_PASS}@github.com/sarugaku/resolvelib@1.0.1" > requirements.txt
$ python3 -m venv .venv
$ . .venv/bin/activate
$ export TEST_USER="test-user"
$ export TEST_PASS="test-pass"
$ pip install -r requirements.txt
Collecting resolvelib@ git+https://test-user:test-pass@github.com/sarugaku/resolvelib@1.0.1
  Cloning https://test-user:****@github.com/sarugaku/resolvelib (to revision 1.0.1) to /tmp/pip-install-tc_m1ch1/resolvelib_3b1b65ad2a5f40d3abd6e2ad2b47bf40
  Running command git clone --filter=blob:none --quiet 'https://test-user:****@github.com/sarugaku/resolvelib' /tmp/pip-install-tc_m1ch1/resolvelib_3b1b65ad2a5f40d3abd6e2ad2b47bf40
  Running command git checkout -q 969c5c75ded426dac43eff9619e1e857f22f37bb
  Resolved https://test-user:****@github.com/sarugaku/resolvelib to commit 969c5c75ded426dac43eff9619e1e857f22f37bb
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: resolvelib
  Building wheel for resolvelib (pyproject.toml) ... done
  Created wheel for resolvelib: filename=resolvelib-1.0.1-py2.py3-none-any.whl size=17195 sha256=65a348cbee61a74aca5e5607396966e334d5f8e4e8e265948390e0ed5dfe7b32
  Stored in directory: /tmp/pip-ephem-wheel-cache-itj5byo9/wheels/d2/9b/d2/f0b866d96f90c3c83d59b56cb3f2925028a79a0742f1c9e52d
Successfully built resolvelib
Installing collected packages: resolvelib
Successfully installed resolvelib-1.0.1
$ python -m json.tool .venv/lib/python3.10/site-packages/resolvelib-1.0.1.dist-info/direct_url.json
{
    "url": "https://github.com/sarugaku/resolvelib",
    "vcs_info": {
        "commit_id": "c9ef371ad96e698bf3e0bb09acc682bd43e39bd7",
        "requested_revision": "1.0.1",
        "vcs": "git"
    }
}

The pip install log shows that the credentials are resolved in the URL by resolvelib and logged in plaintext, then pip install logs them with the password in asterisks, but they don't end up in direct_url.json.

[sbidoul]

Thanks for the reproducer. This looks like a bug, I'll investigate.

[sbidoul]

The fix for the plain text password in logs is in #12350.

[sbidoul]

The OP is about editable installs. That is something we may want to fix although I am not entirely sure how. Probably stripping the user and password entirely, because the environment variables names are lost at that point, as they have been resolved by the shell. This part about editables is unrelated to direct URLs, since for editables, direct_url.json contains the local directory, and pip freeze obtains the VCS URL from the local clone. In the case of the OP, the user and password are stored in clear text in the .git/config.

@connesy's #11410 (comment) is a different issue.
The environment variables are stripped in direct_url.json because they have been resolved early by the requirement file parser.

[connesy]

@sbidoul Do you want me to create a separate issue on this?

[sbidoul]

Do you want me to create a separate issue on this?

Yes please. Thank you.