pip will hang asking for the password of an index server even if the output is redirected

[RonnyPfannschmidt]

No description provided.

[rbtcollins]

pip can't tell the difference between | tee my.log (where you can see the prompt and enter it) and > log.txt.

If you want to run it headless, be sure to redirect input < /dev/null as well. That should make it error rather than hang. If it doesn't, then thats a bug, but on the information so far, I don't see the bug.

[untitaker]

pip should be able to tell the difference. sys.stdout.isatty can be used for this.

[pradyunsg]

@pypa/pip-committers

Between "redirect input < /dev/null" vs "tell the difference sys.stdout.isatty"; which way do you guys think we should go?

/note-to-self mark as awaiting PR once decided.

[chrahunt]

I can confirm the situation has not changed with pip 19.1.1:

Given server.py:

from http.server import BaseHTTPRequestHandler, HTTPServer


class PromptForPassword(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(401)
        self.send_header('WWW-Authenticate', 'Basic')
        self.end_headers()


if __name__ == '__main__':
    server = HTTPServer(('', 8888), PromptForPassword)
    server.serve_forever()

Run python server.py >/dev/null 2>&1 &.

Then the following is pip's current behavior:

No redirection, both "User" and "Password" prompts are visible:

$ pip install --index-url http://localhost:8888/simple requests
Looking in indexes: http://localhost:8888/simple
Collecting requests
User for localhost:8888: guest
Password:
  WARNING: 401 Error, Credentials not correct for http://localhost:8888/simple/requests/
  ERROR: Could not find a version that satisfies the requirement requests (from versions: none)
ERROR: No matching distribution found for requests

With redirection, the "User" prompt is not visible and the "Password" prompt is visible:

$ pip install --index-url http://localhost:8888/simple requests > /dev/null
guest
Password:
  WARNING: 401 Error, Credentials not correct for http://localhost:8888/simple/requests/
  ERROR: Could not find a version that satisfies the requirement requests (from versions: none)
ERROR: No matching distribution found for requests

---

A quick test confirms that isatty cannot distinguish between a user-visible prompt and one that is redirected to a file:

$ alias example='python -c '"'"'import sys, pathlib; pathlib.Path("o").write_text(f"{sys.stdin.isatty()}; {sys.stdout.isatty()}\n")'"'"
$ example
True;True
$ example >/dev/null && cat o
True;False
$ example | tee /dev/null && cat o
True;False
$ example </dev/null && cat o
False;True

I think it is very common to behave differently for TTYs, with an option to force the behavior (see e.g. diff, git, ls and color output), so we should not worry about trying to prompt in the edge case where a user may be able to see the prompt but isatty is False.

---

I would propose the following approach: only prompt if sys.stdout.isatty() and sys.stdin.isatty() and then add a command-line flag (--no-prompt-auth, --prompt-auth) to force one way or another for users that want to ensure correct behavior.

This would give us an opportunity to give a descriptive error message instead of relying on the fact that </dev/null causes an exception:

$ pip install --index-url http://localhost:8888/simple requests < /dev/null

Looking in indexes: http://localhost:8888/simple
Collecting requests
User for localhost:8888: ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 178, in main
    status = self.run(options, args)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 352, in run
    resolver.resolve(requirement_set)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 282, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 198, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/index.py", line 766, in find_requirement
    candidates = self.find_candidates(req.name, req.specifier)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/index.py", line 752, in find_candidates
    self.find_all_candidates(project_name),
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/index.py", line 716, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/index.py", line 846, in _get_pages
    page = _get_html_page(location, session=self.session)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/index.py", line 231, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/index.py", line 179, in _get_html_response
    "Cache-Control": "max-age=0",
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/download.py", line 439, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 653, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/tmp/pip/2920/.venv/lib/python3.7/site-packages/pip/_internal/download.py", line 233, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

It's probably worth considering that the person that would set up scripts/CI to use this approach would not be the same one debugging it later faced with the exception above. Having an explicit option will be cross-platform, easier to google, and easier to test.

[pradyunsg]

Thanks for the investigation here @chrahunt!

Hmm, seems like requests is showing that prompt. Maybe we can tell it not to do that and raise an exception instead?

[cs01]

Just a note that pipx is impacted by this behavior and would benefit from this change, pypa/pipx#219.

Ideally we would be able to pass a --non-interactive or --ci flag, but we could also work with pip being tty-aware and just call pip with output redirected.

[chrahunt]

Have you tried --no-input? It's not documented for some reason (see #2429 (comment)) but should work to avoid prompting for HTTP-based URLs.

[cs01]

I haven't, but that looks like exactly what we're looking for. Thank you!

[mfleader]

My environment, briefly

❯ cat /etc/fedora-release
Fedora release 38 (Thirty Eight)
❯ python --version
Python 3.10.13
❯ python -m pip list
Package    Version
---------- -------
pip        23.3.2
setuptools 69.0.2

Terse example

❯ python -m pip --no-input install nonexistent-module@git+https://github.com/mfleader/nonexistent-module.git &>/dev/null
Username for 'https://github.com': 
Password for 'https://github.com': 

Terse example with stdout and stderr

❯ python -m pip --no-input install nonexistent-module@git+https://github.com/mfleader/nonexistent-module.git
Collecting nonexistent-module@ git+https://github.com/mfleader/nonexistent-module.git
  Cloning https://github.com/mfleader/nonexistent-module.git to /tmp/pip-install-b_ezznu_/nonexistent-module_148ce4e8ef254ae4a1d9ed4bb09b9337
  Running command git clone --filter=blob:none --quiet https://github.com/mfleader/nonexistent-module.git /tmp/pip-install-b_ezznu_/nonexistent-module_148ce4e8ef254ae4a1d9ed4bb09b9337
Username for 'https://github.com': 
Password for 'https://github.com': 
  remote: Repository not found.
  fatal: Authentication failed for 'https://github.com/mfleader/nonexistent-module.git/'
  error: subprocess-exited-with-error
  
  × git clone --filter=blob:none --quiet https://github.com/mfleader/nonexistent-module.git /tmp/pip-install-b_ezznu_/nonexistent-module_148ce4e8ef254ae4a1d9ed4bb09b9337 did not run successfully.
  │ exit code: 128
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet https://github.com/mfleader/nonexistent-module.git /tmp/pip-install-b_ezznu_/nonexistent-module_148ce4e8ef254ae4a1d9ed4bb09b9337 did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

From the error message, the prompt seemed to be coming from the subprocess execution of git clone. I prepended GIT_TERMINAL_PROMPT=0 and it forced non-interactive execution.

❯ GIT_TERMINAL_PROMPT=0 python -m pip install nonexistent-module@git+https://github.com/mfleader/nonexistent-module.git &> /dev/null
❯ echo $?
1