Exception when installing Git VCS dependency with plus symbol in HTTP basic auth #4271

AdrianoKF · 2020-05-28T09:51:38Z

Issue description

Running pipenv install with a Git VCS dependency that has a plus symbol in the HTTP basic auth part of the URL produces an InstallationError due to a failed HTTP basic authentication.

It seems the URL is internally rewritten incorrectly, since the output logs the URL with the plus symbol replaced by a space (%20).

The behaviour was not observed in pipenv==2018.11.26, but occurs in latest pipenv==2020.5.28.

Expected result

Pipenv should install the VCS dependency without errors.

Actual result

pipenv install raises an InstallationError. Note the Git clone URL in the error message: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git', which contains a space character instead of the plus sign:

Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…

⠋
Building requirements...
 Locking...
Resolving dependencies...
�
⠙ Locking...�
⠹ Locking...�
⠸ Locking...�
⠼ Locking...�
⠴ Locking...�
⠦ Locking...�
⠧ Locking...�
⠇ Locking...�
⠏ Locking...�
⠋ Locking...�
⠙ Locking...ERROR:pip.subprocessor:Command errored out with exit status 128:
 command: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms
     cwd: None
Complete output (2 lines):
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aai.lab/tms/tms-core.git/'
----------------------------------------
�
⠹ Locking...�✘ Locking Failed! 
Traceback (most recent call last):
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1090, in actually_resolve_deps
    resolver = Resolver.create(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 647, in create
    constraints, skipped, index_lookup, markers_lookup = cls.get_metadata(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 462, in get_metadata
    constraint_update, lockfile_update = cls.get_deps_from_req(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 540, in get_deps_from_req
    req_list, lockfile = get_vcs_deps(reqs=[req])
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1874, in get_vcs_deps
    with temp_path(), locked_repository(requirement) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 2031, in locked_repository
    with requirement.req.locked_vcs_repo(src_dir=src_dir) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2194, in locked_vcs_repo
    vcsrepo = self.get_vcs_repo(src_dir=src_dir)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2153, in get_vcs_repo
    vcsrepo.obtain()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/vcs.py", line 77, in obtain
    self.repo_backend.obtain(self.checkout_directory, self.parsed_url)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 527, in obtain
    self.fetch_new(dest, url, rev_options)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/git.py", line 225, in fetch_new
    self.run_command(make_command('clone', '-q', url, dest))
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 660, in run_command
    return call_subprocess(cmd, show_stdout, cwd,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 128: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms Check the logs for full command output.
ERROR:pip.subprocessor:Command errored out with exit status 128:
 command: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms
     cwd: None
Complete output (2 lines):
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aai.lab/tms/tms-core.git/'
----------------------------------------
Traceback (most recent call last):
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1090, in actually_resolve_deps
    resolver = Resolver.create(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 647, in create
    constraints, skipped, index_lookup, markers_lookup = cls.get_metadata(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 462, in get_metadata
    constraint_update, lockfile_update = cls.get_deps_from_req(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 540, in get_deps_from_req
    req_list, lockfile = get_vcs_deps(reqs=[req])
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1874, in get_vcs_deps
    with temp_path(), locked_repository(requirement) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 2031, in locked_repository
    with requirement.req.locked_vcs_repo(src_dir=src_dir) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2194, in locked_vcs_repo
    vcsrepo = self.get_vcs_repo(src_dir=src_dir)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2153, in get_vcs_repo
    vcsrepo.obtain()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/vcs.py", line 77, in obtain
    self.repo_backend.obtain(self.checkout_directory, self.parsed_url)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 527, in obtain
    self.fetch_new(dest, url, rev_options)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/git.py", line 225, in fetch_new
    self.run_command(make_command('clone', '-q', url, dest))
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 660, in run_command
    return call_subprocess(cmd, show_stdout, cwd,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 128: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms Check the logs for full command output.

Steps to replicate

Command: pipenv install -d

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
tms = {ref = "develop",git = "https://gitlab+deploy-token-3:nHLsRjx5eCpU6zxmj5sE@gitlab.aai.lab/tms/tms-core.git"}

[packages]

[requires]
python_version = "3.8"

$ pipenv --support

Pipenv version: '2020.5.28'

Pipenv location: '/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv'

Python location: '/home/adriano/.virtualenvs/pipenv2020/bin/python'

Python installations found:

3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3
3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3.8
3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3
3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3.8
3.8.2: /usr/bin/python3
3.8.2: /usr/bin/python3.8
3.8.2: /bin/python3
3.8.2: /bin/python3.8
3.7.7: /home/adriano/.pyenv/versions/3.7.7/bin/python3
2.7.18: /usr/bin/python2.7
2.7.18: /usr/bin/python2
2.7.18: /bin/python2.7
2.7.18: /bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.6.12-1-MANJARO',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Sun May 10 14:36:43 UTC 2020',
 'python_full_version': '3.8.2',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

SHELL
LSCOLORS
WINDOWID
PYENV_HOOK_PATH
COLORTERM
PYENV_SHELL
CSF_MDTVTexturesDirectory
LESS
XDG_SESSION_PATH
CSF_DrawPluginDefaults
I3SOCK
LC_ADDRESS
CSF_LANGUAGE
SSH_AUTH_SOCK
CSF_MIGRATION_TYPES
DESKTOP_SESSION
LC_MONETARY
SSH_AGENT_PID
CSF_OCCTResourcePath
CSF_STEPDefaults
CLOUDSDK_PYTHON_ARGS
PYENV_VERSION
GTK_MODULES
XDG_SEAT
PWD
XDG_SESSION_DESKTOP
LOGNAME
XDG_SESSION_TYPE
DRAWHOME
XAUTHORITY
CSF_StandardLiteDefaults
XDG_GREETER_DATA_DIR
VIRTUALENVWRAPPER_SCRIPT
HOME
LC_PAPER
LANG
LS_COLORS
XDG_CURRENT_DESKTOP
VIRTUAL_ENV
VTE_VERSION
CLOUDSDK_ROOT_DIR
VIRTUALENVWRAPPER_WORKON_CD
XDG_SEAT_PATH
CSF_ShadersDirectory
CSF_EXCEPTION_PROMPT
PYENV_DIR
CSF_XmlOcafResource
CLOUDSDK_PYTHON
WORKON_HOME
CSF_SHMessage
XDG_SESSION_CLASS
TERM
ZSH
GOOGLE_CLOUD_SDK_HOME
USER
CSF_StandardDefaults
CSF_IGESDefaults
VIRTUALENVWRAPPER_PROJECT_FILENAME
DISPLAY
CSF_XCAFDefaults
SHLVL
PAGER
LC_MESSAGES
LC_MEASUREMENT
XDG_VTNR
CSF_PluginDefaults
CSF_TObjMessage
XDG_SESSION_ID
CASROOT
XDG_RUNTIME_DIR
PYENV_ROOT
LC_TIME
CSF_XSMessage
MMGT_CLEAR
PATH
CSF_TObjDefaults
VIRTUALENVWRAPPER_HOOK_DIR
GDMSESSION
DBUS_SESSION_BUS_ADDRESS
MAIL
DRAWDEFAULT
OLDPWD
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /usr/share/pyenv/libexec:/home/adriano/.virtualenvs/pipenv2020/bin:/home/adriano/.yarn/bin:/home/adriano/.yarn/bin:/home/adriano/.config/yarn/global/node_modules/.bin:/home/adriano/.pyenv/shims:/home/adriano/.pyenv/bin:/home/adriano/.local/bin:/opt/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/adriano/.local/bin
SHELL: /usr/bin/zsh
LANG: en_US.UTF-8
PWD: /home/adriano/playground/pipenv-2020-url-sanitation
VIRTUAL_ENV: /home/adriano/.virtualenvs/pipenv2020

Contents of Pipfile ('/home/adriano/playground/pipenv-2020-url-sanitation/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
tms = {ref = "develop",git = "https://gitlab+deploy-token-3:nHLsRjx5eCpU6zxmj5sE@gitlab.aai.lab/tms/tms-core.git"}

[packages]

[requires]
python_version = "3.8"

The text was updated successfully, but these errors were encountered:

zpsjs · 2020-05-28T11:44:15Z

Got the same issue with the actual version:

ERROR:pip.subprocessor:Command errored out with exit status 128: command: git clone -q 'https://****@github.com/...' /tmp/reqlib-srcfjz5vsf1/... cwd: None Complete output (1 lines): fatal: could not read Password for 'https://${ACCESS_TOKEN}@github.com': No such device or address

Pipenv is asking for password although I'm using a token.

AdrianoKF · 2020-05-28T12:51:55Z

After some digging, it seems that pipenv.vendor.requirementslib.models.url.URI doesn't handle this edge case correctly and replaces + with a space in the URL when converting the parsed URI to a str (note, that username is correctly populated in the returned URI instance but broken in the returned string):

>>> from pipenv.vendor.requirementslib.models.url import URI
>>> url = 'git+https://foo+bar:baz@example.org'
>>> URI.parse(url)
URI(host='example.org', scheme='git+https', port=None, path=None, query='', fragment=None, subdirectory=None, ref='', username='foo+bar', query_dict=omdict([('', '')]), name='', extras=(), is_direct_url=False, is_implicit_ssh=False, _fragment_dict={}, _username_is_quoted=True, _password_is_quoted=False)
>>> print(URI.parse(url))
git+https://foo bar:----@example.org

RFC 3986 specifies that the userinfo subcomponent of a URI may contain the sub-delim tokens, so it is okay to include a literal plus sign (not percent-encoded) there, I believe (https://tools.ietf.org/html/rfc3986#section-3.2.1). requirementslib seems to incorrectly assume that the field is always percent-encoded, which is why the + is replaced with a space in URI.get_username().

techalchemy · 2020-05-28T15:51:00Z

Thanks for the report, and for the debugging, and sorry for the issue!

triage-new-issues bot added the triage label May 28, 2020

AdrianoKF mentioned this issue May 28, 2020

URI.get_username() and URI.get_password() replace literal plus signs with spaces sarugaku/requirementslib#245

Closed

GPHemsley mentioned this issue May 28, 2020

Regression in 2020.05.28 -- pipenv sync fails in Gitlab CI context #4272

Closed

techalchemy added Priority: High This item is high priority and should be resolved quickly. Type: Bug 🐛 This issue is a bug. Type: Vendored Dependencies This issue affects vendored dependencies within pipenv. labels May 28, 2020

triage-new-issues bot removed the triage label May 28, 2020

frostming added this to the 2020.6.x bugfix release milestone Jun 1, 2020

frostming mentioned this issue Jun 1, 2020

Update vendor libraries #4302

Merged

2 tasks

techalchemy closed this as completed in #4302 Jun 1, 2020

dependabot bot mentioned this issue Mar 14, 2021

Bump pipenv from 2018.11.26 to 2020.11.15 in /python/helpers codisart/dependabot-core#1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception when installing Git VCS dependency with plus symbol in HTTP basic auth #4271

Exception when installing Git VCS dependency with plus symbol in HTTP basic auth #4271

AdrianoKF commented May 28, 2020

zpsjs commented May 28, 2020

AdrianoKF commented May 28, 2020 •

edited

Loading

techalchemy commented May 28, 2020

Exception when installing Git VCS dependency with plus symbol in HTTP basic auth #4271

Exception when installing Git VCS dependency with plus symbol in HTTP basic auth #4271

Comments

AdrianoKF commented May 28, 2020

Issue description

Expected result

Actual result

Steps to replicate

zpsjs commented May 28, 2020

AdrianoKF commented May 28, 2020 • edited Loading

techalchemy commented May 28, 2020

AdrianoKF commented May 28, 2020 •

edited

Loading