Injecting credentials into Pipfiles via environment variables not working in [packages] section on Win10

[OlafHaag]

Issue description

On Windows 10 environment variables in git URLs using ${VAR} within the [packages] section of the Pipfile aren't expanded. Using %VAR% works on Windows, but isn't cross platform compatible.

This issue might be related to #2635 , #2710

Steps to replicate

Tried to follow guidelines: Injecting credentials into Pipfiles via environment variables
Pipfile:

[packages]
examplepackage = {editable = true, ref = "master", git = "ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git"}

Command: pipenv install --skip-lock

Expected result

Obtaining examplepackage from git+ssh://olaf@192.168.1.64:22/home/repos/ExamplePackage.git@master#egg=examplepackage

Actual result

Obtaining examplepackage from git+ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git@master#egg=examplepackage
Permission denied

Full output

$ pipenv install --skip-lock --verbose

Creating a virtualenv for this project…
Pipfile: D:\Example\Pipfile
Using D:/Anaconda3/python.exe (3.7.0) to create virtualenv…
Running virtualenv with interpreter D:/Anaconda3/python.exe
Using base prefix 'D:\Anaconda3'
d:\anaconda3\lib\site-packages\virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
New python executable in C:\Users\olaf.virtualenvs\Example-MAec1j9C\Scripts\python.exe
Installing setuptools, pip, wheel...done.

Virtualenv location: C:\Users\olaf.virtualenvs\Example-MAec1j9C
Installing -e git+ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git@master#egg=examplepackage…
Installing 'examplepackage'
$ ['C:\Users\olaf\.virtualenvs\Example-MAec1j9C\Scripts\pip.exe', 'install', '--src', 'C:\Users\olaf\.virtualenvs\Example-MAec1j9C\src', '--verbose', '--upgrade', '-e', '"git+ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git@master#egg=examplepackage"', '-i', 'https://pypi.org/simple']
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Created temporary directory: C:\Users\olaf\AppData\Local\Temp\pip-ephem-wheel-cache-sznniqzz
Created temporary directory: C:\Users\olaf\AppData\Local\Temp\pip-req-tracker-eadbpfuh
Created requirements tracker 'C:\Users\olaf\AppData\Local\Temp\pip-req-tracker-eadbpfuh'
Created temporary directory: C:\Users\olaf\AppData\Local\Temp\pip-install-nhvhq061
Obtaining examplepackage from git+ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git@master#egg=examplepackage
Cloning ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git (to revision master) to c:\users\olaf.virtualenvs\Example-maec1j9c\src\examplepackage
Running command git clone -q ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git C:\Users\olaf.virtualenvs\Example-MAec1j9C\src\examplepackage
${USER}@192.168.1.64: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Cleaning up...
Removed build tracker 'C:\Users\olaf\AppData\Local\Temp\pip-req-tracker-eadbpfuh'
Exception information:
Traceback (most recent call last):
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\cli\base_command.py", line 143, in main
status = self.run(options, args)
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\commands\install.py", line 318, in run
resolver.resolve(requirement_set)
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\resolve.py", line 102, in resolve
self._resolve_one(requirement_set, req)
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\resolve.py", line 256, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\resolve.py", line 193, in get_abstract_dist_for
req, self.require_hashes, self.use_user_site, self.finder,
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\operations\prepare.py", line 321, in prepare_editable_requirement
req.update_editable(not self.download_should_save)
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\req\req_install.py", line 654, in update_editable
vcs_backend.obtain(self.source_dir)
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\vcs_init.py", line 332, in obtain
self.fetch_new(dest, url, rev_options)
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\vcs\git.py", line 198, in fetch_new
self.run_command(['clone', '-q', url, dest])
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\vcs_init.py", line 456, in run_command
spinner=spinner)
File "c:\users\olaf.virtualenvs\Example-maec1j9c\lib\site-packages\pip_internal\utils\misc.py", line 705, in call_subprocess
% (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError: Command "git clone -q ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git C:\Users\olaf.virtualenvs\Example-MAec1j9C\src\examplepackage" failed with error code 128 in None

Error: An error occurred while installing -e git+ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git@master#egg=examplepackage!
Command "git clone -q ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git C:\Users\olaf.virtualenvs\Example-MAec1j9C\src\examplepackage" failed with error code 128 in None

---

$ pipenv --support

Pipenv version: '2018.10.13'

Pipenv location: 'd:\\anaconda3\\lib\\site-packages\\pipenv'

Python location: 'd:\\anaconda3\\python.exe'

Python installations found:

• 5.3.0: D:\Anaconda3\python.exe
• 2.7: C:\Program Files\WorldViz\Vizard6\bin\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.17134',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'win32'}

System environment variables:

• ADSK_3DSMAX_X64_2017
• ADSK_3DSMAX_X64_2019
• ADSK_CLM_WPAD_PROXY_CHECK
• ALLUSERSPROFILE
• APPDATA
• ARCH
• BOOST_ROOT
• CHOCOLATEYINSTALL
• CHOCOLATEYLASTPATHUPDATE
• CHOCOLATEYTOOLSLOCATION
• CLASSPATH
• COMMONPROGRAMFILES
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• COMPILER
• COMPUTERNAME
• COMSPEC
• DOKANLIBRARY1
• DRIVERDATA
• FPS_BROWSER_APP_PROFILE_STRING
• FPS_BROWSER_USER_PROFILE_STRING
• GENERATOR
• HOMEDRIVE
• HOMEPATH
• JAVA_HOME
• LIBEVENT_DIR
• LOCALAPPDATA
• LOGONSERVER
• MOBU_DIR
• NUMBER_OF_PROCESSORS
• NVTOOLSEXT_PATH
• OMP_NUM_THREADS
• ONEDRIVE
• OPENSSL_ROOT_DIR
• OS
• PATH
• PATHEXT
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMDATA
• PROGRAMFILES
• PROGRAMFILES(X86)
• PROGRAMW6432
• PROMPT
• PSMODULEPATH
• PUBLIC
• SESSIONNAME
• SSH_AUTH_SOCK
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• THRIFT_DIR
• USER
• TMP
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• VS110COMNTOOLS
• VS140COMNTOOLS
• WINDIR
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH

Pipenvûspecific environment variables:

Debugûspecific environment variables:

• PATH: C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\ProgramData\chocolatey\bin;C:\Program Files\MiKTeX 2.9\miktex\bin\x64\;C:\Program Files (x86)\QuickTime\QTSystem\;C:\Program Files (x86)\GNU\GnuPG\pub;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Common Files\Autodesk Shared\;C:\Program Files\Java\jdk1.8.0_112\bin;C:\Program Files\Java\jdk1.8.0_121\bin;C:\Program Files (x86)\PuTTY\;C:\Program Files\Java\jdk1.8.0_131\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\tools\go\bin;C:\Program Files\Java\jdk1.8.0_144\bin;C:\Program Files\PuTTY\;C:\Program Files (x86)\Inkscape\python\;C:\Program Files (x86)\Inkscape\python\DLLs;C:\Program Files (x86)\Java\jdk1.8.0_152\bin;C:\Program Files (x86)\Java\jdk1.8.0_162\bin;C:\Program Files (x86)\Java\jdk1.8.0_172\bin;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\PsychoPy3;C:\Program Files (x86)\PsychoPy3\DLLs;C:\Program Files\Git\cmd;C:\Program Files\Microsoft VS Code\bin;D:\Anaconda3;D:\Anaconda3\Library\mingw-w64\bin;D:\Anaconda3\Library\usr\bin;D:\Anaconda3\Library\bin;D:\Anaconda3\Scripts;C:\Program Files (x86)\Graphviz2.38\bin\;D:\olaf\Downloads\Programming\CrosslanguageRPCs\thrift;C:\Program Files\CMake\bin;D:\olaf\Downloads\Programming\Python\theano-deps\OpenBLAS\bin\;C:\Users\olaf\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\FileVerifier++\;C:\Users\olaf\AppData\Roaming\MiKTeX\2.9\miktex\bin\x64\;C:\tools\cmder;C:\Users\olaf\AppData\Local\atom\bin;C:\Users\olaf\AppData\Local\Microsoft\WindowsApps;C:\Users\olaf\AppData\Local\Pandoc\;C:\Users\olaf\AppData\Local\Keybase\

---

Contents of Pipfile ('D:\Example\Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
examplepackage = {editable = true, ref = "master", git = "ssh://${USER}@192.168.1.64:22/home/repos/ExamplePackage.git"}

[dev-packages]
pytest = "*"

[requires]
python_version = "3.7"

[techalchemy]

This currently only works in the sources section of your pipfile. The user field is auto-populated during ssh connections anyway, there is no need to specify it.

If you have a need to expand variables here, you will need to write an enhancement proposal and submit it as a PR against the peeps directory according to PEEP-001

See also: #3048, #3049

edit: or you can wait on #3111 which already contains a proposed PEEP on this topic.