Unable to create a new pipenv powered project on Windows

[nickperkins]

Issue description

When trying to set up a new project in pipenv, it is unable to find the scripts dir from the virtual environment

Expected result

Create new virtual environment and create Pipfile for project.

Actual result

$ pipenv install boto3
Creating a virtualenv for this project…
Pipfile: E:\ScriptDir\Generic\test\Pipfile
Using C:/Users/nick.perkins/AppData/Local/Programs/Python/Python37/python.exe (3.7.4) to create virtualenv…
[    ] Creating virtual environment...Already using interpreter C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37\python.exe
Using base prefix 'C:\\Users\\nick.perkins\\AppData\\Local\\Programs\\Python\\Python37'
New python executable in C:\Users\nick.perkins\.virtualenvs\test-9QO8X7jI\Scripts\python.exe
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter C:/Users/nick.perkins/AppData/Local/Programs/Python/Python37/python.exe

Successfully created virtual environment!
Virtualenv location: C:\Users\nick.perkins\.virtualenvs\test-9QO8X7jI
Creating a Pipfile for this project…
Traceback (most recent call last):
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37\Scripts\pipenv.exe\__main__.py", line 9, in <module>
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\cli\command.py", line 252, in install
    site_packages=state.site_packages
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 1929, in do_install
    site_packages=site_packages,
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 615, in ensure_project
    validate=validate, skip_requirements=skip_requirements, system=system
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 320, in ensure_pipfile
    project.create_pipfile(python=python)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\project.py", line 753, in create_pipfile
    required_python = self.which("python", self.virtualenv_location)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\project.py", line 1126, in which
    result = next(iter(filter(None, (find(finder) for finder in self.finders))), None)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\project.py", line 1126, in <genexpr>
    result = next(iter(filter(None, (find(finder) for finder in self.finders))), None)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 134, in which
    return self.system_path.which(exe)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 483, in which
    return next(iter(f for f in filtered if f is not None), None)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 483, in <genexpr>
    return next(iter(f for f in filtered if f is not None), None)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 482, in <genexpr>
    filtered = (sub_which(self.get_path(k)) for k in self.path_order)
  File "c:\users\nick.perkins\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 438, in get_path
    raise ValueError("Path not found or generated: {0!r}".format(path))
ValueError: Path not found or generated: WindowsPath('c:/users/nick.perkins/.virtualenvs/test-9qo8x7ji/scripts')

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

pipenv install boto3

No Pipfile was createdpipenv

---

$ pipenv --support

Pipenv version: '2020.6.2'

Pipenv location: 'c:\\users\\nick.perkins\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\pipenv'

Python location: 'c:\\users\\nick.perkins\\appdata\\local\\programs\\python\\python37\\python.exe'

Python installations found:

• 3.7.4: C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37\python.exe
• 3.7.2: C:\Users\nick.perkins\.pyenv\pyenv-win\versions\3.7.2\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.4',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.14393',
 'python_full_version': '3.7.4',
 'python_version': '3.7',
 'sys_platform': 'win32'}

System environment variables:

• ACLOCAL_PATH
• ALLUSERSPROFILE
• APPDATA
• AWS_PROFILE
• CLIENTNAME
• COMMONPROGRAMFILES
• COMPUTERNAME
• COMSPEC
• CONFIG_SITE
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• DISPLAY
• EXEPATH
• HOME
• HOMEDRIVE
• HOMEPATH
• HOSTNAME
• INFOPATH
• LANG
• LOCALAPPDATA
• LOGONSERVER
• MANPATH
• MINGW_CHOST
• MINGW_PACKAGE_PREFIX
• MINGW_PREFIX
• MSYSTEM
• MSYSTEM_CARCH
• MSYSTEM_CHOST
• MSYSTEM_PREFIX
• NUMBER_OF_PROCESSORS
• OLDPWD
• ORIGINAL_PATH
• ORIGINAL_TEMP
• ORIGINAL_TMP
• OS
• PATH
• PATHEXT
• PKG_CONFIG_PATH
• PLINK_PROTOCOL
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMFILES
• PS1
• PSMODULEPATH
• PUBLIC
• PWD
• PYENV
• PROGRAMDATA
• PROGRAMFILES(X86)
• PROGRAMW6432
• SESSIONNAME
• SHELL
• SHLVL
• SSH_ASKPASS
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TERM
• TMP
• TMPDIR
• USERDNSDOMAIN
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• WINDIR
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

PipenvÔÇôspecific environment variables:

DebugÔÇôspecific environment variables:

• PATH: C:\Users\nick.perkins\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\nick.perkins\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files\Amazon\cfn-bootstrap;C:\Program Files\Git\cmd;C:\Program Files\7-Zip;C:\Program Files\TortoiseGit\bin;C:\Program Files\Amazon\AWSCLIV2;C:\Users\nick.perkins\.pyenv\pyenv-win\bin;C:\Users\nick.perkins\.pyenv\pyenv-win\shims;C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37\Scripts;C:\Users\nick.perkins\AppData\Local\Programs\Python\Python37;C:\Users\nick.perkins\AppData\Local\Microsoft\WindowsApps;C:\Users\nick.perkins\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
• SHELL: C:\Program Files\Git\usr\bin\bash.exe
• LANG: en_US.UTF-8
• PWD: E:/ScriptDir/Generic/test

---

[max-niederman]

+1 had this same problem

[rfkortekaas]

I've investigated this issue and it is introduced in the normalize_path function. At some point the path_order list in the pythonfinder path model contains the paths with normalized case while the get_path function gets a path without normalized case and the path cannot be found in the defaultdict due to case differences. It looks like this only happens on windows when pyenv is also installed.

The normalize_path function was already on the nomination to be refactored in vistir.

[crmne]

+1

[JonasR]

Ran into the same issue today

[dennisroche]

I've investigated this issue and it is introduced in the normalize_path function. At some point the path_order list in the pythonfinder path model contains the paths with normalized case while the get_path function gets a path without normalized case and the path cannot be found in the defaultdict due to case differences. It looks like this only happens on windows when pyenv is also installed.

The normalize_path function was already on the nomination to be refactored in vistir.

is there a known workaround to this?

UPDATE: roll back to 2018.11.26 release (https://pypi.org/project/pipenv/2018.11.26/)

[ghanshyam30]

I've investigated this issue and it is introduced in the normalize_path function. At some point the path_order list in the pythonfinder path model contains the paths with normalized case while the get_path function gets a path without normalized case and the path cannot be found in the defaultdict due to case differences. It looks like this only happens on windows when pyenv is also installed.
The normalize_path function was already on the nomination to be refactored in vistir.

is there a known workaround to this?

UPDATE: roll back to 2018.11.26 release (https://pypi.org/project/pipenv/2018.11.26/)

This resolved my problem, thanks a bunch!

[nickperkins]

It looks like this only happens on windows when pyenv is also installed.

I don't have pyenv installed

[nickperkins]

I've found that if I add a Pipfile to the target folder and then run pip install I can get a new pipenv working

[prataplyf]

update your virtualenv pipenv
pip install pipenv==2018.11.26
this works for me

[nickperkins]

That's great but that's not an upgrade

…

On Fri, 31 Jul 2020, 5:30 pm Ashish Pratap Singh, ***@***.***> wrote: update your virtualenv pipenv pip install pipenv==2018.11.26 this works for me — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4330 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAELERBVBIQM6FQXXXXG6HLR6JXJRANCNFSM4NYCKWIA> .

[prataplyf]

That's great but that's not an upgrade
…
On Fri, 31 Jul 2020, 5:30 pm Ashish Pratap Singh, @.***> wrote: update your virtualenv pipenv pip install pipenv==2018.11.26 this works for me — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4330 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAELERBVBIQM6FQXXXXG6HLR6JXJRANCNFSM4NYCKWIA .

that's true.

might be some of your modules that are not updated and that not support the updated pipenv.

so when you update all modules the updated pipenv will work for you.

[nickperkins]

Just to make it clear to everyone,

pip install pipenv==2018.11.26 installs an older version of pipenv. It is NOT an upgrade.

[darnley]

Same issue here.

[davem99]

Same issue here, can't actually use pipenv.
Going back to the 2018.11.26 version does allow me to pip install inside my project folder.

[meichthys]

@nickperkins

It looks like there was some kind of regression after 2018.11.26
As @rfkortekaas mentioned, seems to be an issue of normalized case in the path. Pipenv is trying to search for a python executable path, but on windows it is trying to find a as_posix path in self.path_order which has paths formatted with double \\ instead of single / (as changed by as_posix)

The relevant pipenv code seems to be in: pipenv\vendor\pythonfinder\models\path.py (Line 432).
I can get the latest version of pyenv to work if i change Line 432 to the following:
if not _path and os.path.normcase(path.as_posix()) in self.path_order:

I'm not sure anything else would need to be done to make this more of a robust workaround, but it should work cross-platform since os.path.normcase does nothing on other platforms.

[meichthys]

@ken-reitz @techalchemy
I've submitted a fix for this upstream. Could we get this pulled in when/if the upstream pull request is accepted?

[frostming]

@meichthys Can you figure out where the back-slashed paths are introduced since I searched all the path_order assignments and all are normalized by as_posix().

[frostming]

Given the complicated and varying env setup, a minimal reproducible example is of great help.

[rfkortekaas]

@frostming some path entries are setup with normalize_path from pipenv\pythonfinder\utils.py. This function uses normcase and normpath. On windows this changes forward slashes to backward slashes. Where as_posix returns the path with forward slashes. It took me a long time the track down the issue and I ended up with removing normcase from normalize_path what basically results in the same as the solution from @meichthys.

The issue only occurs when one of the _setup_* functions are called. So it looks like the path_order with the wrong formatting is coming form the new_instances created in the _setup_* functions.

[frostming]

@rfkortekaas Thanks, that is enlightening. I have addressed it in the upstream. sarugaku/pythonfinder#89

[rfkortekaas]

@frostming Thanks. I've tested your solution in the PR and it looks like everything is working as it should!

[meichthys]

Also tested and working! 🚀

[Timtam]

Sorry to bring this up again, but I'm running into the same issues right now.

pipenv install black isort flake8 mypy --dev

Creating a virtualenv for this project…

Pipfile: F:\Privat\Skripte\pyAGE\Pipfile

Using default python from c:\users\toni\.local\pipx\venvs\pipenv\scripts\python.exe (3.8.2) to create virtualenv…


[    ] Creating virtual environment...�
[=   ] Creating virtual environment...�
[==  ] Creating virtual environment...�
[=== ] Creating virtual environment...�
[ ===] Creating virtual environment...�
[  ==] Creating virtual environment...�
[   =] Creating virtual environment...created virtual environment CPython3.8.2.final.0-64 in 427ms
  creator CPython3Windows(dest=C:\Users\Toni\.virtualenvs\pyAGE-Qx2pNdmj, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\Toni\AppData\Local\pypa\virtualenv)
    added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

�Successfully created virtual environment! 
Virtualenv location: C:\Users\Toni\.virtualenvs\pyAGE-Qx2pNdmj
Creating a Pipfile for this project…
Traceback (most recent call last):
  File "c:\users\toni\.pyenv\pyenv-win\versions\3.8.2\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\toni\.pyenv\pyenv-win\versions\3.8.2\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Toni\.local\bin\pipenv.exe\__main__.py", line 7, in <module>
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\cli\command.py", line 232, in install
    retcode = do_install(
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\core.py", line 1920, in do_install
    ensure_project(
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\core.py", line 614, in ensure_project
    ensure_pipfile(
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\core.py", line 320, in ensure_pipfile
    project.create_pipfile(python=python)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\project.py", line 755, in create_pipfile
    required_python = self.which("python", self.virtualenv_location)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\project.py", line 1128, in which
    result = next(iter(filter(None, (find(finder) for finder in self.finders))), None)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\project.py", line 1128, in <genexpr>
    result = next(iter(filter(None, (find(finder) for finder in self.finders))), None)
  File "c:\users\toni\.local\pipx\venvs\pipenv\lib\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 134, in which
    return self.system_path.which(exe)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 483, in which
    return next(iter(f for f in filtered if f is not None), None)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 483, in <genexpr>
    return next(iter(f for f in filtered if f is not None), None)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 482, in <genexpr>
    filtered = (sub_which(self.get_path(k)) for k in self.path_order)
  File "C:\Users\Toni\.local\pipx\venvs\pipenv\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 438, in get_path
    raise ValueError("Path not found or generated: {0!r}".format(path))
ValueError: Path not found or generated: WindowsPath('c:/users/toni/.virtualenvs/pyage-qx2pndmj/scripts')

Here is the installed pipenv version:

pipx list

venvs are in C:\Users\Toni\.local\pipx\venvs
apps are exposed on your $PATH at C:\Users\Toni\.local\bin
   package pipenv 2020.8.13, Python 3.8.2
    - pipenv-resolver.exe
    - pipenv.exe

Is there no way to solve this except going back to a two-year old version by now?
Thanks.

[rfkortekaas]

@Timtam this has been fixed in the upstream version which haven't been released yet.
You can install the latest version from git:
pip install -e git+https://github.com/pypa/pipenv.git

Or patch your local installation with the fixes in the pull request.

@frostming any idea when a new version will be released?

[sandyscott]

I needed to specify the egg name to pull the latest version
pip install -e git+https://github.com/pypa/pipenv.git#egg=pipenv