Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pipenv 2022.8.5.13 introduced an issue with platformdirs #5254

Closed
OOPMan opened this issue Aug 14, 2022 · 17 comments · Fixed by #5255
Closed

pipenv 2022.8.5.13 introduced an issue with platformdirs #5254

OOPMan opened this issue Aug 14, 2022 · 17 comments · Fixed by #5255
Labels
PR: awaiting-review The PR related to this issue is awaiting review by a maintainer. Type: Bug 🐛 This issue is a bug. Type: Regression This issue is a regression of a previous behavior.

Comments

@OOPMan
Copy link

OOPMan commented Aug 14, 2022

Issue description

We have a job in our CI pipeline to uses pipenv to set up an environment with black and then run said tool. Our CI job for this in Gitlab-CI is extremely simple:

black_linting_check:
  stage: lint
  image: python:3.8
  before_script:
    - python -V
    - pip install pipenv
    - pipenv install --dev
  script:
    - pipenv run black derms derms_core mocks --check

Unfortunately it seems like the release of pipenv 2022.8.5.13 introduced some issue with the platformdirs package.

The following is a log of a CI successful run of our job from August 12th 2022:

Running with gitlab-runner 14.7.0 (98daeee0)
  on ip-17[2](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L2)-[3](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L3)1-5-168 qLr1uLbu
Resolving secrets
00:00
Preparing the "docker" executor
00:09
Using Docker executor with image python:3.8 ...
Authenticating with credentials from /root/.docker/config.json
Pulling docker image python:3.8 ...
Using docker image sha2[5](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L5)[6](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L6):be03ed85[7](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L7)d3127[8](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L8)de24758c22[9](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L9)9c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
Preparing environment
01:12
Running on runner-qlr1ulbu-project-22339857-concurrent-1 via ip-172-31-5-168...
Getting source from Git repository
00:38
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/opusonesolutions/gridos/derms/v3/derms-core/.git/
Created fresh repository.
Checking out 197c530c as DR-2998-deadband-settings-api...
Skipping Git submodules setup
Executing "step_script" stage of the job script
04:06
Using docker image sha256:be03ed857d31278de24758c2299c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
$ python -V
Python 3.8.13
$ pip install pipenv
Looking in indexes: https://pypi.org/simple, https://[MASKED]:****@opusonesolutions.jfrog.io/opusonesolutions/api/pypi/pypi-local/simple
Collecting pipenv
  Downloading pipenv-2022.8.5-py2.py3-none-any.whl (3.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 64.1 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.8/site-packages (from pipenv) (57.5.0)
Collecting certifi
  Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.2/160.2 KB 27.6 MB/s eta 0:00:00
Collecting virtualenv
  Downloading virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 63.1 MB/s eta 0:00:00
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Collecting distlib<1,>=0.3.5
  Downloading distlib-0.3.5-py2.py3-none-any.whl (466 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 467.0/467.0 KB 59.8 MB/s eta 0:00:00
Collecting platformdirs<3,>=2.4
  Downloading platformdirs-2.5.2-py3-none-any.whl ([14](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L14) kB)
Collecting filelock<4,>=3.4.1
  Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Installing collected packages: distlib, virtualenv-clone, platformdirs, filelock, certifi, virtualenv, pipenv
Successfully installed certifi-2022.6.[15](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L15) distlib-0.3.5 filelock-3.8.0 pipenv-2022.8.5 platformdirs-2.5.2 virtualenv-20.[16](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L16).3 virtualenv-clone-0.5.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
$ pipenv install --dev
Creating a virtualenv for this project...
Pipfile: /builds/opusonesolutions/gridos/derms/v3/derms-core/Pipfile
Using /usr/local/bin/python3.8 (3.8.13) to create virtualenv...
created virtual environment CPython3.8.13.final.0-64 in 6[19](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L19)ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/derms-core-Sx_JEpdN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==[22](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L22).2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/derms-core-Sx_JEpdN
Installing dependencies from Pipfile.lock ([29](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L29)83d4)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
$ pipenv run black derms derms_api mocks --check
Loading .env environment variables...
All done! ✨ 🍰 ✨
290 files would be left unchanged.
Cleaning up project directory and file based variables
00:01
Job succeeded

The following is a log of a CI failed run of our job from August 13th 2022:

Running with gitlab-runner 14.7.0 (98daeee0)
  on ip-17[2](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L2)-[3](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L3)1-5-63 oyxCD8yZ
Resolving secrets
00:00
Preparing the "docker" executor
01:2[5](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L5)
Using Docker executor with image python:3.8 ...
Authenticating with credentials from /root/.docker/config.json
Pulling docker image python:3.8 ...
Using docker image sha25[6](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L6):be03ed85[7](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L7)d3127[8](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L8)de24758c22[9](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L9)9c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
Preparing environment
00:03
Running on runner-oyxcd8yz-project-22339857-concurrent-1 via ip-172-31-5-63...
Getting source from Git repository
00:11
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/opusonesolutions/gridos/derms/v3/derms-core/.git/
Checking out ca798f52 as DR-2998-deadband-settings-api...
Skipping Git submodules setup
Executing "step_script" stage of the job script
02:49
Using docker image sha256:be03ed857d31278de24758c2299c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
$ python -V
Python 3.8.13
$ pip install pipenv
Looking in indexes: https://pypi.org/simple, https://[MASKED]:****@opusonesolutions.jfrog.io/opusonesolutions/api/pypi/pypi-local/simple
Collecting pipenv
  Downloading pipenv-2022.8.13-py2.py3-none-any.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 16.2 MB/s eta 0:00:00
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Collecting certifi
  Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.2/160.2 KB 2.4 MB/s eta 0:00:00
Collecting virtualenv
  Downloading virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB [12](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L12).9 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.8/site-packages (from pipenv) (57.5.0)
Collecting platformdirs<3,>=2.4
  Downloading platformdirs-2.5.2-py3-none-any.whl ([14](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L14) kB)
Collecting distlib<1,>=0.3.5
  Downloading distlib-0.3.5-py2.py3-none-any.whl (466 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 467.0/467.0 KB 3.7 MB/s eta 0:00:00
Collecting filelock<4,>=3.4.1
  Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Installing collected packages: distlib, virtualenv-clone, platformdirs, filelock, certifi, virtualenv, pipenv
Successfully installed certifi-2022.6.[15](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L15) distlib-0.3.5 filelock-3.8.0 pipenv-2022.8.13 platformdirs-2.5.2 virtualenv-20.[16](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L16).3 virtualenv-clone-0.5.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version [22](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L22).0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
$ pipenv install --dev
Creating a virtualenv for this project...
Pipfile: /builds/opusonesolutions/gridos/derms/v3/derms-core/Pipfile
Using /usr/local/bin/python3.8 (3.8.13) to create virtualenv...
created virtual environment CPython3.8.13.final.0-64 in 1679ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/derms-core-Sx_JEpdN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/derms-core-Sx_JEpdN
Installing dependencies from Pipfile.lock ([29](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L29)83d4)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
$ pipenv run black derms derms_api mocks --check
Loading .env environment variables...
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/derms-core-Sx_JEpdN/bin/black", line 5, in <module>
    from black import patched_main
  File "src/black/__init__.py", line 42, in <module>
  File "src/black/nodes.py", line [33](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L33), in <module>
  File "src/black/cache.py", line 9, in <module>
ModuleNotFoundError: No module named 'platformdirs'
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

I am pretty sure that this issue is caused by some change between pipenv 2022.8.5 and 2022.8.5.13 for the following reasons:

  1. I noticed this issue initially on a different MR in our Gitlab and so went back and forcibly re-ran the job show above on a different MR that had not changed between the two job runs show above. The result was this previously green job failing with no change other than the version of pipenv (Which, you will note from the CI job details was not version locked)
  2. Modifying our CI job such that the version of pipenv installed was locked to 2022.8.5 corrected the issue

Expected result

pipenv should install platformdirs correctly into the environment

Actual result

pipenv seems to fail to install platformdirs into the enviroment

Steps to replicate

This should be replicable using the steps in the CI job listed above along with any old random codebase

@matteius
Copy link
Member

This is an interesting one @OOPMan -- I was able to reproduce it, but when I passed the verbose argument, I saw in the output that platformdirs was already installed to my site-packages, and so it was ignoring installing it.

$ pipenv --no-site-packages install platformdirs -v
Installing platformdirs...
Installing package: platformdirs
Writing supplied requirement line to temporary file: 'platformdirs'
Installing 'platformdirs'
$ '\c\Users\matte\.virtualenvs\pipenv-5254-UiSfqJAX\Scripts\python.exe' 'C:\Users\matte\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\__pip-runner__.py' install --verbose --upgrade --exists-
action=i -r 'c:\users\matte\appdata\local\temp\pipenv-ofc42xsn-requirements\pipenv-_t1s_993-requirement.txt' -i https://pypi.org/simple
Using source directory: '\\c\\Users\\matte\\.virtualenvs\\pipenv-5254-UiSfqJAX\\src'
Adding platformdirs to Pipfile's [packages]...
Installation Succeeded
Installing dependencies from Pipfile.lock (d573fd)...
Writing supplied requirement line to temporary file: 'platformdirs==2.5.2 --hash=sha256:027d8e83a2d7de06bbac4e5ef7e023c02b863d7ea5d079477e722bb41ab25788 --hash=sha256:58c8abb07dcb441e6ee4b11d8df0ac856038f944ab9
8b7be6b27b2a3c7feef19'
Installing 'platformdirs'
$ '\c\Users\matte\.virtualenvs\pipenv-5254-UiSfqJAX\Scripts\python.exe' 'C:\Users\matte\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\__pip-runner__.py' install --verbose --upgrade --require
-hashes --no-deps --exists-action=i -r 'c:\users\matte\appdata\local\temp\pipenv-ofc42xsn-requirements\pipenv-y0ny1g83-requirement.txt' -i https://pypi.org/simple
Using source directory: '\\c\\Users\\matte\\.virtualenvs\\pipenv-5254-UiSfqJAX\\src'
Using pip 22.2.2 from C:\Users\matte\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip (python 3.10)
Requirement already satisfied: platformdirs==2.5.2 in c:\users\matte\appdata\roaming\python\python310\site-packages (from -r c:\users\matte\appdata\local\temp\pipenv-ofc42xsn-requirements\pipenv-y0ny1g83-requir
ement.txt (line 1)) (2.5.2)

The only way I was able to get it to work was by first uninstalling platformdirs using pip:

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pip uninstall platformdirs --yes
Found existing installation: platformdirs 2.5.2
Uninstalling platformdirs-2.5.2:
  Successfully uninstalled platformdirs-2.5.2

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pip uninstall platformdirs --yes
WARNING: Skipping platformdirs as it is not installed.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pipenv run python -c "import platformdirs"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'platformdirs'

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pipenv --no-site-packages install platformdirs -v
Installing platformdirs...
Installing package: platformdirs
Writing supplied requirement line to temporary file: 'platformdirs'
Installing 'platformdirs'
$ '\c\Users\matte\.virtualenvs\pipenv-5254-UiSfqJAX\Scripts\python.exe' 'C:\Users\matte\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\__pip-runner__.py' install --verbose --upgrade --exists-
action=i -r 'c:\users\matte\appdata\local\temp\pipenv-745_7y99-requirements\pipenv-kzsguwdl-requirement.txt' -i https://pypi.org/simple
Using source directory: '\\c\\Users\\matte\\.virtualenvs\\pipenv-5254-UiSfqJAX\\src'
Adding platformdirs to Pipfile's [packages]...
Installation Succeeded
Installing dependencies from Pipfile.lock (d573fd)...
  ================================ 0/0 - 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pipenv run python -c "import platformdirs"

I am not sure why it would be preferring what is in site-packages and not adding it the virtualenv. I looked over the changes from the two versions and it doesn't stand out to me yet what would be causing it.

@matteius matteius added Type: Bug 🐛 This issue is a bug. Type: Regression This issue is a regression of a previous behavior. labels Aug 14, 2022
@matteius
Copy link
Member

@OOPMan I think I have a fix that worked locally. Are you able to try your case using this branch: #5255

@matteius matteius added the PR: awaiting-review The PR related to this issue is awaiting review by a maintainer. label Aug 14, 2022
@OOPMan
Copy link
Author

OOPMan commented Aug 14, 2022

@matteius I will probably only be able to take a look at it tomorrow on my side, but will give it a go

@ajvogel
Copy link

ajvogel commented Aug 14, 2022

I'm experiencing the same issue on Ubuntu 22.04. It doesn't seem limited to platformdirs. I also notices it affects certifi etc. Downgraded to v2022.8.5 and can confirm the issue is not present there.

@lender-hunters
Copy link

We are experiencing the exact same issue with a different package. we have also retraced to the same root cause and came up with the mentioned mitigations described by all of the above

@matteius
Copy link
Member

@ajvogel @lender-hunters Please try the proposed fix branch and provide feedback on if it resolves the issue.

@lender-hunters
Copy link

@matteius it did not solve it for me unfortunately. haven't mentioned it yet but I actually use
pipenv install --deploy

@matteius
Copy link
Member

@lender-hunters Are you sure you tried the branch, and if so how did you install it? When you run pipenv --version what do you see? I am unable to reproduce this with my changes in the branch so far.

@matteius
Copy link
Member

I pushed a new commit to that branch to address the failing test.

@chruss2
Copy link

chruss2 commented Aug 15, 2022

The branch doesn't work for me either.

I want to point out that it appears to be finding the correct python (the python inside the venv) but the pip command is different.

Version 2022.8.14 (with or without your patch):
/Users/cheath/.local/share/virtualenvs/myproject-y3fxrpXI/bin/python /Users/cheath/.local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py install --verbose --upgrade --require-hashes --no-deps --exists-action=i -r /var/folders/fn/xlh8w1yx211gp06fhfrl5cm40000gp/T/pipenv-pn2c7vey-requirements/pipenv-0mik4v9c-requirement.txt -i https://pypi.org/simple

Version 2022.8.5:
/Users/cheath/.local/share/virtualenvs/myproject-y3fxrpXI/bin/python -m pip install --verbose --upgrade --require-hashes --no-deps --exists-action=i -r /var/folders/fn/xlh8w1yx211gp06fhfrl5cm40000gp/T/pipenv-jzs_6bee-requirements/pipenv-vg12o6io-requirement.txt -i https://pypi.org/simple

@matteius
Copy link
Member

@chrahunt The change to use the vendor'd pip was intentional -- I would think that since the python being invoked is from the virtualenv that it would prefer to install the dependencies to the virtualenv. For my testing I found the branch to resolve the issue, as I have certifi installed in my site-packages as well as platformdirs, and pipenv created my virtualenv with the requested dependencies:

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pip freeze | grep platformdirs
platformdirs==2.5.2

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pip freeze | grep certifi
certifi==2022.6.15

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pip freeze | grep requests
requests==2.28.1
requests-toolbelt==0.9.1

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pipenv --rm
Removing virtualenv (\c\Users\matte\.virtualenvs\pipenv-5254-UiSfqJAX)...

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pipenv install --deploy
Creating a virtualenv for this project...
Pipfile: C:\Users\matte\Projects\pipenv-triage\pipenv-5254\Pipfile
Using C:/Program Files/WindowsApps/PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0/python3.10.exe (3.10.5) to create virtualenv...
[  ==] Creating virtual environment...created virtual environment CPython3.10.5.final.0-64 in 2371ms
  creator CPython3Windows(dest=C:\c\users\matte\.virtualenvs\pipenv-5254-UiSfqJAX, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\matte\AppData\Local\pypa\virtualenv)
    added seed packages: pip==22.2.1, setuptools==63.2.0, wheel==0.37.1
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: \c\Users\matte\.virtualenvs\pipenv-5254-UiSfqJAX
Installing dependencies from Pipfile.lock (12e6f7)...
  ================================ 6/6 - 00:00:02

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pipenv run pip freeze
certifi==2022.6.15
charset-normalizer==2.1.0
idna==3.3
platformdirs==2.5.2
requests==2.28.1
urllib3==1.26.11

But then again, I am getting that behavior on the main branch now too :-/

@matteius
Copy link
Member

@chruss2 I just pushed a change to that branch to --ignore-installed on the pip install command -- I am wondering if this helps resolve the issue or if there is still a problem where sometimes it is trying to install to a location outside the virtualenv?

@matteius
Copy link
Member

Ok if I install pipenv from pip then I can reproduce the issue: pip install pipenv
However if I checkout the main branch and install pipenv then I am unable to reproduce the issue: python -m pip install -e . --upgrade

So beyond noting the difference in behavior is really weird, my question now is for those who have tried my branch, how did you install it? I'd like to try installing it that way so I can think more about what the difference could be.

@matteius
Copy link
Member

matteius commented Aug 15, 2022

Ok I found I could reproduce it with main by installing via: pip install git+https://github.com/pypa/pipenv.git@main
Then I removed pipenv and installed again with the branch pip install git+https://github.com/pypa/pipenv.git@issue-5254
and it appears that --ignore-installed resolves the issue, and doesn't require repeat installation upon subsequent runs of pipenv sync.

Checking my prior commit to be sure: pip install git+https://github.com/pypa/pipenv.git@4b5945ea87c76b3ac6345f20a9d0533507a11a42 It did in fact not solve the issue.

I believe now though the latest version of the branch is solving the issue at this time. If I could get confirmation from someone else that you agree, I will feel good about getting this patched up in a new release.

@chruss2
Copy link

chruss2 commented Aug 15, 2022

Yes, the latest commit works now. 👍

BTW, I was curious why this was happening, so I looked into how pip is vendored. The vendoring is adding line 29 to main.py:
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__)))))

This line is needed in order to import pipenv.patched.pip._internal.cli.main, but also affects how packages are installed.
That line is not reached if you call pip with -m pip, but is reached now that you're calling __pip-runner__.py directly, which was a change in 2022.8.13.

I figured I'd mention this, since changing sys.path may affect pip in other areas of the code too.

@matteius
Copy link
Member

@chruss2 This is really interesting what you point out, I have wondered about that post pip import patch before and tried removing it without success. I do wonder if its necessary that it insert the path to be first, but probably that is safer to ensure that pipenv imported is the one that is being invoked. The tests do pass however if I modify the line to append (instead of insert to front) the path instead -- I am not sure if that would change the outcome of any of this original issue though.

Just want to call out for this pain point/bump in the road, I had several motivations for making pipenv rely exclusively on the vendor'd pip:

  • We have patched it for index restricted packages, but we only used it primarily during the resolution phase.
  • Many issue reports have been expressed that have come down to conflicts between the pip version the user has installed and what pipenv is expecting.
  • So more confidence in pipenv using the expected version of pip, and what the expected outcomes are.
  • Not requiring pip be installed to use pipenv.
  • It was always confusing to not be able to fully hit break-points in the vendor'ed pip when debugging issues, some things used vendor'd pip and some things used user or system pip -- it was kind of maddening.
  • Potentially other more minor reasons I may have forgotten about.

That being said I wanted to head off any issues as soon as possible, because it was hard to be certain what kinds of side effects would be brought about. That is why I opted for a Saturday release and to be available this weekend for fixing any issues not caught by the test runners.

@matteius
Copy link
Member

pipenv==2022.8.15 has been released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR: awaiting-review The PR related to this issue is awaiting review by a maintainer. Type: Bug 🐛 This issue is a bug. Type: Regression This issue is a regression of a previous behavior.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants