Test suite compatibility with the latest virtualenv >= 20.0.0 #8441

frenzymadness · 2020-06-15T07:52:10Z

I'd like to continue with the effort from #7698 and #7718 and make the pip test suite use the latest released virtualenv.

This PR is WIP created to see the CI status.

frenzymadness · 2020-06-15T13:59:05Z

After hours of debugging, I've finaly found the root cause of Python 2.7 and test_build_env_isolation failure.

--------------------------------------------------------- Captured stdout call ---------------------------------------------------------
Script result: python /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/scratch/build_env.py /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/scratch/test.py
  return code: 1
-- stderr: --------------------
imported `pkg` from `/tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/venv/lib/python2.7/site-packages/pkg/__init__.pyc`
system sites:
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/venv/lib/python2.7/site-packages
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/venv/lib64/python2.7/site-packages
sys.path:
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/scratch
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/tmp/pip-build-env-TxuwZV/site
  /usr/lib/python27.zip
  /usr/lib64/python2.7
  /usr/lib64/python2.7/plat-linux2
  /usr/lib64/python2.7/lib-tk
  /usr/lib64/python2.7/lib-old
  /usr/lib64/python2.7/lib-dynload
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/tmp/pip-build-env-TxuwZV/overlay/lib/python2.7/site-packages
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/tmp/pip-build-env-TxuwZV/overlay/lib64/python2.7/site-packages
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/tmp/pip-build-env-TxuwZV/normal/lib/python2.7/site-packages
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/tmp/pip-build-env-TxuwZV/normal/lib64/python2.7/site-packages
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/venv/lib64/python2.7/site-packages
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/venv/lib/python2.7/site-packages
  /tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/scratch/pth_install
  /tmp/pytest-of-lbalhar/pytest-0/setuptools0/install
  /tmp/pytest-of-lbalhar/pytest-0/pip0/pip/src
  /usr/lib64/python2.7/site-packages
  /usr/lib64/python2.7/site-packages/gtk-2.0
  /usr/lib/python2.7/site-packages
Traceback (most recent call last):
  File "/tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/scratch/build_env.py", line 33, in <module>
    subprocess.check_call((sys.executable, sys.argv[1]))
  File "/usr/lib64/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/venv/bin/python', '/tmp/pytest-of-lbalhar/pytest-0/test_build_env_isolation0/workspace/scratch/test.py')' returned non-zero exit status 1

pkg is available but it should not be in the isolated build environment. The problem seems to be that the fixture virtualenv_template has

pip/tests/conftest.py

Lines 337 to 340 in 445711c

    
           # Enable user site packages. 
        
           venv.user_site_packages = True

which is then translated to include-system-site-packages = true in

pip/tests/lib/venv.py

Lines 134 to 146 in 201c9fc

    
           @user_site_packages.setter 
        
           def user_site_packages(self, value): 
        
               self._user_site_packages = value 
        
               self._customize_site() 
        
               pyvenv_cfg = self.location.joinpath("pyvenv.cfg") 
        
               modified_lines = [] 
        
               for line in pyvenv_cfg.read_text().splitlines(): 
        
                   k, v = line.split("=", 1) 
        
                   if k.strip() == "include-system-site-packages": 
        
                       line = "{}= {}".format(k, "true" if value else "false") 
        
                   modified_lines.append(line) 
        
               pyvenv_cfg.write_text("\n".join(modified_lines))

sys.path is adjusted in BuildEnvironment context manager (via sitecustomize.py) but the new virtualenv seems to load site Python module before it adds global packages to sys.path:

https://github.com/pypa/virtualenv/blob/f99353ca3d0ca9e23cfe4b66e54ba653bf99ab4a/src/virtualenv/create/via_global_ref/builtin/python2/site.py#L10-L21

I honestly don't know how to fix that because the whole tests is kinda complex/complicated and the chain of fixtures (test_build_env_isolation → script → virtualenv → virtualenv_factory → virtualenv_template) as well. Any help would be very appreciated. I'll investigate other failures in the meantime.

frenzymadness · 2020-06-16T08:14:21Z

Correction to my last comment about test_build_env_isolation: It's not caused by availability of system-site packages but by virtualenv which alters sys.path after sitecustomize.py is loaded. And because I think of it as about a bug, I've filled: pypa/virtualenv#1861

frenzymadness · 2020-06-16T09:20:48Z

The problem with these two (py38):

FAILED tests/functional/test_install.py::test_install_pep508_with_url - AssertionError: Script result: python -m pip install --no-ind...
FAILED tests/functional/test_install.py::test_install_pep508_with_url_in_install_requires

is that the virtual environment has access to system site-packages and packaging is already installed there.

frenzymadness · 2020-06-16T11:00:12Z

Well, it seems that we have some progress here 🎉

Is there anybody who can help me with the issues on Windows? I don't have windows so I cannot reproduce them.

I'll take a look on pypy.

uranusjr · 2020-06-16T13:57:25Z

Thanks for all the investigation effort, I incerely hope we’ll be able to come up with a solution soon 😃

The sys.path manipulation sounds hairy. Regardless of whether virtualenv doing it after *customize.py is correct or not, I have a feeling it does not really have an alternative due to how Python 2 startup functions, and we’ll likely need to find a workaround somehow.

pfmoore · 2020-06-16T14:43:48Z

Just a note, but if this is specific Python 2 hackery (which I'm not clear on, because this comment suggests there's something up on Python 3) I'd be fine with patching over the problem, as we'll be dropping Python 2 support fairly soon.

frenzymadness · 2020-06-16T15:06:40Z

Just a note, but if this is specific Python 2 hackery (which I'm not clear on, because this comment suggests there's something up on Python 3) I'd be fine with patching over the problem, as we'll be dropping Python 2 support fairly soon.

It's only about Python 2 and a hacky way test_build_env_isolation is implemented and based on virtualenv because virtualenv has to do some dark magic for Python 2 to guarantee that sys.path contains all paths from virtualenv. See the discussion in pypa/virtualenv#1861

If you are okay with that, I'd prefer to skip that test for Python 2 rather than investing energy to find a solution for soon-to-be-dropped Python 2.

pradyunsg · 2020-06-17T13:49:37Z

I'm 100% on board to just straight up skip the test, with a (long) comment describing why we're skipping it and why we're hitting this issue.

frenzymadness · 2020-06-25T11:46:27Z

I'll get back to it in ~2 weeks.

chrahunt · 2020-07-14T01:08:12Z

I don't have windows so I cannot reproduce them.

FWIW, Microsoft offers Windows VM images here that are good for like 90 days. That's what I use.

frenzymadness · 2020-07-28T11:09:16Z

PR is rebased on top of the latest master and I'll continue.

BrownTruck · 2020-08-17T21:45:11Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

virtualenv 20.0+ completely revamps its API: * There's no path_locations() anymore. Stdlib sysconfig can replace it completely. * The Python API to create an environment is split into several pieces. Use the entry point function for now to provide the functionality.

I *think* virtualenv fixed these? We'll see.

This is what the old virtualenv implementation did, so we match that.

The latest virtualenv does some dark magic for sys.path in Python 2 so the BuildEnvironment is no longer isolated here. See: pypa/virtualenv#1861

These tests should not have access to site packages so they always install the package into an empty environment.

frenzymadness · 2020-09-01T06:43:56Z

I'll try to move this forward. One question: If I understand it correctly, the new virtualenv uses the same configuration as the builtin venv module so the code for the legacy virtualenv can be deleted, right? For example, functions checking the availability of system site-packages _no_global_under_regular_virtualenv.

uranusjr · 2020-09-01T06:53:25Z

The main source src/pip still need to contain code to detect legacy virtualenvs (probably until the end of time), since not everyone has upgraded (will ever upgrade) to virtualenv 20.

frenzymadness · 2020-09-02T09:40:38Z

I'm afraid I won't be able to continue on this effort. The codebase is too complex for me and I don't have enough free time to dive deep enough to move pip to the new virtualenv while still support compatibility with venv and old virtualenv.

Is there anybody who can help me with this or take over it?

pradyunsg · 2020-09-02T11:52:20Z

Ahoy! What kind of help would you want with this? I'm happy to help. :)

frenzymadness · 2020-09-07T12:43:43Z

Ahoy! What kind of help would you want with this? I'm happy to help. :)

I'd like an insight into pip codebase because it seems that pip itself has some code to detect different kinds of virtual environments to be able to install packages to them and also its tests depend heavily on venv/virtualenv.

Are virtual environments in tests used to test how pip handles them or just to isolate one test from another? Do we want to test pip with venv, new virtualenv, and also the old virtualenv? It seems to me that the virtual environments created by venv are very similar to those produced by the new virtualenv so is there any reason to distinguish between them?

BrownTruck · 2021-02-04T16:15:13Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

pradyunsg · 2021-04-02T09:37:26Z

Looking at this now, this likely needs a lot of additional work.

I'll pick this up this weekend, and see where I can take it. :)

pradyunsg · 2021-04-02T09:39:09Z

I'll close this for now, since this one is significantly bitrotten; and filing a new PR is cheap. We can have a further discussion in #8273. ^>^

frenzymadness force-pushed the virtualenv-20 branch 2 times, most recently from cd18c2c to 329819d Compare June 16, 2020 10:20

frenzymadness mentioned this pull request Jun 17, 2020

Adapt virtualenv 20 API #7718

Closed

frenzymadness force-pushed the virtualenv-20 branch 4 times, most recently from a72e4c9 to a8a8fa8 Compare June 17, 2020 10:19

frenzymadness force-pushed the virtualenv-20 branch from a8a8fa8 to 3290da5 Compare July 28, 2020 11:08

pradyunsg changed the title ~~Compatibility with the latest virtualenv >= 20.0.0~~ Test suite compatibility with the latest virtualenv >= 20.0.0 Jul 28, 2020

BrownTruck added the needs rebase or merge PR has conflicts with current master label Aug 17, 2020

uranusjr and others added 8 commits September 1, 2020 08:18

Use virtualenv from Git for SCM compatibility

90e6bc6

Remove virtualenv fixup

9270814

I *think* virtualenv fixed these? We'll see.

Always use PEP 405 sitecustomize

9fe203b

Use public virtualenv CLI entry point

346f1b8

Better virtualenv introspection

c2825b8

Sync system site packages on user_site_packages

72d33a7

This is what the old virtualenv implementation did, so we match that.

Use latest released version of virtualenv instead master branch

7272fa2

frenzymadness added 4 commits September 1, 2020 08:20

Skip test_build_env_isolation on Python 2

02fb33f

The latest virtualenv does some dark magic for sys.path in Python 2 so the BuildEnvironment is no longer isolated here. See: pypa/virtualenv#1861

Fix test_install_pep508_with_url_*

2301e62

These tests should not have access to site packages so they always install the package into an empty environment.

Use regex for some tests to not depend on number of spaces

85e69b7

Newer virtualenv also for Azure pipelines

88ca32a

frenzymadness force-pushed the virtualenv-20 branch from 3290da5 to 6d69ce1 Compare September 1, 2020 06:30

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Sep 1, 2020

frenzymadness force-pushed the virtualenv-20 branch from 6d69ce1 to 58bb3e4 Compare September 1, 2020 06:49

frenzymadness added 2 commits September 1, 2020 09:22

Virtualenv no longer uses no-global-site-packages.txt

f7bb8a5

Update tests for virtualenv

78ef0f6

frenzymadness force-pushed the virtualenv-20 branch from 58bb3e4 to 78ef0f6 Compare September 1, 2020 07:22

BrownTruck added the needs rebase or merge PR has conflicts with current master label Feb 4, 2021

pradyunsg mentioned this pull request Apr 2, 2021

Tests are broken with virtualenv's rewrite (>=20.0.0) #8273

Open

pradyunsg closed this Apr 2, 2021

github-actions bot locked as resolved and limited conversation to collaborators Sep 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test suite compatibility with the latest virtualenv >= 20.0.0 #8441

Test suite compatibility with the latest virtualenv >= 20.0.0 #8441

frenzymadness commented Jun 15, 2020

frenzymadness commented Jun 15, 2020

frenzymadness commented Jun 16, 2020

frenzymadness commented Jun 16, 2020

frenzymadness commented Jun 16, 2020

uranusjr commented Jun 16, 2020

pfmoore commented Jun 16, 2020

frenzymadness commented Jun 16, 2020

pradyunsg commented Jun 17, 2020

frenzymadness commented Jun 25, 2020

chrahunt commented Jul 14, 2020 •

edited

Loading

frenzymadness commented Jul 28, 2020

BrownTruck commented Aug 17, 2020

frenzymadness commented Sep 1, 2020

uranusjr commented Sep 1, 2020

frenzymadness commented Sep 2, 2020

pradyunsg commented Sep 2, 2020

frenzymadness commented Sep 7, 2020

BrownTruck commented Feb 4, 2021

pradyunsg commented Apr 2, 2021

pradyunsg commented Apr 2, 2021 •

edited

Loading

Test suite compatibility with the latest virtualenv >= 20.0.0 #8441

Test suite compatibility with the latest virtualenv >= 20.0.0 #8441

Conversation

frenzymadness commented Jun 15, 2020

frenzymadness commented Jun 15, 2020

frenzymadness commented Jun 16, 2020

frenzymadness commented Jun 16, 2020

frenzymadness commented Jun 16, 2020

uranusjr commented Jun 16, 2020

pfmoore commented Jun 16, 2020

frenzymadness commented Jun 16, 2020

pradyunsg commented Jun 17, 2020

frenzymadness commented Jun 25, 2020

chrahunt commented Jul 14, 2020 • edited Loading

frenzymadness commented Jul 28, 2020

BrownTruck commented Aug 17, 2020

frenzymadness commented Sep 1, 2020

uranusjr commented Sep 1, 2020

frenzymadness commented Sep 2, 2020

pradyunsg commented Sep 2, 2020

frenzymadness commented Sep 7, 2020

BrownTruck commented Feb 4, 2021

pradyunsg commented Apr 2, 2021

pradyunsg commented Apr 2, 2021 • edited Loading

chrahunt commented Jul 14, 2020 •

edited

Loading

pradyunsg commented Apr 2, 2021 •

edited

Loading