Update test suite to use virtualenv >= 20.0.0

[uranusjr]

Let’s restart #7718, we must do this before 3.11 is out.

This is mostly #8441 re-applied to main with some very minor changes.

[pradyunsg]

FWIW, I've been looking into this as well (https://github.com/pradyunsg/pip/tree/improve-test-isolation), although I'm trying there to move us off of virtualenv entirely and that involves getting the venv-based setup to be at-parity with the existing virtualenv-based setup (which it's not).

I've still not gotten all the tests to pass locally though, so I didn't file a PR for that yet. 😅

[uranusjr]

My current impression is if we manage to get new virtualenv to work, we should be very close with replacing it altogether with venv. The main blocker to virtualenv>=20 is how PEP 503 controls global site-packages access and sitecustomize differently, and if we manage to fix that, most of the venv incompatibilities automatically goes away.

[pradyunsg]

Yea, that's my understanding as well. I figured that it's not that big of a hop to try complete removal in the first shot -- the problems that need to be solved are roughly the same and it removed the conditionals for which "venv backend" we're using, which I was found were making it difficult for me to keep track of what's happening in the code.

The thing that I last struggled with is that the code was still picking up the sitecustomize from the base environment -- thanks to Homebrew having one -- which borked my Homebrew installation of Python and, I ended up stopping after fixing that mess for the third time and stopped at some point while trying to understand where it was being added in site.py and how to prevent that from happening again (i.e. not leaking global site-customise into the isolated test environment). 😅

Don't enable user site in template venv

IIRC, a bunch of tests rely on the virtualenv having user-site visible -- which isn't a problem since we have the isolate fixture which makes the user's home directory point to a temp directory, during the pip execution. I think I hit this as part of working on that branch as well. 😅

[pfmoore]

PEP 503

Typo? Sorry, I couldn't work out what the correct PEP number was.

[pradyunsg]

I'm gonna guess that was supposed to be https://peps.python.org/pep-0405/ (for pyvenv.cfg).

[sbidoul]

@uranusjr @pradyunsg By reading the above and looking at the number of failing tests it seems you have a good sense of what needs to be done and have made good progress.

With the RM hat on, I'm trying to get a feeling of when I'll push 22.2.1 out and if it should wait for this in order to have more assurance that everything works fine on Python 3.11.

So no pressure at all, of course, but if you have hints to share about your plans regarding this topic that'll help me.

[uranusjr]

Don’t let this block the release. This doesn’t need to be tied to one since the changes should be only in the test suite (aside from trivial renames).

[pradyunsg]

I missed this mention, but yea, this doesn’t block a bug fix. It’s more a logistical item needed for testing 3.11 and for reducing long term pain in this space.

[uranusjr]

The main changes are made to accomodate bad interactions between distutils and the new virtualenv hacks. Since old virtualenv only does not work in modern Python versions, I decided to only use new virtualenv with Python 3.10 and later and leave the old test setups as-is. A few tests also have the same distutils issues due to relying on setup.py install, and are rewritten to use wheels instead.

[uranusjr]

I think this actually works now; the only failure is the submodule one.

The modern virtual environment structure does not allow us to enable "fake user site" while disabling the global site, so we need to do more fine-grained configuration to correctly set up test environments for each test case.

With this done, we can also properly support the stdlib venv ad the test environment backend, since it basically works identically with modern virtualenv. The incompatible_with_test_venv marker is thus removed.