feature(pex.tools): prefer a pex-file-provided package versions when --pip flag is also provided

[cognifloyd]

Given a reqs.pex file that includes a version of pip, pex-tools reqs.pex --pip should prefer the versions of packages included in the pex.

This may mean that pex has to downgrade the version of pip (and possibly setuptools, which pip depends on) to match whatever is included in the pex file.

Today, pex creates the virtualenv using the appropriate commands to include pip. Then it installs the requirements from the pex file, but any packages that already exist in the virtualenv, like pip, do not get downgraded to the version in the pex file. This feature request should reverse that.

pex-tools should report a warning about the collision between the pex file and the --pip flag (since the --pip flag is the only reason the virtualenv will have contents before installing things from the pex file). That warning should be silenced by --collisions-ok.

Background: pants

When running ./pants export ..., pants uses pex-tools venv --pip --collisions-ok for mutable virtualenvs.
https://github.com/pantsbuild/pants/blob/8fef19f9050d42c949edf781125cdebd6851fc57/src/python/pants/backend/python/goals/export.py#L239-L253
(that link has the following, which I'm including here since github doesn't include cross-repo snippets):

                    complete_pex_env.create_argv(
                        os.path.join(tmpdir_under_digest_root, pex_pex.exe),
                        *[
                            os.path.join(tmpdir_under_digest_root, requirements_pex.name),
                            "venv",
                            "--pip",
                            "--collisions-ok",
                            output_path,
                        ],
                    ),
                    {
                        **complete_pex_env.environment_dict(python=requirements_pex.python),
                        "PEX_MODULE": "pex.tools",
                    },
                ),

For StackStorm, we depend on pip+setuptools at runtime, and I want to make sure that the locked version is always used in the venv.

Slack

This request follows on a discussion in slack:

• slack question (linen mirror)
• slack conversation (linen mirror)