Remove leftover pip-23.2.dist-info in site-packages

[ezio-melotti]

Description

While running a brand-new image, there are two .dist-info dirs for pip:

$ ls -l /opt/hostedtoolcache/Python/3.12.1/x64/lib/python3.12/site-packages
drwxrwxrwx+ 5 runner runneradmin 4096 Dec 17 22:19 pip
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.2.1.dist-info
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.3.2.dist-info

The first (pip-23.2.1.dist-info) belongs to the pip version that was initially installed, the second (pip-23.3.2.dist-info) to the updated version. Apparently this is because pip is updated using --ignore-installed, which leaves behind the old pip-23.2.1.dist-info.

Even though the latest version of pip is installed and used, the presence of the two dirs can create issues. For example, tools like safety detect the old version and report it since it has vulnerabilities, causing CI failures:

• Fix CI: ignore CVE-2023-5752 python/cherry-picker#102

Unless there is a valid reason to keep the old .dist-info around, I suggest removing the --ignore-installed flag, so that pip-23.2.1.dist-info is automatically removed during the pip update.

Click to see the full analysis of the issue

This initially came up because of a CI failure triggered by safety which detected an old version of pip, even though we were running the latest version. This lead to this issue:

• Fix CI: ignore CVE-2023-5752 python/cherry-picker#102

To debug the issue, I created the following test PR:

• Show pip version in the linter CI. ezio-melotti/cherry-picker#1

The output shows that the latest version of pip was installed from the beginning and correctly used by the other commands, but an ls shows 2 .dist-info dirs for pip.

To double-check, I created an empty workflow that only executes the ls, and the two .dist-info are still present:

• Check pip version on an empty runner ezio-melotti/cherry-picker#2

I looked at the code of this repo to see how Python and pip where installed, and apparently it happens in:

https://github.com/actions/runner-images/blob/266f9413d39fc77ade974757b633ef98873c9c21/images/ubuntu/scripts/build/Install-Toolset.ps1#L50C1-L51

This loop installs all the tools, including Python, from https://github.com/actions/python-versions

The code that actually installs Python and updates pip should be:

https://github.com/actions/python-versions/blob/af22c2b8e41acf6dc7c64030339622962820df9e/installers/nix-setup-template.sh#L51-L53

Here the --ignore-installed flag is used:

-I, --ignore-installed      Ignore the installed packages, overwriting them. This can break your system
                            if the existing package is of a different version or was installed with a
                            different package manager!

I'm not sure if/why this is needed, but I verified locally that this flag leaves around the old .dist-info. When --ignore-installed is not used, only a .dist-info dir is present after the upgrade:

$ python3 -m venv venv && source venv/bin/activate
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.2.dist-info
$ pip install --upgrade pip
...
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.3.2.dist-info
$ deactivate && rm -rf venv

When --ignore-installed is used, the old pip-23.2.dist-info dir is left behind after the upgrade:

$ python3 -m venv venv && source venv/bin/activate
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.2.dist-info
$ pip install --ignore-installed --upgrade pip
...
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.2.dist-info
pip-23.3.2.dist-info

Removing --ignore-installed from nix-setup-template.sh should therefore fix the issue, assuming it is not needed for other reasons.

Also note that the same flag is also used elsewhere, e.g. in install-pypy.sh.

If my analysis is correct, I can prepare a PR (or more) to remove the --ignore-installed flag.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

This was tested on the following image:

• Image: ubuntu-22.04
• Version: 20231217.2.0

It likely affects other (all?) images.

See e.g. https://github.com/ezio-melotti/cherry-picker/actions/runs/7386618932/job/20093604056?pr=2

Is it regression?

No

Expected behavior

There should be only one version of pip installed, and only one .dist-info dir that matches the installed version.

Actual behavior

There are two .dist-info dirs.

Repro steps

Run this workflow to check:

name: Check installed pip versions
on: [pull_request, push, workflow_dispatch]
jobs:
  check_pip:
    runs-on: ubuntu-latest
    steps:
      - run: ls -l /opt/hostedtoolcache/Python/3.12.1/x64/lib/python3.12/site-packages

It will output this:

drwxrwxrwx+ 5 runner runneradmin 4096 Dec 17 22:19 pip
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.2.1.dist-info
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.3.2.dist-info

[mikhailkoliada]

Transferring to setup-python as we do not maintain the hostedtoolcache archives, we only provide them as they are in the image

[HarithaVattikuti]

Hello, @ezio-melotti ! Thank you for reporting this issue, we will look into it :)

[konstin]

This bug causes problems with uv, because it expects (as python does too) that there is only one version per package (astral-sh/uv#1848 (comment))

#713 is a duplicate of this

[ezio-melotti]

It looks like @mayeut created and merged a PR that superseded the one I made, and this should have fixed the issue:

• fix: use --upgrade rather than --ignore-installed to upgrade pip python-versions#268
• Use --upgrade instead of --ignore-installed to upgrade pip. python-versions#254

There are a few releases in actions/python-versions dated after the merge so, unless there is something else missing, I believe this issue can be closed.

[aparnajyothi-y]

Hello @ezio-melotti, Closing this issue as this feature request is implemented in the merged PR.