Don't bail out the upgrade when RECORD is missing and the new version would be installed elsewhere

[hroncok]

What's the problem this feature will solve?

In Fedora, we package Python packages in RPMs without RECORD, with rpm in INSTALLER.

When users want to remove them with sudo pip, it errors:

$ cat /usr/lib/python3.11/site-packages/requests-2.28.1.dist-info/INSTALLER 
rpm

$cat /usr/lib/python3.11/site-packages/charset_normalizer-2.1.0.dist-info/INSTALLER 
rpm

# pip uninstall requests
Found existing installation: requests 2.28.1
ERROR: Cannot uninstall requests 2.28.1, RECORD file not found. Hint: The package was installed by rpm.

So far so good. Now consider the user wants to use sudo pip to upgrade requests.

(I'll actually downgrade in the examples because 2.28.1 is the latest version.)

# pip install --upgrade 'requests<2.28.1'
Collecting requests<2.28.1
  Downloading requests-2.28.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 2.6 MB/s eta 0:00:00
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.11/site-packages (from requests<2.28.1) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.11/site-packages (from requests<2.28.1) (1.26.12)
Collecting certifi>=2017.4.17
  Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 161.1/161.1 kB 5.2 MB/s eta 0:00:00
Installing collected packages: charset-normalizer, certifi, requests
  Attempting uninstall: charset-normalizer
    Found existing installation: charset-normalizer 2.1.0
ERROR: Cannot uninstall charset-normalizer 2.1.0, RECORD file not found. Hint: The package was installed by rpm.

It is good that pip refuses to uninstall charset-normalizer. However, if pip would simply skip the uninstallation in this case, it would happily install a different charset-normalizer version to a different prefix, because our installation prefix differs (we patch sysconfig to install to /usr/local).

Describe the solution you'd like

When pip:

• replaces one version of a package with another
• the to-be-uninstalled version is missing the RECORD file
• the to-be-installed version is going to be installed in another directory

I'd like pip to skip the uninstallation (possibly informing the user about doing so) and proceed with the installation.

Alternative Solutions

In Fedora, we currently patch pip to always skip the uninstallation on upgrades when the to-be-unisntalled packages are installed in the /usr prefix. This solves the problem but is utterly Fedora-specific and cannot be upstreamed. The problem is when users use pip to upgrade pip, they lose this patch.

Additional context

We try to educate our users not to use pip with sudo (under root), but they keep doing this. Despite all the dead kittens (a kitten is killed every time somebody runs pip as root), we try to make the experience as flawless as possible.

We started removing RECORDs and adding our custom INSTALLERs which made the system harder to break (users unable to sudo pip uninstall RPM packages) but arguably also harder to use (users unable to sudo pip upgrade RPM packages).

---

I am willing o have a look at this implementation-wise, but I've opened this issue to see if this would be acceptable.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Certainly. This is one of the pieces of PEP 668 that needs to be implemented, and has been on the radar for years: #4575