pip cannot uninstall an editable install in some cases

[NeilGirdhar]

Description

When a project that contains only pyproject.toml is installed in editable mode, pip cannot uninstall it.

pip version

21.3.1

Python version

3.9

OS

Ubuntu

How to Reproduce

$ git clone git@github.com:NeilGirdhar/tjax.git
Cloning into 'tjax'...
remote: Enumerating objects: 1819, done.
remote: Counting objects: 100% (223/223), done.
remote: Compressing objects: 100% (134/134), done.
remote: Total 1819 (delta 146), reused 134 (delta 89), pack-reused 1596
Receiving objects: 100% (1819/1819), 526.17 KiB | 3.96 MiB/s, done.
Resolving deltas: 100% (1288/1288), done.

$ pip install -e tjax
Obtaining file:///home/neil/tjax
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: networkx<3.0,>=2.4 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from tjax==0.14.2) (2.6.3)
Requirement already satisfied: yapf>=0.31 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from tjax==0.14.2) (0.31.0)
Requirement already satisfied: jax<0.3.0,>=0.2.21 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from tjax==0.14.2) (0.2.22)
Requirement already satisfied: optax<1,>=0 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from tjax==0.14.2) (0.0.9)
Requirement already satisfied: numpy>=1.21 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from tjax==0.14.2) (1.21.2)
Requirement already satisfied: matplotlib<4.0,>=3.3 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from tjax==0.14.2) (3.4.3)
Requirement already satisfied: colorful<0.6.0,>=0.5.4 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from tjax==0.14.2) (0.5.4)
Requirement already satisfied: absl-py in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from jax<0.3.0,>=0.2.21->tjax==0.14.2) (0.13.0)
Requirement already satisfied: opt_einsum in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from jax<0.3.0,>=0.2.21->tjax==0.14.2) (3.3.0)
Requirement already satisfied: scipy>=1.2.1 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from jax<0.3.0,>=0.2.21->tjax==0.14.2) (1.7.1)
Requirement already satisfied: pyparsing>=2.2.1 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from matplotlib<4.0,>=3.3->tjax==0.14.2) (2.4.7)
Requirement already satisfied: pillow>=6.2.0 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from matplotlib<4.0,>=3.3->tjax==0.14.2) (8.3.2)
Requirement already satisfied: python-dateutil>=2.7 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from matplotlib<4.0,>=3.3->tjax==0.14.2) (2.8.2)
Requirement already satisfied: kiwisolver>=1.0.1 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from matplotlib<4.0,>=3.3->tjax==0.14.2) (1.3.2)
Requirement already satisfied: cycler>=0.10 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from matplotlib<4.0,>=3.3->tjax==0.14.2) (0.10.0)
Requirement already satisfied: jaxlib>=0.1.37 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from optax<1,>=0->tjax==0.14.2) (0.1.71)
Requirement already satisfied: chex>=0.0.4 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from optax<1,>=0->tjax==0.14.2) (0.0.8)
Requirement already satisfied: six in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from absl-py->jax<0.3.0,>=0.2.21->tjax==0.14.2) (1.16.0)
Requirement already satisfied: dm-tree>=0.1.5 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from chex>=0.0.4->optax<1,>=0->tjax==0.14.2) (0.1.6)
Requirement already satisfied: toolz>=0.9.0 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from chex>=0.0.4->optax<1,>=0->tjax==0.14.2) (0.11.1)
Requirement already satisfied: flatbuffers<3.0,>=1.12 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from jaxlib>=0.1.37->optax<1,>=0->tjax==0.14.2) (2.0)
Installing collected packages: tjax
  Running setup.py develop for tjax
Successfully installed tjax

$ pip uninstall tjax
WARNING: Skipping tjax as it is not installed.

$ cat /home/neil/.pyenv/versions/3.9.6/lib/python3.9/site-packages/easy-install.pth
/home/neil/tjax

This uninstalls it:

$ cat /dev/null > /home/neil/.pyenv/versions/3.9.6/lib/python3.9/site-packages/easy-install.pth

Also, for some reason, the egg in ~/tjax is called "UNKNOWN.egg-info", but pyproject.toml clearly specifies the name "tjax".

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[saaketp]

Also, for some reason, the egg in ~/tjax is called "UNKNOWN.egg-info", but pyproject.toml clearly specifies the name "tjax".

This is the real problem, pip installs the package with the name UNKNOWN. So it will uninstall it with pip uninstall UNKNOWN not with pip uninstall tjax.

Not sure if this UNKNOWN name is a pip problem or poetry problem.
It is partially a pip problem I guess because it tells you Successfully installed tjax even though it actually installed UNKNOWN.

[pradyunsg]

/cc @sbidoul

Looks like pip is trying to install a pyproject.toml based project via setup.py develop, using our "shim" and setuptools is being setuptools.

[sbidoul]

What happens here, I think, is:

• pip enquires if the build backend (poetry) supports PEP 660: it does not (it does not, right?)
• consequently, pip tries to fallback on a setuptools based editable install, which it thinks it can do because there is a setup.cfg
• and setuptools does its thing even though it has no configuration

I don't think there is much we can do in pip.

I'd recommend two things:

• the tjax project to move its flake8 configuration from setup.cfg to .flake8
• reaching out to setuptools to ask if it could reject projects that don't even have a name configured

[NeilGirdhar]

@sbidout I have a name configured in my pyproject.toml

[sbidoul]

But setuptools does not read pyproject.toml. So until poetry supports PEP 660, you need to configure setuptools if you want editable installs. Le dim. 14 nov. 2021 à 15:24, Neil Girdhar ***@***.***> a écrit :

…

@sbidout I have a name configured in my pyproject.toml — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10659 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFI625XLZC54BLEX43MCLLUL7BARANCNFSM5H7XLITQ> .

[pradyunsg]

Poetry does support PEP 660.

[pradyunsg]

python-poetry/poetry-core#182

[sbidoul]

When I pip install poetry_core, I get version 1.0.7. Which, at first sight, does not have build_editable in poetry/core/masonry/api.py. Also I don't find 660 in the published poetry release notes. I've not much time to dig into this today, and I'm not familiar with the poetry release process but I suspect they have not released that yet.

[NeilGirdhar]

@sbidoul Good catch. So shall I wait for the poetry release and try again?

(Incidentally, which project is at fault given that poetry doesn't support editable installs?)

[sbidoul]

If you want editable install before poetry releases the only way is to configure setuptools in setup.cfg or setup.py.

As to what needs to change to avoid the surprising behavior you noticed, I think the only possible place is setuptools which should reject projects with no name (although they might have reasons to install UNKNOWN, I don't know).

[sbidoul]

And if you don't need editable installs now and can wait until the poetry release, I suggest you rename setup.cfg to .flake8 to avoid confusion. pip will then output a clear error message.

[NeilGirdhar]

Thanks. I'm just asking which project should be giving the error now though. If I don't change anything, I should still be getting an error, right? Where should that be coming from?

[sbidoul]

Thanks. I'm just asking which project should be giving the error now though. If I don't change anything, I should still be getting an error, right? Where should that be coming from?

@NeilGirdhar As I said above, setuptools, IMHO.

[NeilGirdhar]

Thanks, sorry I missed that 😊

[sbidoul]

Closing as we have not identified anything actionable for pip.

[jaraco]

• consequently, pip tries to fallback on a setuptools based editable install, which it thinks it can do because there is a setup.cfg

I do think there's something pip could do here. If the pyproject.toml specifies a build backend, it should not invoke setuptools. In fact I'd go further and say that pip should not fall back to building with Setuptools unless there's a setup.py file. It's true that Setuptools projects can be defined by setup.cfg only, but if that's what the user wants, they should specify setuptools as the backend.

Maybe that's not possible, given current assumptions (projects are allowed to specify one backend, but expect to use setuptools as a fallback for editable).

There's something else pip could do - if it's going to install a project as editable using setuptools, it should use the name that Setuptools resolves for the package, regardless what pip might expect by reading pyproject.toml.

[uranusjr]

if it's going to install a project as editable using setuptools, it should use the name that Setuptools resolves for the package

Setuptools handle all the installation when being used to install an editable, and the project name is resolved by only setuptools, pip does not do anything here.

I agree with your earlier point there’s something pip can do to make things more sensible though. It’s true that a project can be setup.cfg-only, but a setup.cfg-only project without an explicit backend declaration in pyproject.toml is arguably more likely to be a user error than intentionally configured.

[sbidoul]

It’s true that a project can be setup.cfg-only, but a setup.cfg-only project without an explicit backend declaration in pyproject.toml is arguably more likely to be a user error than intentionally configured.

This would not help with this specific issue, though, as the OP's example does have a build backend declared. So I'd argue this should be the topic of an other issue, if people want that.

Also noting that I'd rather be in favor of dropping support for setup.cfg-only projects than going further down that hole with more setuptools-specific code. It was a mistake to support that, IMO.

[NeilGirdhar]

I'm just curious, but were jaraco's suggestions unreasonable?

1. If the pyproject.toml specifies a build backend, it should not invoke setuptools.
2. pip should not fall back to building with Setuptools unless there's a setup.py file.

Am I wrong that either of those suggestions would have prevented my problem by causing an immediate error (since the backend I chose doesn't yet support editable installs)?

[sbidoul]

@NeilGirdhar both would be compatibility-breaking changes for pip, unfortunately.

[NeilGirdhar]

Would emitting a warning be compatibility-breaking?

[pradyunsg]

No, but it’ll be the first step toward changing our behaviour.

[sbidoul]

But what should we warn about exactly, and can we do it without building more setuptools knowledge into pip ?