[24.1] `pip install py_find_1st` fails with 24.1, whereas 24.0 does not

[xmatthias]

Description

When installing a package - py_find_1st - the install is failing with pip 24.1 - but works with 24.0.

The error happens with

  File "/usr/lib/python3.12/tarfile.py", line 2725, in _find_link_target
    raise KeyError("linkname %r not found" % linkname)
KeyError: "linkname 'py_find_1st-1.1.6/README' not found"

it's unclear to me why this would work in 24.0 - but fail in 24.1.
The package uses a link from readme.md to readme - but i don't think that's disallowed?

~ tar tzvpf py_find_1st-1.1.6.tar.gz 
drwxr-xr-x roebel/staff      0 2023-10-28 18:01 py_find_1st-1.1.6/
-rw-r--r-- roebel/staff   7156 2023-10-28 17:15 py_find_1st-1.1.6/LONG_DESCR
-rw-r--r-- roebel/staff   8020 2023-10-28 18:01 py_find_1st-1.1.6/PKG-INFO
-rw-r--r-- roebel/staff   6896 2023-10-28 17:15 py_find_1st-1.1.6/README
hrw-r--r-- roebel/staff      0 2023-10-28 17:15 py_find_1st-1.1.6/README.md link to py_find_1st-1.1.6/README
-rw-r--r-- roebel/staff    159 2021-02-02 13:17 py_find_1st-1.1.6/pyproject.toml
-rw-r--r-- roebel/staff     34 2023-10-28 17:54 py_find_1st-1.1.6/requirements.txt
-rw-r--r-- roebel/staff     40 2019-06-28 23:32 py_find_1st-1.1.6/setup.cfg
-rw-r--r-- roebel/staff   6570 2023-10-28 17:54 py_find_1st-1.1.6/setup.py
drwxr-xr-x roebel/staff      0 2023-10-28 18:01 py_find_1st-1.1.6/test/
-rwxr-xr-x roebel/staff   1856 2023-10-28 17:10 py_find_1st-1.1.6/test/test_find_1st.py
drwxr-xr-x roebel/staff      0 2023-10-28 18:01 py_find_1st-1.1.6/utils_find_1st/
-rw-r--r-- roebel/staff    305 2023-10-28 18:01 py_find_1st-1.1.6/utils_find_1st/__init__.py
-rw-r--r-- roebel/staff   5475 2019-08-04 17:17 py_find_1st-1.1.6/utils_find_1st/find_1st.cpp

The changelog doesn't indicate something that would break like this ... unless i've missed that point ?

Expected behavior

Works on 24.0 and 24.1

pip version

24.1

Python version

3.12

OS

ubuntu

How to Reproduce

apt install build-essential
pip install pip==24.1
pip install py_find_1st
# Notice it failing
pip install pip==24.0
pip install py_find_1st

Output

...
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 185, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 231, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 303, in __init__
    super().__init__(
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 235, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 314, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 598, in _prepare_linked_requirement
    local_file = unpack_url(
                 ^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 180, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "/.venv/lib/python3.12/site-packages/pip/_internal/utils/unpacking.py", line 316, in unpack_file
    untar_file(filename, location)
  File "/.venv/lib/python3.12/site-packages/pip/_internal/utils/unpacking.py", line 235, in untar_file
    tar.extractall(location, filter=pip_filter)
  File "/usr/lib/python3.12/tarfile.py", line 2269, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.12/tarfile.py", line 2332, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.12/tarfile.py", line 2423, in _extract_member
    self.makelink(tarinfo, targetpath)
  File "/usr/lib/python3.12/tarfile.py", line 2521, in makelink
    self._extract_member(self._find_link_target(tarinfo),
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/tarfile.py", line 2725, in _find_link_target
    raise KeyError("linkname %r not found" % linkname)
KeyError: "linkname 'py_find_1st-1.1.6/README' not found"

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[ichard26]

We did update our tar unpacking logic to rely more on the stdlib implementation to benefit from tarfile data filters (#12214). I don't have access to a development environment so I won't be able to debug this until later today, but for now, you can downgrade pip to 24.0.

[notatallshaw]

FYI I can reproduce with python -m pip install --dry-run py_find_1st

And the exception in question hasn't recently changed: https://github.com/python/cpython/blame/v3.12.4/Lib/tarfile.py#L2725

And Python can extract the tar file:

$ wget https://files.pythonhosted.org/packages/1d/24/4b4dc8fcf9ad5e6ac8ac6c6bd4be8100f07846c7a7ce6aa9c44ee1eaa429/py_find_1st-1.1.6.tar.gz
$ python -m tarfile -e py_find_1st-1.1.6.tar.gz

So it's probably related to the filtering feature.

[henryiii]

FWIW, python -m tarfile --filter data -e py_find_1st-1.1.6.tar.gz seems fine too. Filtering added in #12214. Ahh, but it's wrapped with "pip specific" behavior. And I don't see a test to make sure good symlinks work, just tests for rejecting bad symlinks.

[notatallshaw]

I did a little investigation but didn't have time to post my results. The issue seems to be related to this tar file has a hard link that points from README.md to README.

I wasn't able to find another example of a real Python package that had the same problem.

It was very likely caused by #12214 if @encukou you want to take a look.

[shahidhafiz]

I had the same issue until I ran it using powershell as admin. worked first time.

[gneil90]

getting the same on osx

[apenney]

We use pandocfilters==1.4.1 and that exhibits the problem if you need a real world example of this breaking in the wild!

[pfmoore]

The pandocfilters 1.4.1 sdist seems to be broken:

Note that the target of README.rst is pandocfilters-1.4.1/README, which does not exist as the error message says.

The same is true of py_find_1st, which suggests that this might be a build backend bug creating an invalid sdist.

I don't think there's a pip issue here (even though things are reported as working in 24.0, I'd argue that's a bug in 24.0 rather than a regression).

Edit: Although on further reflection, this may depend on how a relative target for a hard link in a tar file is meant to be interpreted - as relative to the root of the tarfile, or as relative to the link. I don't know the answer to that, or even if there is a standard answer.

[pfmoore]

Ah. @henryiii pointed out here that we have a filter with pip-specific behaviour. I suspect the issue is that our filter is checking for symlinks pointing out of the tarfile, but it's using the same logic for hard links, and not taking into account the fact that symlinks are relative to the link itself, whereas hardlinks are relative to the tarfile root (as I noted above).

That sounds like a bit of a mess to sort out. @encukou as the author of that change, can you take a look?