Regression: ${PROJECT_ROOT} for relative paths to local packages not working in pdm > 2.2.1 #1658

dairiki · 2023-02-01T19:30:40Z

I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

Start with minimal pyproject.toml:

[tool.pdm]
[project]
requires-python = ">3.7"

Add local package dependency (mypkg/ contains a minimal setup.py-based package).

pdm add ./mypkg

Using pdm 2.2.1, the resulting pyproject.toml is:

[tool.pdm]
[project]
requires-python = ">=3.7"
dependencies = [
    "my-test-package @ file:///${PROJECT_ROOT}/mypkg",
]
dynamic = ["dependencies"]

Upgrade pdm to anything > 2.2.1 and attempt to use that pyproject.toml.

Actual behavior

Attempts to use that pyproject.toml with any later version of pdm (I've tried 2.4.2 and 2.3.0) fail (with FileNotFoundError: [Errno 2] No such file or directory: '/${PROJECT_ROOT}/mypkg.

E.g.

$ pdm --version
PDM, version 2.4.2

$ pdm lock -v
STATUS: Resolving dependencies
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   my-test-package @ file:///${PROJECT_ROOT}/mypkg
pdm.termui:   python>=3.7
pdm.termui:   Adding requirement my-test-package @ file:///${PROJECT_ROOT}/mypkg
pdm.termui:   Adding requirement python>=3.7
pdm.termui: ======== Starting round 0 ========
STATUS: Resolving: new pin python>=3.7
pdm.termui: Pinning: python None
pdm.termui: ======== Ending round 0 ========
pdm.termui: ======== Starting round 1 ========
Traceback (most recent call last):
  File ".../.local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/core.py", line 259, in main
    return Core().main(args)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/core.py", line 191, in main
    raise cast(Exception, err).with_traceback(traceback)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/core.py", line 186, in main
    f(project, options)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/cli/commands/lock.py", line 53, in handle
    actions.do_lock(
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/cli/actions.py", line 95, in do_lock
    mapping, dependencies = resolve(
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/resolver/core.py", line 35, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 521, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 402, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 238, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 228, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/resolver/providers.py", line 189, in get_dependencies
    deps, requires_python, _ = self.repository.get_dependencies(candidate)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/models/repositories.py", line 78, in get_dependencies
    requirements, requires_python, summary = getter(candidate)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/models/repositories.py", line 35, in wrapper
    result = func(self, candidate)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/models/repositories.py", line 228, in _get_dependencies_from_metadata
    deps = prepared.get_dependencies_from_metadata()
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/models/candidates.py", line 539, in get_dependencies_from_metadata
    self.req.project_name, self.metadata.requires or [], extras  # type: ignore
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/models/candidates.py", line 523, in metadata
    result = self.prepare_metadata()
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/models/candidates.py", line 442, in prepare_metadata
    self.obtain(allow_all=True)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/pdm/models/candidates.py", line 432, in obtain
    result = finder.download_and_unpack(
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/unearth/finder.py", line 358, in download_and_unpack
    file = unpack_link(
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/unearth/preparer.py", line 295, in unpack_link
    validator.validate_path(artifact)
  File ".../.local/share/pdm/venv/lib/python3.8/site-packages/unearth/preparer.py", line 116, in validate_path
    with path.open("rb") as f:
  File "/usr/lib/python3.8/pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/lib/python3.8/pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/${PROJECT_ROOT}/mypkg'

Expected behavior

Continuing with pdm 2.2.1, things appear to work correctly, e.g.

$ pdm --version
PDM, version 2.2.1
$ pdm lock -v
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   my-test-package @ file:///${PROJECT_ROOT}/mypkg
pdm.termui:   python>=3.7
pdm.termui:   Adding requirement my-test-package @ file:///${PROJECT_ROOT}/mypkg
pdm.termui:   Adding requirement python>=3.7
pdm.termui: ======== Starting round 0 ========
pdm.termui: Pinning: python None
pdm.termui: ======== Ending round 0 ========
pdm.termui: ======== Starting round 1 ========
pdm.termui: Running PEP 517 backend to get metadata for <Link file:///[...]/mypkg (from None)>
pdm.termui: Preparing isolated env for PEP 517 build...
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   wheel
pdm.termui:   setuptools>=40.8.0
pdm.termui:   python>=3.10.9,<3.10.10
pdm.termui:   Adding requirement wheel
pdm.termui:   Adding requirement setuptools>=40.8.0
pdm.termui:   Adding requirement python>=3.10.9,<3.10.10
pdm.termui: ======== Starting round 0 ========
pdm.termui: Pinning: python None
pdm.termui: ======== Ending round 0 ========
pdm.termui: ======== Starting round 1 ========
pdm.termui: Pinning: setuptools 67.1.0
pdm.termui: ======== Ending round 1 ========
pdm.termui: ======== Starting round 2 ========
pdm.termui: Pinning: wheel 0.38.4
pdm.termui: ======== Ending round 2 ========
pdm.termui: ======== Starting round 3 ========
pdm.termui: ======== Resolution Result ========
pdm.termui: Stable pins:
pdm.termui:       python None
pdm.termui:   setuptools 67.1.0
pdm.termui:        wheel 0.38.4
pdm.termui: Installing setuptools 67.1.0
unearth.preparer: Using cached <Link https://files.pythonhosted.org/packages/c8/24/157fed9a57108b3e1b6ccd86e95823327f231ef12464b0eca13650eb33ac/setuptools-67.1.0-py3-none-any.whl (from None)>
pdm.termui: Installing wheel 0.38.4
unearth.preparer: Using cached <Link https://files.pythonhosted.org/packages/bd/7c/d38a0b30ce22fc26ed7dbc087c6d00851fb3395e9d0dac40bec1f905030c/wheel-0.38.4-py3-none-any.whl (from None)>
pdm.termui: running egg_info
pdm.termui: writing my_test_package.egg-info/PKG-INFO
pdm.termui: writing dependency_links to my_test_package.egg-info/dependency_links.txt
pdm.termui: writing top-level names to my_test_package.egg-info/top_level.txt
pdm.termui: reading manifest file 'my_test_package.egg-info/SOURCES.txt'
pdm.termui: writing manifest file 'my_test_package.egg-info/SOURCES.txt'
pdm.termui: running dist_info
pdm.termui: creating /tmp/pdm-meta-uu6utr5w/my_test_package.egg-info
pdm.termui: writing /tmp/pdm-meta-uu6utr5w/my_test_package.egg-info/PKG-INFO
pdm.termui: writing dependency_links to /tmp/pdm-meta-uu6utr5w/my_test_package.egg-info/dependency_links.txt
pdm.termui: writing top-level names to /tmp/pdm-meta-uu6utr5w/my_test_package.egg-info/top_level.txt
pdm.termui: writing manifest file '/tmp/pdm-meta-uu6utr5w/my_test_package.egg-info/SOURCES.txt'
pdm.termui: reading manifest file '/tmp/pdm-meta-uu6utr5w/my_test_package.egg-info/SOURCES.txt'
pdm.termui: writing manifest file '/tmp/pdm-meta-uu6utr5w/my_test_package.egg-info/SOURCES.txt'
pdm.termui: creating '/tmp/pdm-meta-uu6utr5w/my_test_package-0.1.dist-info'
pdm.termui: Pinning: my-test-package 0.1
pdm.termui: ======== Ending round 1 ========
pdm.termui: ======== Starting round 2 ========
pdm.termui: ======== Resolution Result ========
pdm.termui: Stable pins:
pdm.termui:            python None
pdm.termui:   my-test-package file:///${PROJECT_ROOT}/mypkg
🔒 Lock successful
Changes are written to pdm.lock.

PDM 2.2.1 is installed, while 2.4.2 is available.
Please run `pdm self update` to upgrade.
Run `pdm config check_update false` to disable the check.

Notes

If, in step 2 of "Step to Reproduce", one uses pdm 2.4.2 to add the local package (pdm add ./mypkg), the resulting pyproject.toml includes the absolute path to the local package, rather than using ${PROJECT_ROOT}:

[tool.pdm]
[project]
requires-python = ">=3.7"
dependencies = [
    "my-test-package @ file:///abs/path/to/project/mypkg",
]

This is less than ideal, since it eliminates the ability to relocate the project (e.g. by cloning the containing git repo to another directory.)

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
$ pdm info
PDM version:
  2.4.2
Python Interpreter:
  /path/to/project/.venv/bin/python (3.10)
Project Root:
  /path/to/project
Project Packages:
  None
$ pdm info --env
{
  "implementation_name": "cpython",
  "implementation_version": "3.10.9",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "5.14.0-1056-oem",
  "platform_system": "Linux",
  "platform_version": "#63-Ubuntu SMP Fri Dec 16 14:32:59 UTC 2022",
  "python_full_version": "3.10.9",
  "platform_python_implementation": "CPython",
  "python_version": "3.10",
  "sys_platform": "linux"
}

The text was updated successfully, but these errors were encountered:

frostming · 2023-02-02T00:45:54Z

Because the new version doesn't include a default build-system table anymore, and it falls back to setuptools. You can add it to the pyproject.toml:

[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"

dairiki · 2023-02-02T06:21:52Z

@frostming Thank you for the quick response. That works. I figured I was missing something.

Since I am just working on a "project" rather than a "library", and never intend to build a distribution, it never occurred to me that the build-system setting was relevant.

dairiki added the 🐛 bug Something isn't working label Feb 1, 2023

frostming closed this as completed in 91a500d Feb 2, 2023

hannahdiels-ontoforce mentioned this issue May 15, 2023

setuptools backend does not find PROJECT_ROOT dependencies #1927

Closed

j178 mentioned this issue Apr 3, 2024

recursive script j178/pdm#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regression: ${PROJECT_ROOT} for relative paths to local packages not working in pdm > 2.2.1 #1658

Regression: ${PROJECT_ROOT} for relative paths to local packages not working in pdm > 2.2.1 #1658

dairiki commented Feb 1, 2023

frostming commented Feb 2, 2023 •

edited

Loading

dairiki commented Feb 2, 2023

Regression: ${PROJECT_ROOT} for relative paths to local packages not working in pdm > 2.2.1 #1658

Regression: ${PROJECT_ROOT} for relative paths to local packages not working in pdm > 2.2.1 #1658

Comments

dairiki commented Feb 1, 2023

Steps to reproduce

Actual behavior

Expected behavior

Notes

Environment Information

frostming commented Feb 2, 2023 • edited Loading

dairiki commented Feb 2, 2023

frostming commented Feb 2, 2023 •

edited

Loading