Install cache is not concurrent-proof - errors on parallel executions

[f3flight]

• I have searched the issue tracker and believe that this is not a duplicate.

Disclaimer - this may be more of a feature request then a bug, I guess somewhere in between.

Steps to reproduce

1. enable pdm install cache (in this example .pdm is used as cache folder):

Project configuration (/home/user/Repos/testproj/.pdm.toml):
cache_dir = .pdm
install.cache = True

2. run pdm in parallel using tox-pdm plugin

Actual behavior

Sporadic pdm sync failures with errors like:

[FileExistsError]: [Errno 17] File exists: '.pdm/wheels/c8/11/71/9a7a3b9e3aea83b0244ccc8f58d1d3bc150090bc56133cc8c0'

Expected behavior

pdm cache is usable in parallel executions / concurrent runs

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
[user@vm testproj]$ ./pdm info && ./pdm info --env
PDM version:
  2.4.9
Python Interpreter:
  /home/user/Repos/testproj/.venv/bin/python (3.11)
Project Root:
  /home/user/Repos/testproj
Project Packages:
  None
{
  "implementation_name": "cpython",
  "implementation_version": "3.11.1",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "3.10.0-1160.88.1.el7.x86_64",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Tue Mar 7 15:41:52 UTC 2023",
  "python_full_version": "3.11.1",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "linux"
}

[frostming]

Can you provide the traceback by adding the -v option to pdm invocations on where this error comes from?

[f3flight]

Sorry I tried reproducing a few times but failed, haha =) But I see you've made some changes anyway, so maybe you were able to figure it out? I'll reopen if I'm able to reproduce this again, sorry for the late response.

[f3flight]

I ran into this again with pdm 2.6.0 and managed collect traceback from a debug log:

pdm.termui: Installing flit-core@3.8.0...
pdm.termui: Installing wheel into cached location /export/home/tester/.cache/pdm/packages/flit_core-3.8.0-py3-none-any
pdm.termui: Error occurs
Traceback (most recent call last):
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/termui.py", line 236, in logging
    yield logger
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/cli/actions.py", line 219, in do_sync
    synchronizer.synchronize()
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/synchronizers.py", line 422, in synchronize
    handlers[kind](key, progress)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/synchronizers.py", line 293, in update_candidate
    self.manager.install(can)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/manager.py", line 33, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/models/candidates.py", line 373, in build
    self.wheel = Path(builder.build(build_dir, metadata_directory=self._metadata_dir))
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/builders/wheel.py", line 27, in build
    self.install(requires)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/builders/base.py", line 294, in install
    install_requirements(missing, env)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/core.py", line 25, in install_requirements
    resolved, _ = resolve(
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/resolver/core.py", line 35, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/resolvelib/resolvers.py", line 426, in resolve
    name = min(unsatisfied_names, key=self._get_preference)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/resolvelib/resolvers.py", line 203, in _get_preference
    return self._p.get_preference(
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/resolver/providers.py", line 85, in get_preference
    is_backtrack_cause = any(dep.identify() in backtrack_identifiers for dep in self.get_dependencies(candidate))
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/resolver/providers.py", line 182, in get_dependencies
    deps, requires_python, _ = self.repository.get_dependencies(candidate)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/models/repositories.py", line 82, in get_dependencies
    requirements, requires_python, summary = getter(candidate)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/models/repositories.py", line 41, in wrapper
    result = func(self, candidate)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/models/repositories.py", line 213, in _get_dependencies_from_metadata
    deps = prepared.get_dependencies_from_metadata()
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/models/candidates.py", line 510, in get_dependencies_from_metadata
    self.req.project_name, self.metadata.requires or [], extras  # type: ignore[arg-type]
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/models/candidates.py", line 496, in metadata
    result = self.prepare_metadata()
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/models/candidates.py", line 479, in prepare_metadata
    self._metadata_dir = builder(self._unpacked_dir, self.environment).prepare_metadata(metadir_parent)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/builders/wheel.py", line 13, in prepare_metadata
    self.install(self._requires, shared=True)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/builders/base.py", line 294, in install
    install_requirements(missing, env)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/core.py", line 32, in install_requirements
    syncer.synchronize()
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/synchronizers.py", line 243, in synchronize
    manager.install(can)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/manager.py", line 33, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/installers.py", line 195, in install_wheel_with_cache
    _install_wheel(wheel=wheel, destination=destination)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/installers.py", line 268, in _install_wheel
    install(source, destination, additional_metadata=additional_metadata or {})
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/installer/_core.py", line 109, in install
    record = destination.write_file(
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/installer/destinations.py", line 207, in write_file
    return self.write_to_fs(scheme, path_, stream, is_executable)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/pdm/installers/installers.py", line 144, in write_to_fs
    return super().write_to_fs(scheme, path, stream, is_executable)
  File "/export/home/tester/.shiv/lipy-toolchain/site-packages/installer/destinations.py", line 167, in write_to_fs
    raise FileExistsError(message)
FileExistsError: File already exists: /export/home/tester/.cache/pdm/packages/flit_core-3.8.0-py3-none-any/lib/flit_core/__init__.py

[f3flight]

@frostming not sure how to reopen, looks like I cannot. Hope the traceback helps