Project with path dependency cannot be installed with pip install . (or via requirements file) due to "RequirementParseError: Invalid URL"

[huonw]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 11.4
• Poetry version: 1.1.13, 1.2.0b2.dev0 (f2c365f)
• Link of a Gist with the contents of your pyproject.toml file: https://github.com/huonw/poetry-path-deps-issue

Issue

We're working within a monorepo where we have several projects that depend on each other via path = "..." relative dependencies. Installing such a project directly with pip install fails with pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a.

Why are we trying to do this? We want to have access to pip install --target to install packages to a specific directory, when packaging for an AWS Lambda .zip file. This means we're exporting to a requirements file and installing that.

This is potentially caused by only absolute paths being turned into a URL, in https://github.com/python-poetry/poetry-core/blob/c5f4cda45402c38bb87ab73d0e32de509ba68f41/src/poetry/core/packages/directory_dependency.py#L125

Potentially related issues:

• similar error messages: Poetry install with path dependencies and entry points is unable to load the package #3930, "poetry build" generates wheel with invalid Requires-Dist URL when pyproject.toml specifies dependency by path #3899, Local path in wheel metadata seems wrong #4709, Wheel installation fails when local path dependency introduced #4868
• similar set-up: requirements.txt not installable by pip when local package depends on another poetry-plugin-export#147
• Monorepo / Monobuild support? #936 for the monorepo style with a lot of path deps

Example

https://github.com/huonw/poetry-path-deps-issue contains a full reproducible example:

cd /tmp
git clone https://github.com/huonw/poetry-path-deps-issue.git
pip install poetry-path-deps-issue/b

The core layout is:

.
├── a
│   ├── a
│   │   └── __init__.py
│   └── pyproject.toml
├── b
│   ├── b
│   │   └── __init__.py
│   └── pyproject.toml
└── c
    ├── c
    │   └── __init__.py
    ├── poetry.lock
    └── pyproject.toml

Where b/pyproject.toml contains a dependency a = { path = "../a" } (c is left over from an older version of this issue, see below).

The pip install . fails with:

Processing /poetry-path-deps-issue/b
...
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a
...

Click for older details using poetry export

c/pyproject.toml contains a dependency b = { path = "../b" }.

Commands:

cd poetry-path-deps-issue/c
poetry export -o requirements.txt
pip install -r requirements.txt

After running those commands, the requirements.txt contains something like:

a @ file:///private/tmp/poetry-path-deps-issue/a; python_version >= "3.9" and python_version < "4.0"
b @ file:///private/tmp/poetry-path-deps-issue/b; python_version >= "3.9" and python_version < "4.0"

And the pip install output is:

Processing /private/tmp/poetry-path-deps/a
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Processing /private/tmp/poetry-path-deps/b
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
ERROR: Exception:
...
  File ".../lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a

Workaround

We're preprocessing all pyproject.toml files to remove the path dependencies, after exporting the requirements, but before installing them (via find -name pyproject.toml | xargs sed -i '/path =/d'). The export should contain all of the dependencies required, so we don't them to be re-read out of each pyproject.toml.

Full output

Processing /poetry-path-deps-issue/b
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/packaging/requirements.py", line 117, in __init__
    raise InvalidRequirement(f"Invalid URL: {req.url}")
pip._vendor.packaging.requirements.InvalidRequirement: Invalid URL: ../a

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 341, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in <listcomp>
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 246, in iter_dependencies
    requires = self.dist.iter_dependencies() if with_requires else ()
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/metadata/pkg_resources.py", line 200, in iter_dependencies
    return self._dist.requires(extras)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3033, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3094, in parse_requirements
    yield Requirement(line)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a

[Secrus]

Hi. As with release 1.2, export is gonna be moved to plugin poetry-plugin-export, could you please check if this issue is still present with poetry 1.2 and plugin? If so, please report it in the plugin repository (linked above).

[huonw]

Thanks for checking in, @Secrus. The problem still exists with poetry 1.2 (on the master branch) plus poetry-plugin-export, however, on reflection, talking about poetry export is actually a distraction: running just pip install . in b shows the same problem, both in 1.1 and 1.2.

I'll update the issue.

[fx-kirin]

this pip issue might be related. They said poetry itself should be followed pip's way 'file://' and should not allow relative path.

[koffie]

I have the feeling this is a duplicate of #3148 , there is a workaround mentioned there in a comment #3148 (comment)

[jcosmao]

Just got same issue, i was able to install correctly with:

# relative deps from pyproject.toml are converted to absolute path
poetry export -f requirements.txt -o requirements.txt --without-hashes
pip install -r requirements.txt
# ignore dependencies as it has already been installed by requirements
pip install . --no-dependencies

[dimbleby]

Looks like this is languishing for lack of interest and indeed I don't myself really understand why it's wanted.

But it looks pretty trivial to fix this so that it always uses an absolute path, so if anyone is motivated to put together a pull request...

(cf python-poetry/poetry-core#141)

[TBBle]

The use-case I have for this is wheels that are built into a known location as part of a Dockerfile stage, and then installed from that location by a later stage from a fresh base using pip where neither Poetry nor build are present. So the wheels are never distributed, or even moved.

Our current workaround is to use sed to turn the pathed requirement into a * requirement before building the wheel (using build and hence poetry-core, but poetry itself is not present at this stage), and then we install wheels by-name in dependency order so that each such "sed'd local dependency" is seen by pip as already-fulfilled, and it doesn't try and grab unrelated stuff off PyPI.

Once challenge in this use-case that is orthogonal to this issue is that the absolute paths generated would need to be the lower-level wheel paths, rather than the absolute path of the source in the monorepo.

It only just now occurred to me that if we used sed to rewrite the pathed dependency into being the generated absolute wheel path rather than a project-name dependency, then this bug would no longer be relevant there as Poetry would (untested) generate a pip-valid file:// URL from that. So we'd be working around the 'challenge' here, but not this bug.

[adriangb]

I get the feeling that most folks using path dependencies are working with some sort of monorepo setup. There are two use cases I've seen for path dependencies in these workflows:

• Include the source code and dependencies
• Treat it like a published package with a version

For the first case I think the logical thing to do is to treat a build or pip install as including the path dependencies source files like if they had been specified via packages or include in this project's pyproject.toml and it's dependencies like if they had been included as direct dependencies in this project's pyproject.toml.

In other words, for the example given in this issue doing cd ./b && poetry build would treat this as if it was a project structured like:

[tool.poetry]
name = "b"
packages = [{include = "a", from = "../a"}]

[tool.poetry.dependencies]
python = "^3.9"
# all of a's dependencies

poetry-core could do the same. Then both poetry build and pip install would work (I think, I hacked up some stuff and it builds wheels correctly for me).

The second case is a bit more complex. There's questions about what version to use and such. Not unsolvable issues, but requiring discussion. And I think this is the minority of the use cases (also not this issue) so worth discussing elsewhere.

All of this would only work if the path dependency is a poetry project, but that seems like a reasonable limitation.
It could be made opt-in/backwards compatible with something like a = { path = "../a", build-as = "{path (default), include, require" }.

[nathanpainchaud]

I get the feeling that most folks using path dependencies are working with some sort of monorepo setup.

I think another common and legitimate use case is when working with git submodules. That's my case at least, and it would be great to be able pip install/publish my projects w/o requiring users to learn how to use poetry.

[neersighted]

@adriangb the use case you are tracking is tracked in #4583 -- it has technical challenges, but there is a user trying to implement it in python-poetry/poetry-core#273, more or less.

The second is tracked at #2270.

This issue specifically is tracking relative path dependencies in a non-monorepo setting and the ability to build a wheel from that -- the known path case in Docker above is likely the main use case (secondarily, NFS shares) and is the only issue not tracked somewhere else/described by the original issue.

[adriangb]

Thanks for linking me to those issues.

#4583 seems like a really nice solution. It seems like no one has attempted to implement it yet; I suspect the main challenge will be the same as python-poetry/poetry-core#356: we'll need to re-write the pyproject.toml and setup.py (or just not generate the latter for these cases) so that the paths point to the paths inside the artifact. This just requires some head banging (I tried for a bit, quit at 3am but will give it another shot at some point).

I think that python-poetry/poetry-core#273 isn't a true replacement for these other issues. Introducing workspace support into poetry-core seems like a much more long-term thing than fixing a buggy/undefined behavior that can have other applications. In particular, I've found it pretty straightforward to write a plugin that "virtually" transforms:

[tool.workspaces]
members = ["projects/foo"]

Into:

[tool.poetry.groups.foo.dependencies]
foo = {path = "projects/foo", develop = true}

I would like to also do the transformation proposed in #5273 (comment) so that the individual sub-packages can be built and/or published but that's not possible because of the bug/behavior in this issue.

That is to say, I think if we can get #4583 or python-poetry/poetry-core#356 working it will open up the door to plugins (including the one that prompted python-poetry/poetry-core#273) that do most if not all of what python-poetry/poetry-core#273 wants to do without introducing a workspace concept into poetry-core. For a feature like that it would probably be good to have more experimentation in plugins before absorbing the feature but it's currently very hard for plugins to experiment with the frontend/UX since they can only provide half baked functionality or have to resort to monkey patching stuff to work around #5273 (this issue) and other limitations.

[neersighted]

I think another common and legitimate use case is when working with git submodules. That's my case at least, and it would be great to be able pip install/publish my projects w/o requiring users to learn how to use poetry.

@nathanpainchaud, that's #2828, which could be argued is a specialized variant of #4583 if submodules are used.

[dimbleby]

if you are going as far as monkey-patching around this in plugins you really might as well submit the MR in poetry-core to have it use absolute paths...

[neersighted]

@adriangb, I am conceptualizing python-poetry/poetry-core#273 as a successor to python-poetry/poetry-core#356 -- I think the shape of it will have to change to be something in the middle, ideally implementing #4583.

Maybe it would be better to close that PR and re-open it as an explicit attempt at #4583, taking the best parts of the code and dropping the "workspace" marker bit (which is the only part that relates to workspaces at all -- the more I've thought about it, the more I think it's unnecessary).

We're looking at dropping setup.py generation soon (python-poetry/poetry-core#318), so synthesizing a pyproject.toml or rewriting all the paths in the tar (this is my preferred approach) ala-#4583 is the way to make this feature work.

Essentially, I am of the opinion that #4583 answers most of these use cases and forcing explicit from/to when relative paths are in play should be the way to solve the paths/pyproject.toml issue.

This issue is trivial to solve, if anyone wants to submit a PR to poetry-core to force the use of absolute paths. Many of the users here need #4583, but for the users who just want to be able to build a valid distfile (and let the consequences of absolute paths be what they may), we can solve that easily enough today.

[adriangb]

if you are going as far as monkey-patching around this in plugins you really might as well submit the MR in poetry-core to have it use absolute paths...

Well yes, I agree, that's what python-poetry/poetry-core#273 is attempting to do. I'm just saying that if #4583 or python-poetry/poetry-core#356 were fixed (hence closing the issue we are currently discussing) that monkey patching would not be needed, so introducing workspaces into poetry-core (python-poetry/poetry-core#273) is not the only solution.

Maybe it would be better to close that PR [https://github.com/python-poetry/poetry-core/pull/273] and re-open it as an explicit attempt at #4583, taking the best parts of the code and dropping the "workspace" marker bit (which is the only part that relates to workspaces at all -- the more I've thought about it, the more I think it's unnecessary).

I agree that trying to focus on just fixing the include from/to issue seems like a good plan to avoid bikeshedding on the whole workspace thing. I don't know that python-poetry/poetry-core#273 has to be closed, I think the "add a workspace concept to poetry" thing can be entertained in parallel (the author may want to close it if they are able to do everything in their plugin without any hacks, but that's for them to decide I guess).

We're looking at dropping setup.py generation soon (python-poetry/poetry-core#318), so synthesizing a pyproject.toml or rewriting all the paths in the tar (this is my preferred approach) ala-#4583 is the way to make this feature work.

👍🏻 yep makes sense. From the time I spent on python-poetry/poetry-core#356 rewriting the paths in pyproject.toml seems pretty straightforward, the setup.py part not so much, so if setup.py python-poetry/poetry-core#356 or a successor that uses from/to should be pretty viable.

[neersighted]

I think the bit of confusion here is the author of python-poetry/poetry-core#273 did a demo for project maintainers last week, and it turned out to basically be what I would consider to be a partial attempt at #4583 -- the workspace bit is really a red herring as it's only a marker file that prevents further path traversal.

Critically, that PR does not (look at the code, since the body is still a bit confusing since the author is from a different ecosystem) canonicalize path dependencies (dependencies, not package include entries) like the ask in this issue. I think we should migrate all discussion to #4583 of anything that isn't the requested normalization (absolute path) of path dependencies when exported in PEP 508 format for a build.

[dimbleby]

yep, the discussion here has drifted significantly but I think that what this issue about is just that poetry-core puts relative paths in metadata, and this is easily fixed by anyone who cares: tweak this code to use an absolute version of the full path that is already known to the directory dependency.

(and similar for file dependency)

[dimbleby]

duplicate #3148

[neersighted]

This issue is more descriptive and has more discussion, I'm going to duplicate in the other direction. Thanks for the link.

[luca-medeiros]

Facing a similar error when adding a local path to extras.

[tool.poetry]
name = "poetrylib"
version = "0.0.1"

[tool.poetry.dependencies]
python = "^3.8, <3.11"
testlib = {path = "path/testlib", optional = true} 

[tool.poetry.extras]
testlib = ["testlib"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

poetry install works fine but pip install . doesn't.

    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Parse error at "'extra =='": Expected string_end

[dimbleby]

not the same thing at all. You haven't said which extra(s) require testlib. Once you fix that, then you'll likely hit the thing this issue describes

too fast, too careless: but this still isn't the same thing that this issue describes!

[luca-medeiros]

@dimbleby

Can you point me to where my issue is coming from then? Should I open a new issue page?
I thought the root of it would be similar to this page's issue.

.
├── poetrylib
│   ├── testlib
│   │   └── __init__.py
│   │   └── pyproject.toml
│   └── __init__.py
│   └── pyproject.toml

testlib's pyproject is just a simple:

[tool.poetry]
name = "testlib"
version = "0.0.1"
packages = [
    { include = "*" },
]

[tool.poetry.dependencies]
python = "^3.8, <3.11"
numpy = "*" # for example

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

The error happens with both pip install . and pip install ".[testlib]"

[neersighted]

This issue is for a very specific issue relating to RequirementParseError: Invalid URL. A more generic issue was duplicated here as despite the generic description, it only addressed the same URL requirement issue. If Poetry is producing invalid metadata beyond the expected path URL, please open an issue with a minimal reproduction that anyone can clone to demonstrate the issue.

[absassi]

This issue has more discussion than #3148, but the discussion here went into a significantly different way than the discussion there. Here, we are talking mostly about dealing with workspaces and etc. and it's nice and important (and also discussed in many other issues as well), but the particular pkg_resources error itself is trivial to fix and do not require discussing what to do with workspaces.

Simply having any kind of valid string compliant to PEP 508 is enough to let pip install . work when the dependency is already installed and also resolves all sorts of issues when people use pkg_resources itself on an environment where everything is already installed, such as for loading entry points or programmatically listing dependencies.

Workspaces and discussions related to that are trying to find ways to solve how pip install . would find the relative-path dependency to install it, but most people that need that should just do poetry install instead, while those who use pkg_resources directly and those who use pip to install directly from a wheel file, are those who either already have all dependencies installed or know well where they are (and, in case of a set of wheel files, the relative path becomes meaningless anyway).

So, my suggestion is to just fix this specific invalid URL in metadata issue with either the absolute URL or no URL at all, regardless of workspaces. If some future need requires one of the two specific options, then it can be changed later, but at least those who don't care about the URL don't need to wait for the fix.

[neersighted]

Nobody here has suggested that we don't fix the relative path issue or that it something else needs to be addressed first -- feel free to send a PR that makes the paths absolute. All that has happened is that no one who has participated here/expressed interest in fixing a low-hanging bug has sent a patch and gotten it/its tests through code review.

[dimbleby]

bumping this comment to the end of the thread again

this is easily fixed by anyone who cares: tweak this code to use an absolute version of the full path that is already known to the directory dependency (and similar for the file dependency)

if the next person who gets here and cares about this would be good enough simply to submit an MR, that would be great!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.