Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Project with path dependency cannot be installed with pip install . (or via requirements file) due to "RequirementParseError: Invalid URL" #5273

Closed
3 tasks done
huonw opened this issue Mar 4, 2022 · 27 comments · Fixed by python-poetry/poetry-core#512
Labels
kind/bug Something isn't working as expected

Comments

@huonw
Copy link

huonw commented Mar 4, 2022

  • I am on the latest Poetry version.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Issue

We're working within a monorepo where we have several projects that depend on each other via path = "..." relative dependencies. Installing such a project directly with pip install fails with pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a.

Why are we trying to do this? We want to have access to pip install --target to install packages to a specific directory, when packaging for an AWS Lambda .zip file. This means we're exporting to a requirements file and installing that.

This is potentially caused by only absolute paths being turned into a URL, in https://github.com/python-poetry/poetry-core/blob/c5f4cda45402c38bb87ab73d0e32de509ba68f41/src/poetry/core/packages/directory_dependency.py#L125

Potentially related issues:

Example

https://github.com/huonw/poetry-path-deps-issue contains a full reproducible example:

cd /tmp
git clone https://github.com/huonw/poetry-path-deps-issue.git
pip install poetry-path-deps-issue/b

The core layout is:

.
├── a
│   ├── a
│   │   └── __init__.py
│   └── pyproject.toml
├── b
│   ├── b
│   │   └── __init__.py
│   └── pyproject.toml
└── c
    ├── c
    │   └── __init__.py
    ├── poetry.lock
    └── pyproject.toml

Where b/pyproject.toml contains a dependency a = { path = "../a" } (c is left over from an older version of this issue, see below).

The pip install . fails with:

Processing /poetry-path-deps-issue/b
...
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a
...
Click for older details using poetry export

c/pyproject.toml contains a dependency b = { path = "../b" }.

Commands:

cd poetry-path-deps-issue/c
poetry export -o requirements.txt
pip install -r requirements.txt

After running those commands, the requirements.txt contains something like:

a @ file:///private/tmp/poetry-path-deps-issue/a; python_version >= "3.9" and python_version < "4.0"
b @ file:///private/tmp/poetry-path-deps-issue/b; python_version >= "3.9" and python_version < "4.0"

And the pip install output is:

Processing /private/tmp/poetry-path-deps/a
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Processing /private/tmp/poetry-path-deps/b
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
ERROR: Exception:
...
  File ".../lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a

Workaround

We're preprocessing all pyproject.toml files to remove the path dependencies, after exporting the requirements, but before installing them (via find -name pyproject.toml | xargs sed -i '/path =/d'). The export should contain all of the dependencies required, so we don't them to be re-read out of each pyproject.toml.

Full output

Processing /poetry-path-deps-issue/b
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/packaging/requirements.py", line 117, in __init__
    raise InvalidRequirement(f"Invalid URL: {req.url}")
pip._vendor.packaging.requirements.InvalidRequirement: Invalid URL: ../a

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 341, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in <listcomp>
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 246, in iter_dependencies
    requires = self.dist.iter_dependencies() if with_requires else ()
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/metadata/pkg_resources.py", line 200, in iter_dependencies
    return self._dist.requires(extras)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3033, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3094, in parse_requirements
    yield Requirement(line)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a
@huonw huonw added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Mar 4, 2022
@Secrus
Copy link
Member

Secrus commented May 22, 2022

Hi. As with release 1.2, export is gonna be moved to plugin poetry-plugin-export, could you please check if this issue is still present with poetry 1.2 and plugin? If so, please report it in the plugin repository (linked above).

@Secrus Secrus removed the status/triage This issue needs to be triaged label May 22, 2022
@huonw
Copy link
Author

huonw commented May 22, 2022

Thanks for checking in, @Secrus. The problem still exists with poetry 1.2 (on the master branch) plus poetry-plugin-export, however, on reflection, talking about poetry export is actually a distraction: running just pip install . in b shows the same problem, both in 1.1 and 1.2.

I'll update the issue.

@huonw huonw changed the title Directory path dependency chain of length >= 3 cannot be installed via poetry export/pip install -r ... Project with path dependency cannot be installed with pip install . (or via requirements file) due to Invalid URL May 22, 2022
@huonw huonw changed the title Project with path dependency cannot be installed with pip install . (or via requirements file) due to Invalid URL Project with path dependency cannot be installed with pip install . (or via requirements file) due to "RequirementParseError: RequirementParseErroInvalid URL" May 22, 2022
@huonw huonw changed the title Project with path dependency cannot be installed with pip install . (or via requirements file) due to "RequirementParseError: RequirementParseErroInvalid URL" Project with path dependency cannot be installed with pip install . (or via requirements file) due to "RequirementParseError: Invalid URL" May 22, 2022
@fx-kirin
Copy link

fx-kirin commented Jun 1, 2022

this pip issue might be related. They said poetry itself should be followed pip's way 'file://' and should not allow relative path.

@koffie
Copy link

koffie commented Jul 19, 2022

I have the feeling this is a duplicate of #3148 , there is a workaround mentioned there in a comment #3148 (comment)

@jcosmao
Copy link

jcosmao commented Jul 23, 2022

Just got same issue, i was able to install correctly with:

# relative deps from pyproject.toml are converted to absolute path
poetry export -f requirements.txt -o requirements.txt --without-hashes
pip install -r requirements.txt
# ignore dependencies as it has already been installed by requirements
pip install . --no-dependencies

@dimbleby
Copy link
Contributor

dimbleby commented Oct 9, 2022

Looks like this is languishing for lack of interest and indeed I don't myself really understand why it's wanted.

But it looks pretty trivial to fix this so that it always uses an absolute path, so if anyone is motivated to put together a pull request...

(cf python-poetry/poetry-core#141)

@TBBle
Copy link
Contributor

TBBle commented Oct 11, 2022

The use-case I have for this is wheels that are built into a known location as part of a Dockerfile stage, and then installed from that location by a later stage from a fresh base using pip where neither Poetry nor build are present. So the wheels are never distributed, or even moved.

Our current workaround is to use sed to turn the pathed requirement into a * requirement before building the wheel (using build and hence poetry-core, but poetry itself is not present at this stage), and then we install wheels by-name in dependency order so that each such "sed'd local dependency" is seen by pip as already-fulfilled, and it doesn't try and grab unrelated stuff off PyPI.

Once challenge in this use-case that is orthogonal to this issue is that the absolute paths generated would need to be the lower-level wheel paths, rather than the absolute path of the source in the monorepo.

It only just now occurred to me that if we used sed to rewrite the pathed dependency into being the generated absolute wheel path rather than a project-name dependency, then this bug would no longer be relevant there as Poetry would (untested) generate a pip-valid file:// URL from that. So we'd be working around the 'challenge' here, but not this bug.

@adriangb
Copy link
Contributor

I get the feeling that most folks using path dependencies are working with some sort of monorepo setup. There are two use cases I've seen for path dependencies in these workflows:

  • Include the source code and dependencies
  • Treat it like a published package with a version

For the first case I think the logical thing to do is to treat a build or pip install as including the path dependencies source files like if they had been specified via packages or include in this project's pyproject.toml and it's dependencies like if they had been included as direct dependencies in this project's pyproject.toml.

In other words, for the example given in this issue doing cd ./b && poetry build would treat this as if it was a project structured like:

[tool.poetry]
name = "b"
packages = [{include = "a", from = "../a"}]

[tool.poetry.dependencies]
python = "^3.9"
# all of a's dependencies

poetry-core could do the same. Then both poetry build and pip install would work (I think, I hacked up some stuff and it builds wheels correctly for me).

The second case is a bit more complex. There's questions about what version to use and such. Not unsolvable issues, but requiring discussion. And I think this is the minority of the use cases (also not this issue) so worth discussing elsewhere.

All of this would only work if the path dependency is a poetry project, but that seems like a reasonable limitation.
It could be made opt-in/backwards compatible with something like a = { path = "../a", build-as = "{path (default), include, require" }.

@nathanpainchaud
Copy link

I get the feeling that most folks using path dependencies are working with some sort of monorepo setup.

I think another common and legitimate use case is when working with git submodules. That's my case at least, and it would be great to be able pip install/publish my projects w/o requiring users to learn how to use poetry.

@neersighted
Copy link
Member

neersighted commented Oct 11, 2022

@adriangb the use case you are tracking is tracked in #4583 -- it has technical challenges, but there is a user trying to implement it in python-poetry/poetry-core#273, more or less.

The second is tracked at #2270.

This issue specifically is tracking relative path dependencies in a non-monorepo setting and the ability to build a wheel from that -- the known path case in Docker above is likely the main use case (secondarily, NFS shares) and is the only issue not tracked somewhere else/described by the original issue.

@adriangb
Copy link
Contributor

Thanks for linking me to those issues.

#4583 seems like a really nice solution. It seems like no one has attempted to implement it yet; I suspect the main challenge will be the same as python-poetry/poetry-core#356: we'll need to re-write the pyproject.toml and setup.py (or just not generate the latter for these cases) so that the paths point to the paths inside the artifact. This just requires some head banging (I tried for a bit, quit at 3am but will give it another shot at some point).

I think that python-poetry/poetry-core#273 isn't a true replacement for these other issues. Introducing workspace support into poetry-core seems like a much more long-term thing than fixing a buggy/undefined behavior that can have other applications. In particular, I've found it pretty straightforward to write a plugin that "virtually" transforms:

[tool.workspaces]
members = ["projects/foo"]

Into:

[tool.poetry.groups.foo.dependencies]
foo = {path = "projects/foo", develop = true}

I would like to also do the transformation proposed in #5273 (comment) so that the individual sub-packages can be built and/or published but that's not possible because of the bug/behavior in this issue.

That is to say, I think if we can get #4583 or python-poetry/poetry-core#356 working it will open up the door to plugins (including the one that prompted python-poetry/poetry-core#273) that do most if not all of what python-poetry/poetry-core#273 wants to do without introducing a workspace concept into poetry-core. For a feature like that it would probably be good to have more experimentation in plugins before absorbing the feature but it's currently very hard for plugins to experiment with the frontend/UX since they can only provide half baked functionality or have to resort to monkey patching stuff to work around #5273 (this issue) and other limitations.

@neersighted
Copy link
Member

I think another common and legitimate use case is when working with git submodules. That's my case at least, and it would be great to be able pip install/publish my projects w/o requiring users to learn how to use poetry.

@nathanpainchaud, that's #2828, which could be argued is a specialized variant of #4583 if submodules are used.

@dimbleby
Copy link
Contributor

if you are going as far as monkey-patching around this in plugins you really might as well submit the MR in poetry-core to have it use absolute paths...

@neersighted
Copy link
Member

@adriangb, I am conceptualizing python-poetry/poetry-core#273 as a successor to python-poetry/poetry-core#356 -- I think the shape of it will have to change to be something in the middle, ideally implementing #4583.

Maybe it would be better to close that PR and re-open it as an explicit attempt at #4583, taking the best parts of the code and dropping the "workspace" marker bit (which is the only part that relates to workspaces at all -- the more I've thought about it, the more I think it's unnecessary).

We're looking at dropping setup.py generation soon (python-poetry/poetry-core#318), so synthesizing a pyproject.toml or rewriting all the paths in the tar (this is my preferred approach) ala-#4583 is the way to make this feature work.

Essentially, I am of the opinion that #4583 answers most of these use cases and forcing explicit from/to when relative paths are in play should be the way to solve the paths/pyproject.toml issue.

This issue is trivial to solve, if anyone wants to submit a PR to poetry-core to force the use of absolute paths. Many of the users here need #4583, but for the users who just want to be able to build a valid distfile (and let the consequences of absolute paths be what they may), we can solve that easily enough today.

@adriangb
Copy link
Contributor

if you are going as far as monkey-patching around this in plugins you really might as well submit the MR in poetry-core to have it use absolute paths...

Well yes, I agree, that's what python-poetry/poetry-core#273 is attempting to do. I'm just saying that if #4583 or python-poetry/poetry-core#356 were fixed (hence closing the issue we are currently discussing) that monkey patching would not be needed, so introducing workspaces into poetry-core (python-poetry/poetry-core#273) is not the only solution.

Maybe it would be better to close that PR [https://github.com/python-poetry/poetry-core/pull/273] and re-open it as an explicit attempt at #4583, taking the best parts of the code and dropping the "workspace" marker bit (which is the only part that relates to workspaces at all -- the more I've thought about it, the more I think it's unnecessary).

I agree that trying to focus on just fixing the include from/to issue seems like a good plan to avoid bikeshedding on the whole workspace thing. I don't know that python-poetry/poetry-core#273 has to be closed, I think the "add a workspace concept to poetry" thing can be entertained in parallel (the author may want to close it if they are able to do everything in their plugin without any hacks, but that's for them to decide I guess).

We're looking at dropping setup.py generation soon (python-poetry/poetry-core#318), so synthesizing a pyproject.toml or rewriting all the paths in the tar (this is my preferred approach) ala-#4583 is the way to make this feature work.

👍🏻 yep makes sense. From the time I spent on python-poetry/poetry-core#356 rewriting the paths in pyproject.toml seems pretty straightforward, the setup.py part not so much, so if setup.py python-poetry/poetry-core#356 or a successor that uses from/to should be pretty viable.

@neersighted
Copy link
Member

neersighted commented Oct 11, 2022

I think the bit of confusion here is the author of python-poetry/poetry-core#273 did a demo for project maintainers last week, and it turned out to basically be what I would consider to be a partial attempt at #4583 -- the workspace bit is really a red herring as it's only a marker file that prevents further path traversal.

Critically, that PR does not (look at the code, since the body is still a bit confusing since the author is from a different ecosystem) canonicalize path dependencies (dependencies, not package include entries) like the ask in this issue. I think we should migrate all discussion to #4583 of anything that isn't the requested normalization (absolute path) of path dependencies when exported in PEP 508 format for a build.

@dimbleby
Copy link
Contributor

yep, the discussion here has drifted significantly but I think that what this issue about is just that poetry-core puts relative paths in metadata, and this is easily fixed by anyone who cares: tweak this code to use an absolute version of the full path that is already known to the directory dependency.

(and similar for file dependency)

@dimbleby
Copy link
Contributor

duplicate #3148

@neersighted
Copy link
Member

This issue is more descriptive and has more discussion, I'm going to duplicate in the other direction. Thanks for the link.

@luca-medeiros
Copy link

Facing a similar error when adding a local path to extras.

[tool.poetry]
name = "poetrylib"
version = "0.0.1"

[tool.poetry.dependencies]
python = "^3.8, <3.11"
testlib = {path = "path/testlib", optional = true} 

[tool.poetry.extras]
testlib = ["testlib"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

poetry install works fine but pip install . doesn't.

    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Parse error at "'extra =='": Expected string_end

@dimbleby
Copy link
Contributor

dimbleby commented Oct 25, 2022

not the same thing at all. You haven't said which extra(s) require testlib. Once you fix that, then you'll likely hit the thing this issue describes

too fast, too careless: but this still isn't the same thing that this issue describes!

@luca-medeiros
Copy link

@dimbleby

Can you point me to where my issue is coming from then? Should I open a new issue page?
I thought the root of it would be similar to this page's issue.

.
├── poetrylib
│   ├── testlib
│   │   └── __init__.py
│   │   └── pyproject.toml
│   └── __init__.py
│   └── pyproject.toml

testlib's pyproject is just a simple:

[tool.poetry]
name = "testlib"
version = "0.0.1"
packages = [
    { include = "*" },
]

[tool.poetry.dependencies]
python = "^3.8, <3.11"
numpy = "*" # for example

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

The error happens with both pip install . and pip install ".[testlib]"

@neersighted
Copy link
Member

This issue is for a very specific issue relating to RequirementParseError: Invalid URL. A more generic issue was duplicated here as despite the generic description, it only addressed the same URL requirement issue. If Poetry is producing invalid metadata beyond the expected path URL, please open an issue with a minimal reproduction that anyone can clone to demonstrate the issue.

@absassi
Copy link

absassi commented Oct 25, 2022

This issue has more discussion than #3148, but the discussion here went into a significantly different way than the discussion there. Here, we are talking mostly about dealing with workspaces and etc. and it's nice and important (and also discussed in many other issues as well), but the particular pkg_resources error itself is trivial to fix and do not require discussing what to do with workspaces.

Simply having any kind of valid string compliant to PEP 508 is enough to let pip install . work when the dependency is already installed and also resolves all sorts of issues when people use pkg_resources itself on an environment where everything is already installed, such as for loading entry points or programmatically listing dependencies.

Workspaces and discussions related to that are trying to find ways to solve how pip install . would find the relative-path dependency to install it, but most people that need that should just do poetry install instead, while those who use pkg_resources directly and those who use pip to install directly from a wheel file, are those who either already have all dependencies installed or know well where they are (and, in case of a set of wheel files, the relative path becomes meaningless anyway).

So, my suggestion is to just fix this specific invalid URL in metadata issue with either the absolute URL or no URL at all, regardless of workspaces. If some future need requires one of the two specific options, then it can be changed later, but at least those who don't care about the URL don't need to wait for the fix.

@neersighted
Copy link
Member

Nobody here has suggested that we don't fix the relative path issue or that it something else needs to be addressed first -- feel free to send a PR that makes the paths absolute. All that has happened is that no one who has participated here/expressed interest in fixing a low-hanging bug has sent a patch and gotten it/its tests through code review.

@dimbleby
Copy link
Contributor

dimbleby commented Oct 25, 2022

bumping this comment to the end of the thread again

this is easily fixed by anyone who cares: tweak this code to use an absolute version of the full path that is already known to the directory dependency (and similar for the file dependency)

if the next person who gets here and cares about this would be good enough simply to submit an MR, that would be great!

Copy link

github-actions bot commented Mar 1, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.