Cannot install a direct reference dependency as an "editable" dependency

[AustinTSchaffer]

Description

It seems that you can't install a package in editable mode, while simultaneously installing a package that depends on that package via a direct reference.

I came across this issue when splitting a project into 2 packages that both live in the same repository. Package A now depends on Package B via a direct reference. Since both are in the same repository, I tried installing both in editable mode, but pip came back with:

# pip install --use-feature=in-tree-build -e package_a/ -e package_b/
Obtaining file:///home/austin/workspace/pypa/test_project/package_a
Obtaining file:///home/austin/workspace/pypa/test_project/package_b
Processing ./package_b
ERROR: Cannot install package-a==1.0.0 and package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b)
    package-a 1.0.0 depends on package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

It seems that this action should be allowed taking the wording of the error message at face value.

I did some debugging and found that Resolution._add_to_criteria from src/pip/_vendor/resolvelib/resolvers.py raises a RequirementsConflicted exception. The resolver cannot resolve the 2 requirements below. The resolver fails to generate any candidates apparently because both requirements are instances of ExplicitRequirement.

RequirementInformation(requirement=ExplicitRequirement(EditableCandidate('file:///home/austin/workspace/pypa/test_project/project_b')), parent=None)
RequirementInformation(requirement=ExplicitRequirement(LinkCandidate('file:///home/austin/workspace/pypa/test_project/project_b')), parent=EditableCandidate('file:///home/austin/workspace/pypa/test_project/project_a'))

We have 2 workarounds for this:

1. Remove the direct reference from package_a, and just have package_a depend on any package named package_b. This is an issue because Package A won't know where to get Package B unless you install them both at the same time with pip install -e ./package_a/ -e ./package_b/. More worrying is if someone registers a package on PyPI that matches the name of our "Package B".
2. Install "Package A" first, then install "Package B" later. This is the solution we went with, since Pip now takes previously installed packages into consideration when solving an environment, and we keep the versions of dependencies of both packages relatively in sync. This does feel like it could cause problems down the road.

Expected behavior

I'm not sure what should be expected in this situation? It seems like one of these should be true:

1. Pip should allow pip install -e A -e B when A depends on B via a direct reference
2. Pip's error message for Point 1 should indicate why you can't do that
3. Setuptools should allow you to install a direct-reference package in editable mode

Point 3 doesn't feel right, but just wanted to throw that out there.

pip version

21.3.dev0 (commit: a53f888)

Python version

3.9.5

OS

Ubuntu 21.04 hirsute

How to Reproduce

Create a directory named "package_b". Create a setup.py in that directory with the contents:

from setuptools import setup
setup(name="package_b", version="1.0.0")

Create a directory named "package_a" in the same directory that contains the "package_b" directory. Create a setup.py in that directory with the contents:

from setuptools import setup
import os
package_b_dir = os.path.dirname(__file__).replace("package_a", "package_b")
setup(
    name="package_a",
    version="1.0.0",
    install_requires=[
        "package_b @ file://%s" % package_b_dir,
    ],
)

You should now have:

.
├── package_a
│   └── setup.py
└── package_b
    └── setup.py

Now run pip install -e ./package_a/ -e ./package_b/.

Output

The output is above in the description section, but pasting it here as well:


# pip install --use-feature=in-tree-build -e package_a/ -e package_b/
Obtaining file:///home/austin/workspace/pypa/test_project/package_a
Obtaining file:///home/austin/workspace/pypa/test_project/package_b
Processing ./package_b
ERROR: Cannot install package-a==1.0.0 and package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b)
    package-a 1.0.0 depends on package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[surgura]

I can reproduce this exact same error with pip 22.0.3. Workaround 2 works but is not as nice when publishing a package.

[1-900-MIXALOT]

"Setuptools should allow you to install a direct-reference package in editable mode"

I ran into this limitation as well. Would be great to see a solution.

[alextremblay]

Is there anything that can be done to fix this?

I'd like to try and contribute a solution, but I'm a bit lost on where to start.

[pfmoore]

See my comment on #12533. I think this is the same issue, and my comments there apply here too. Thanks for the explanation of how you ended up in this situation, it's easier to see the issue with a real-world example. But I think the comment I made there remains true - pip can't ensure that the dependencies (the two repositories for the two packages involved) will stay in sync, so it's unable to guarantee that any resolution it comes up with is safe.

If you have to use this layout, then I think your two workarounds are your best option.

[alextremblay]

Hi @pfmoore, I read your comments in the linked issue, and I understand what you’re saying.

I have a couple points I think are worth consideration:

don't think it's something we'd want to do just to support a usage which is at best questionable

there is one use case for this feature which is more than just questionable, it’s a use case that’s becoming more and more prevalent, barring roadblocks like this: monorepos. I do all of my development in a monorepo, and would dearly love to not have to jump through hoops to make it work with python

pip can't ensure that the dependencies (the two repositories for the two packages involved) will stay in sync

my view (and the reason I offered to submit a PR) is that a RequirementsConflicted exception where both requirements point to the same package at the same location is not actually a conflict and should not be treated as a conflict.

In the case where one of those two requirements also happens to be editable, there should be some method (command line flag or something) to allow pip to resolve the conflict by choosing either the editable or non-editable variant of that dependency

as I said in a previous comment, I’m willing to start work on a PR, I just need some guidance on portions of the code to focus my efforts

[pfmoore]

it’s a use case that’s becoming more and more prevalent, barring roadblocks like this: monorepos

I can't really comment on the monorepo use case, as I don't use them myself, but as far as I'm aware there are a number of difficulties with them, which I think need to be discussed at a higher level, to establish what the "monorepo workflow" is and what needs to change in the packaging ecosystem to properly support them - assuming we want to¹. The need to support relative pathnames as a legitimate dependency (or package) specifier is the most obvious example, and this is part of that question.

I’m willing to start work on a PR, I just need some guidance on portions of the code to focus my efforts

The problem here is that you'd need to start by looking at how pip decides whether two packages are "the same". That's a very basic decision that impacts a lot of the logic, so you could end up going down something of a rabbit hole. The basic answer is that two "candidates" (to use the term the resolver works with) are the same if they have the same package name and version. However, layered on top of that are a bunch of heuristics around source trees identified by URLs or local paths. And editable installs are another layer on top of all that, which are not really considered in the resolution algorithm at all². By all means take a look, but please don't be surprised if the problem turns out to be a lot more complex than you'd hoped.

Also, when you say "both requirements point to the same package at the same location" you need to consider how you define "the same location" - are C:\Work\myproject and T:\myproject the same location? Does your answer change if T: is mapped to C:\Work? What about c:/work/myproject or file:///c:/work/myproject? I could continue, but my point is that someone will no doubt raise an issue if we don't take account of these things - and who's to say their use case is less important than yours?

Sorry if this sounds negative. It's sort of what I was trying to get at when I said we're "doing the best we can" 🙂 I appreciate the offer of help, it's just that the learning curve is steep, and this isn't an easy place to start...

Footnotes

1. I say that on the basis that I don't know if packaging tools in other language ecosystems support monorepos, or if they are a uniquely Python workflow. ↩

2. I was one of the developers of the new resolution algorithm, and from what I recall, there is nothing in that code which takes account of the "editable" flag. ↩

[AustinTSchaffer]

@pfmoore thanks so much for all of the details and background on this issue. I've also been working with monorepos more across various languages, and I think this issue was a result of me making a foray into that line of thinking.

I think in the future I'll just avoid doing the part that got me into this mess:

I came across this issue when splitting a project into 2 packages that both live in the same repository.

I honestly don't really remember what I was trying to accomplish by doing this. Maybe dependency isolation?

I think in the future I'll probably just use a single package definition, instead of trying to divide one large package into components. The "optional-dependencies" feature would allow downstream applications to reduce the total number of packages which are installed, based on need. Also there's no reason why the applications themselves couldn't live in the same repository under the same package structure, with different entrypoints defined via project.scripts.

The workaround that I actually ended up going with was to not try installing the package(s) in editable mode. If I made changes to one of the underlying packages while working on one of the downstream applications, I just reinstalled the dependencies locally after making changes to them. I also got a new job shortly afterward, which helped as well.

[potiuk]

FYI. @AustinTSchaffer: There are various Ideas how monorepos can be handled and there are various tools supporting them. Many people understand monorepos in various ways. And I think personally that until there is a PEP/standard describing it it's quite good that pip does not take an opinionated approach and leave it for other tools (that are compliant with other standards) to propose their ways.

(Note - I am not a maintainer or heavy contributor to pip - mostly a heavy user and Airflow "The CI/Dev tool guy").

From my research in this area:

• Hatch does not yet have "good" monorepo support - it's being thought about - Monorepo support hatch#233 , but I was able to very nicely implement monorepo approach wher multiple projects share the same source tree using a combination of standard-compliant build hooks in hatchling, and some packaging scripting around building multiple packages. We are mostly dynamically generating pyproject.tomls and using flit to build "sub-packages" in Airlfow - we can do it mostly because the 90+ packages we have are following the same patterns so we can easily generate "similar pyproject.toml" and automatically merge information from dependencies to common pyproject.toml (using pre-commit scripts). It's pretty complex to setup but it does the job for us and keeps us running until there will be some standard PEPs and tooling implementing them in "good enough" way for us that we can switch to (similarly to what we've done in December to catch up with pretty much every accepted (and few Draft ones) Packaging PEP out there and chose hatchling as built backend, and recommend hatch as frontend (without requiring it though).

Happy to share my experiences, but we have 90+ packages in Airlfow's monorepo with a great development workflow and release process as of apache/airflow#36537 - following all the standard, pyproject.toml only and with nice set of development extras, It requires quite a lot of tooling around, but it works really nicely for Airflow.

• Rye has an interesting approch for workspaces https://rye-up.com/guide/workspaces/ - which seems to be a nice light-weight approach that should cover a lot of needs of most projects that just need to have several editable projects installed together from the same monorepo together. This seems like as little opinionated as possible and I will likely take a look at it as an option and at the very least it might be great inspiration/approach for hatch future monorepo support (though rye has opinionated choices in other areas - dev dependencies etc.) and those are also not standardized yet (though it's very close I think with https://peps.python.org/pep-0735/ being in Draft (and i thnk ryes approach to monorepo is simple and nice).

• Poetry does not have direct support, there is Support subprojects in a poetry project python-poetry/poetry#2270 where things are being discussed, but IMHO generally (historically) poetry's approach was quite a bit too opinionated for me (mostly because of lack of standards, so it's not a complaint - it's changing now similarly as with all other tools which are heading into "we are supporting all the Packaging standards out-of-the-box direction, which is cool)

• Polylith - https://polylith.gitbook.io/polylith/ is another interesting approach that introduce a complete, though super-opinionated approach where you can built your application out of building-blocks similar to building stuff from LEGO blocks (that's even in part inspiration for their naming where base in polylith is like the wide plain base you usually use to build somethign on top of) - while interesting to build polylithic apps, it requires the tooling and building your app in a specific way, so i find it personally quite a bit too constraining. They used to be a poetry plugin, but as of last few months they have a standalone, independent tooling and you can use it without poetry.

I likely miss a few others, but those are the ones I am following very closely and try to contribute where I see possibility to get it more standards. My recommentation is though that until the standards are there, it's good to choose a solution that will not make you "tied" to a particular solution too much but will keep you going and allow you to "catch up" when the others will provide good tooling following standards. Like in our case pip is the way we recommend our users to install and deploy airflow, but we use hatchling as (Packaging standar) backend, recomment hatch as frontend. Even recently we are switching to the new kid on the block from ruff creators - uv to speed up our CI / image building process (roughly by 60%-70% from the initial tests I've done). PR here: apache/airflow#37683

That's all thanks to the standard developed by Packaging team becoming implemented and followed - creating the situatuions where in the same project you can even mix and match the different tools do the job they are best in. We are currently using pip, hatchling, hatch, flit and soon uv + our own custom monorepo setup in one project and it all makes perfect sense (and I am sure in the future we will be adapting and changing it - either merging some usages or even introducing other / new tools as needed- though with the current setup is allready feels a bit enough tools for now) .