feat: allow for editable pypi source dependencies when using `path` #1044

tdejager · 2024-03-22T15:30:05Z

This PR adds the ability to do editable installs for pypi-packages. This requires the project to also use a pyproject.toml so that we know what build system to use.

This PR takes great inspiration on how UV does it with some changes.

We resolve editables for multiple platforms by using the metadata separately.
We only* actually build the wheel on install time.
There is some deviating installation logic because we can install directly from locked dependencies.

I think that when this PR lands you mostly want to use editable installs for path based dependencies that you control, instead of regular path based dependencies.

Locking

For locking we actually lock whether the dependency is editable and also save a hash to the pyproject.toml, setup.cfg, or setup.py files. If any of these change, the lock file is invalidated, this is analogous to how UV does this for determining whether to re-install but their code uses the timestamp instead.

ruben-arts

Nice work!! It worked for some project I have on my machine. Lets share it with the world.

Some small remarks but in general I approve!

ruben-arts · 2024-03-26T15:23:25Z

Cargo.toml

+# pep440_rs = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# pep508_rs = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# uv-build = { git = "https://github.com/wolfv/uv", tag = "expose-yanks" }
+# uv-cache = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# uv-client = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# uv-dispatch = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# uv-distribution = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# uv-installer = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# uv-interpreter = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# uv-normalize = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# uv-resolver = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# uv-traits = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# distribution-filename = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# distribution-types = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# install-wheel-rs = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# platform-tags = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# pypi-types = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }
+# requirements-txt = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }


Suggested change

# pep440_rs = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# pep508_rs = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# uv-build = { git = "https://github.com/wolfv/uv", tag = "expose-yanks" }

# uv-cache = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# uv-client = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# uv-dispatch = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# uv-distribution = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# uv-installer = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# uv-interpreter = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# uv-normalize = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# uv-resolver = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# uv-traits = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# distribution-filename = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# distribution-types = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# install-wheel-rs = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# platform-tags = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# pypi-types = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

# requirements-txt = { git = "https://github.com/wolfv/uv", branch = "expose-yanks" }

No we need to keep this because we have to patch it very often.

ruben-arts · 2024-03-26T15:41:20Z

src/install_pypi.rs

+    let installed = site_packages.iter().filter(|dist| {
+        dist.installer()
+            .unwrap_or_default()
+            .is_some_and(|installer| installer == "uv")


Is this still uv or should this also be pixi-uv?

That PR is still open :)

…pi-dependencies

henryiii · 2024-04-03T21:35:39Z

This doesn't seem to be doing what it's supposed to do. If I add the following to particle, for example:

[tool.pixi.project]
name = "particle"
channels = ["conda-forge"]
platforms = ["osx-64"]

[tool.pixi.dependencies]
particle  = { path = ".", editable = true}

Then it does not do an editable install, and it does not install from the local path. Instead, it pulls particle from conda-forge. If the name in tool.pixi.dependencies is not valid from conda-forge, it errors. I do get this warning:

 WARN pixi::project::manifest: BETA feature `[pypi-dependencies]` enabled!

henryiii · 2024-04-03T21:46:11Z

Ahh, I bet I got the wrong section name. Yep, [tool.pixi.pypi-dependencies] works. It would have been nice if editable/path was an error for conda dependencies.

tdejager · 2024-04-04T05:10:30Z

Indeed, let's make an issue out of that :)

pavelzw · 2024-04-04T17:09:06Z

examples/docker/pixi.toml

@@ -12,6 +12,9 @@ python = ">=3.12"
 flask = ">=3.0.2,<3.1"
 gunicorn = ">=21.2.0,<21.3"

+[pypi-dependencies]
+docker-project = { path = ".", editable = true }


doesn't this clash with postinstall-production below @tdejager?

Hmm yes it will get overwriten probably. But the .pth files might incorrectly stay in the prefix. We need the #1092 to do it correctly.

wip: editable installs

c039e40

tdejager marked this pull request as draft March 22, 2024 15:30

tdejager and others added 15 commits March 25, 2024 10:43

Merge branch 'main' into feat/editable-installs

09b02df

wip: integration of editables

7a8c73b

feat: added editable to resolve

61a2a87

fix: satisfiability of editables

186b933

fix: depend on rattler revision

88440be

feat: different way of building pypi deps

2b6906a

wip: installation of editables

cae1a65

feat: now uses editables for installation

12e1a36

fix: test

0eaa839

fix: some small fixes

aa6cd43

feat: changed docker example to use local editable

df5a652

fix: clippy fixes

52bf729

feat: added example for editable with extras

aca7870

fix: typo

9dcc591

feat: fix the case where the path is not a directory

54c6356

tdejager marked this pull request as ready for review March 26, 2024 14:52

tdejager and others added 3 commits March 26, 2024 16:04

feat: add editable to schema

8e3b34a

fix: sort cargo file

7861593

Merge branch 'main' into feat/editable-installs

dff1c2e

tdejager requested review from ruben-arts and baszalmstra March 26, 2024 15:10

tdejager added 2 commits March 26, 2024 16:20

fix: remove vendored function

2917844

fix: satifies not using correct path in some cases

16f1701

ruben-arts approved these changes Mar 26, 2024

View reviewed changes

ruben-arts and others added 4 commits March 27, 2024 16:26

docs: update documentation for editable and source dependencies in py…

4948526

…pi-dependencies

Merge branch 'main' into feat/editable-installs

aafe11f

feat: tried to fix satifiability for windows

cb18494

fix: normalize file new-lines when computing hashes

c27515a

Merge branch 'main' into feat/editable-installs

8e72765

ruben-arts merged commit 36ac560 into prefix-dev:main Mar 28, 2024
19 of 20 checks passed

olivier-lacroix mentioned this pull request Mar 29, 2024

Install project in editable mode when using a pyproject.toml manifest automatically #1083

Closed

henryiii mentioned this pull request Apr 3, 2024

pixi 0.18.0 Homebrew/homebrew-core#167888

Merged

pavelzw reviewed Apr 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: allow for editable pypi source dependencies when using `path` #1044

feat: allow for editable pypi source dependencies when using `path` #1044

tdejager commented Mar 22, 2024 •

edited

Loading

ruben-arts left a comment

ruben-arts Mar 26, 2024

tdejager Mar 28, 2024

ruben-arts Mar 26, 2024

tdejager Mar 28, 2024

henryiii commented Apr 3, 2024 •

edited

Loading

henryiii commented Apr 3, 2024 •

edited

Loading

tdejager commented Apr 4, 2024

pavelzw Apr 4, 2024

tdejager Apr 4, 2024 •

edited

Loading

feat: allow for editable pypi source dependencies when using path #1044

feat: allow for editable pypi source dependencies when using path #1044

Conversation

tdejager commented Mar 22, 2024 • edited Loading

Locking

ruben-arts left a comment

Choose a reason for hiding this comment

ruben-arts Mar 26, 2024

Choose a reason for hiding this comment

tdejager Mar 28, 2024

Choose a reason for hiding this comment

ruben-arts Mar 26, 2024

Choose a reason for hiding this comment

tdejager Mar 28, 2024

Choose a reason for hiding this comment

henryiii commented Apr 3, 2024 • edited Loading

henryiii commented Apr 3, 2024 • edited Loading

tdejager commented Apr 4, 2024

pavelzw Apr 4, 2024

Choose a reason for hiding this comment

tdejager Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

feat: allow for editable pypi source dependencies when using `path` #1044

feat: allow for editable pypi source dependencies when using `path` #1044

tdejager commented Mar 22, 2024 •

edited

Loading

henryiii commented Apr 3, 2024 •

edited

Loading

henryiii commented Apr 3, 2024 •

edited

Loading

tdejager Apr 4, 2024 •

edited

Loading