Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for git source dependencies in pixi build #2680

Merged
merged 35 commits into from
Dec 19, 2024

Conversation

nichmor
Copy link
Contributor

@nichmor nichmor commented Dec 10, 2024

Overview

Adding support of git dependencies for pixi build.
New and important changes:

  • adding subdirectory in PixiSpec. This means that user can write this in pixi.toml:
    boost-check = { git = "https://github.com/nichmor/pixi-build-examples", subdirectory="something" }
    Other supported git references:
    boost-check = { git = "https://ghp_yourpersonal_token@github.com/nichmor/pixi-build-examples.git", branch = "main"}
    boost-check = { git = "ssh://git@github.com/nichmor/pixi-build-examples", branch = "fix-backend" }
    or any URLs with user and password

In the screenshot, with the green color I highlighted this field.
image

  • Adding pixi_git crate. This is copied mostly from uv-git and trying to keep minimum additional changes.
    It expose GitUrl ( which is not aware of subdirectory, just a url from git perspective ) and a GitResolver , that can receive GitUrl and fetch it somewhere in the cache and return the path to fetched repository.
    In screenshot we can see a small overview of it:
    image

  • on pixi_build, I changed PinnedGitSpec. It now contains a new field of PinnedGitCheckout type which represent a solved checkout of some source git spec. This, later, will be transformed into a LockedUrl and saved in pixi.lock.
    LockedUrl redact all the credentails, contains a mandatory git+ suffix and have exact commit in the fragments.
    In the attached screenshot, with green I highlighted the new types and how they behave.
    Screenshot 2024-12-13 at 17 51 16

  • get_update_lock_file_and_prefix now also contain store_credentials_from_project function. This function is responsible for loading all the credentials from pixi.toml, and apply them to the locked urls ( because they are redacted and don't contain anymore any creds)

My vision is that by using LockedGitUrl , we can also represent PyPi types, which can be latter transformed back from lock file into some PyPi type. Also, rattler-build or pixi-build-backends can reuse pixi_git Giturl for they needs when solving some dependencies ( as they are unaware about subdirectories ) and can be constructed from the Urls.

Blocked things that needs to be merged before this PR:

  • pixi-build-backends should support new subdirectory in the manifest

Missing things that will be added in following PR's:

  • pixi add should also support adding git url
  • we should add progress reporter to pixi_git.
  • some integration tests/unit tests.

@nichmor nichmor marked this pull request as ready for review December 13, 2024 16:03
@nichmor nichmor changed the title feat: [wip] add support for git source dependencies in pixi build feat: add support for git source dependencies in pixi build Dec 13, 2024
crates/pixi_git/Cargo.toml Outdated Show resolved Hide resolved
crates/pixi_git/src/resolver.rs Outdated Show resolved Hide resolved
crates/pixi_git/src/sha.rs Show resolved Hide resolved
crates/pixi_record/src/pinned_source.rs Show resolved Hide resolved
crates/pixi_record/src/pinned_source.rs Show resolved Hide resolved
crates/pixi_record/src/pinned_source.rs Show resolved Hide resolved
crates/pixi_record/src/pinned_source.rs Outdated Show resolved Hide resolved
.rev
.clone()
.unwrap_or_else(|| GitReference::Rev(self.commit.clone()));
let locked_git_ref = self.source.reference.clone();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

short_sha == long_sha if it is the same :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we always record requested rev, so if user previously short sha we will check against it

@tdejager
Copy link
Contributor

Great work thank you 🥂 !

@Hofer-Julian Hofer-Julian enabled auto-merge (squash) December 19, 2024 14:38
@Hofer-Julian Hofer-Julian merged commit 26451a0 into prefix-dev:main Dec 19, 2024
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants