Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

introduce the zig fetch subcommand and symlink support in zig packages #17363

Merged
merged 6 commits into from
Oct 3, 2023
Merged

introduce the zig fetch subcommand and symlink support in zig packages #17363

merged 6 commits into from
Oct 3, 2023

Conversation

andrewrk
Copy link
Member

@andrewrk andrewrk commented Oct 2, 2023 •
edited
Loading 

zig fetch [options] <url>
zig fetch [options] <path>

Fetches a package which is found at <url> or <path> into the global cache directory, printing the package hash to stdout.

Closes #16972
Related to #14280

Additionally, this branch:

Merge Checklist

  • support symlinks in Package/git.zig
  • more kinds of symlink metadata in std.tar eagerly awaiting @vesim987's upcoming PR for this
  • handle Diagnostics from std.tar

@andrewrk
Copy link
Member Author

andrewrk commented Oct 2, 2023

cc @ianprime0509 - would you be willing to add the git.zig logic to add symlink support here?

@ianprime0509
Copy link
Contributor

Awesome! Sure, I'll take a closer look at this after work and send a patch to add support for this in Git.

@andrewrk
std.tar: support symlinks
a5144d1 
closes #16678
@andrewrk
compiler: extract package hashing logic to separate file
a435298 
There are no functional changes in this commit.
@andrewrk
std.fs: give readLink an explicit error set
309c532
@andrewrk
introduce the 'zig fetch' command + symlink support
ef9966c 
zig fetch [options] <url>
zig fetch [options] <path>

Fetches a package which is found at <url> or <path> into the global
cache directory, printing the package hash to stdout.

Closes #16972
Related to #14280

Additionally, this commit:

* Adds uncompressed .tar support to package fetching
* Introduces symlink support to package fetching
@andrewrk
zig fetch: enhanced error reporting
2118118 
* Package: use std.tar diagnostics to give detailed error messages
* std.tar: add diagnostic for unsupported file type
@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

more kinds of symlink metadata in std.tar

eagerly awaiting @vesim987's upcoming PR for this

@andrewrk andrewrk enabled auto-merge October 3, 2023 00:22
@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

Note that zig fetch on a directory will be a bit useless until #14311 is resolved, because the package will potentially include files from version control which would not make it into a tarball.

@ianprime0509
Copy link
Contributor

@andrewrk not sure what would be most convenient for you (maybe I should just send a follow-up PR once this is merged), but here's a patch to integrate this with git.zig as well: https://gist.github.com/ianprime0509/f097b73a77e0c9412fe4a11824bd169d

This is able to fetch the dependency in this PR: kristoff-it/orca-ui-hello-zig#1

⬢[ian@toolbox orca-ui-hello-zig]$ ~/src/zig/build/stage4/bin/zig build
Fetch Packages... orca [9MiB] /var/home/ian/src/tmp/orca-ui-hello-zig/build.zig.zon:8:20: error: dependency is missing hash field
            .url = "git+https://git.handmade.network/hmn/orca.git#c4a1ad52a07bae2e6710c38f41a3c64ed8f93136",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: expected .hash = "1220a913fdaac3962e4331bd71d729b70da9ecfa2a959d62d6c491f18b4619cdca61",

Sample when the underlying filesystem doesn't support symlinks:

⬢[ian@toolbox orca-ui-hello-zig]$ ZIG_GLOBAL_CACHE_DIR=$HOME/src/tmp/fat.mnt ~/src/zig/build/stage4/bin/zig build
Fetch Packages... orca [8MiB] Checkout... /var/home/ian/src/tmp/orca-ui-hello-zig/build.zig.zon:8:20: error: unable to unpack packfile
            .url = "git+https://git.handmade.network/hmn/orca.git#c4a1ad52a07bae2e6710c38f41a3c64ed8f93136",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: unable to create symlink from 'src/ext/wasm3/platforms/android/app/src/main/cpp/m3' to '../../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/arduino/lib/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/bluepill/lib/wasm3/src' to '../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/esp32-idf-wasi/main/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/esp32-pio/lib/wasm3/src' to '../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/esp8266/lib/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/fomu/src/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/hifive1/lib/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/particle/lib/wasm3/src' to '../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/wm_w600/wasm3' to '../../../source': AccessDenied
note: unable to create symlink from 'tests/files/data/jail/dir_escape' to '..': AccessDenied
note: unable to create symlink from 'tests/files/data/jail/file_escape' to '../regular.txt': AccessDenied
note: unable to create symlink from 'tests/files/data/symlink' to 'regular.txt': AccessDenied

@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

If you're fine with it, I can cherry-pick from your zig fork into this branch

@ianprime0509
Copy link
Contributor

That sounds like an easier idea, I just pushed it up here: https://github.com/ianprime0509/zig/tree/tar-symlinks

@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

Commit looks great 👍

@andrewrk andrewrk merged commit df4853a into master Oct 3, 2023
@andrewrk andrewrk deleted the tar-symlinks branch October 3, 2023 10:33
@kassane kassane mentioned this pull request Oct 9, 2023
Open
1 task
@kassane kassane mentioned this pull request Nov 1, 2023
Open
@ianprime0509 ianprime0509 mentioned this pull request Nov 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

build.zig.zon improve workflow to update dependency hash std.tar: handle tar files with symbolic links
2 participants
@andrewrk @ianprime0509