-
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
make build.zig.zon contain all urls, including mirrors, and hashes of entire dependency tree #14309
Comments
rather than mixing manual and automatic ones and since its going to contain info on the entire tree, imo it should be its own |
I doubt it's a good idea to modify user's |
It's already planned to modify |
Looks like we need a QA stream with describing complete package management workflow by writing an RFC there😅 |
Copying my comment: #14290 (comment)
EDIT: NuGet does store its full dependency tree in its lock file. However, that's a separate file that's not meant to be edited by the developer. |
While caching this information for performance does make sense, I do think it would be better for the full dependency tree to be cached in a different file, maybe |
This sounds a lot like a lockfile to me, which I'm missing in zig build now. I've fixed nixe's scripts in ZLS after ZIG introduced the package manager. My solution was basically to hardcode the url's of all dependencies and to use some ugly string manipulations to turn the zon file into an importable json. I never really liked this solution and now decided to give it another shot and create a more reliable way of building zig projects in nix, only to realize that there is no lockfile and the information in the zon file is insufficient. The information you list in this issue (the entire dependency tree) is important when you want to publish to Flathub too. Just like nix, Flathub's build system does not allow you to download files, so you have to create a JSON file containing all of the dependencies (those you cannot find in Flathub itself) so the system can download them and put them in the right place. And just like nix, it also requires a hash of the downloaded file so it can make sure it got the right thing before even attempting to build. I was first bitten by the lack of tarball hashes, and then by the realization that I have no information about the dependencies of the dependencies. The only way for me to circumvent that is to do what Also, while here, I would like to note that a lockfile should have information about all packages in the dependency tree, no matter if it is used or not during the build. That is, it may be possible to disable packages depending on the platform, but a lockfile should have information about those nonetheless, since the build platform might be different from the one creating the lockfile. I kinda got bit by it when building a rust project with |
I could talk anyone's ear off on lockfiles right now. I work in Amazon and work on maintaining their large-scale codebase. (And of course, any opinions here or ever are mine and not my employers.) There are going to be use-cases for and against lockfiles, where I'm defining a lockfile as the collection of hashes of an entire transitive dependency tree. I really really appreciate Zig's opinionated approach to the language, and the "no really, there is only one way to do this backbone. Lockfiles are most useful for projects that are distributed across many developers, projects that are dependency-heavy, or "production" applications that occasionally need to make updates to some software but not all. Lockfiles are just noise and hassle when the project is used by a small team, the project has few dependencies, and is just some library (where the lockfile might be of occasional interest, but in general just an opportunity for conflicts). I think Zig benefits most if libraries are checking in a build.zig.zon (even if they do not contain hashes). Putting the ever-changing lockfile hashes directly into the build.zig.zon will discourage checking in the whole file for some developer populations. |
Of course, it's fine to cache the artifacts (IDK in Above that... With how principled I've seen Zig be with things like "unused variable is a compilation error" I'd be a little wary of anything that gives incentives to make enormous dependency trees, or makes enormous dependency trees too pleasant to work with (say, like the uglier sides of the NPM ecosystem). On the same note you also don't want to make code sharing too difficult -- that makes an ecosystem where there ends up being only a handful of shared do-all-the-things libraries (say, Boost) and the language will get pressure to add things that could come from a library ecosystem. I'd need to think more on whether this is an argument for or against my previous recommendations, though. |
Lockfiles are pretty much required on nix, otherwise you have to use separate program to generate lockfile and commit it to your repo for nix build to be aware of any network calls it has to made (or lookups from binary cache). Without them offline builds are pretty much no-go unless we require people to commit their "$ZIG_GLOBAL_CACHE/p" folder. |
https://github.com/Cloudef/zig2nix can now build build.zig.zon projects with nix |
@andrewrk EDIT: nevermind, the lock file still should have all deps to prevent problems as described by #14309 (comment) |
Note that |
For reference, how I currently handle zig.zon deps: https://github.com/Cloudef/zig2nix/tree/master/tools |
Extracted from #14265.
The motivation for this is faster dependency fetching, avoiding round-trips from the network.
The idea here is that, after fetching all dependencies, recursively, zig would edit the user's
build.zig.zon
file and automatically add a section that contains all urls & hashes (including mirrors) for the entire dependency tree. It might look something like this:In this case it matches exactly with #14292 which could be nice, or, on the other hand, it might be important to make a distinction between automatically added ones (which should also be removed automatically) or manually added ones (which should always be preserved).
Depending on how #14288 is resolved with regards to conflict resolution, this section would perhaps be combined with more tooling-managed data about each dependency, such as dependency overrides:
When combined with #14294, this may additionally need to float the fetch plugins to the top along with mirrors.
The text was updated successfully, but these errors were encountered: