package manager

[andrewrk]

Latest Proposal

---

Zig needs to make it so that people can effortlessly and confidently depend on each other's code.

Depends on #89

[phase]

My thoughts on Package Managers:

• Packages should be immutable in the package repository (so the NPM problem doesn't arise).

• Making a new release with a nice changelog should be simple. Maybe integration for reading releases from GitHub or other popular source hosts?

• Packages should only depend on packages that are in the package repository (no GitHub urls like Go uses).

• Private instances of the package repository should be supported from the get go (I believe this was a problem for Rust or some other newer language).

---

Thoughts that might not be great for Zig:

• Enforcing Semver is an interesting concept, and projects like Elm have done it.

[andrewchambers]

This is a good reference for avoiding the complexity of package managers like cargo, minimal version selection is a unique approach that avoids lockfiles, .modverify avoids deps being changed out from under you.

https://research.swtch.com/vgo

The features around verifiable builds and library verification are also really neat. Also around staged upgrades of libraries and allowing multiple major versions of the same package to be in a program at once.

[bnoordhuis]

Packages should be immutable in the package repository (so the NPM problem doesn't arise).

I assume you mean authors can't unpublish without admin intervention. True immutability conflicts with the hoster's legal responsibilities in most jurisdictions.

minimal version selection

I'd wait a few years to see how that pans out for Go.

[andrewchambers]

Note that by minimal, they mean minimal that the authors said was okay. i.e. the version they actually tested. The author of the root module is always free to increase the minimum. It is just that the minimum isn't some arbitrary thing that changes over time when other people make releases.

[BraedonWooding]

My top three things are;

• No lockfiles
• KISS, I don't want to fight with the package manager, also should fully be integrated into build.zig no external programs.
• Allow a file to be on each dependency to allow that dependency to go and download its own dependencies, this file should also maintain a change-log.

A good package manager can break/make a language, one of the reasons why Go has ditched atleast one of its official package managers and completely redid it (it may even be two, I haven't kept up to date with that scene).

[andrewrk]

The first thing I'm going to explore is a decentralized solution. For example, this is what package dependencies might look like:

const Builder = @import("std").build.Builder;
const builtin = @import("builtin");

pub fn build(b: &Builder) void {
    const mode = b.standardReleaseOptions();

    var exe = b.addExecutable("tetris", "src/main.zig");
    exe.setBuildMode(mode);

    exe.addGitPackage("clap", "https://github.com/Hejsil/zig-clap",
        "0.2.0", "76c50794004b5300a620ed71ef58e4444455fd72e7f7e8f70b7d930a040210ff");
    exe.addUrlPackage("png", "http://example.com/zig-png.tar.gz",
        "00e27a29ead4267e3de8111fcaa59b132d0533cdfdbdddf4b0604279acbcf4f4");

    b.default_step.dependOn(&exe.step);
}

Here we provide a mapping of a name and a way for zig to download or otherwise acquire the source files of the package to depend on.

Since the build system is declarative, zig can run it and query the set of build artifacts and their dependencies, and then fetch them in parallel.

Dependencies are even stricter than version locking - they are source-locked. In both examples we provide a SHA-256 hash, so that even a compromised third party provider cannot compromise your build.

When you depend on a package, you trust it. It will run zig build on the dependency to recursively find all of its dependencies, and so on. However, by providing a hash, you trust only the version you intend to; if the author updates the code and you want the updates, you will have to update the hash and potentially the URL.

Running zig build on dependencies is desirable because it provides a package the ability to query the system, depend on installed system libraries, and potentially run the C/C++ compiler. This would allow us to create Zig package wrappers for C projects, such as ffmpeg. You would even potentially use this feature for a purely C project - a build tool that downloads and builds dependencies for you.

[ghost]

and potentially run the C/C++ compiler
cmd/go: arbitrary code execution during “go get” #23672

although you might argue

Dependencies are even stricter than version locking - they are source-locked. In both examples we provide a SHA-256 hash, so that even a compromised third party provider cannot compromise your build.

in that case you'd have to check all the reps of all your reps recursively (manually?) on each shape change though to be really sure

[andrewrk]

in that case you'd have to check all the reps of all your reps recursively (manually?) on each shape change though to be really sure

This is already true about all software dependencies.

[BraedonWooding]

I've been considering how one could do this for the past few days, here is what I generally came up with (this is based off @andrewrk 's idea), I've kept out hashes to make it easier, I'm more talking about architecture then implementation details here;

• No lock files, purely source driven
• Have a central repository of all downloaded files (like under /usr/local/zig-vendor and have a built in to access them like @vImport("BraedonWooding/ZigJSON"), or have a unique vendor location for each 'zig' build file or rather each project, in which case we autogenerate a nice index.zig file for you to access like const vendor = @import("vendor/index.zig"); const json = vendor.ZigJSON.
• Utilise them during building, we can go and download any new dependencies.
• Automatically download the latest dependency as per the user requirements that is either (for structure x.y.z);
  • Keep major version i.e. upgrade y and z but not x
  • Keep minor and major i.e. upgrade all 'z' changes.
  • Keep version explicitly stated
  • Always upgrade to latest

This would also solve the issue of security fixes as most users would keep the second option which is intended for small bug fixes that don't introduce any new things, whereas the major version is for breaking changes and the minor is for new changes that are typically non-breaking.

Your build file would have something like this in your 'build' function;

...
builder.addDependency(builder.Dependency.Git, "github.com.au", "BraedonWooding", "ZigJSON", builder.Versions.NonMajor);
// Or maybe
builder.addDependency(builder.Dependency.Git, "github.com.au/BraedonWooding/ZigJSON", builder.Versions.NonMajor);
// Or just
builder.addGitDependency("github.com.au/BraedonWooding/ZigJSON", builder.Versions.NonMajor);
...

Keeping in mind that svn and mercurial (as well as plenty more) are also used quite a bit :). We could either use just a folder system of naming to detect what we have downloaded, or have a simple file storing information about all the files downloaded (note: NOT a lock file, just a file with information on what things have been downloaded). Would use tags to determine versions but could also have a simple central repository of versions linking to locations like I believe what other things have.

[isaachier]

How would you handle multiple definitions of the same function? I find this to be the most difficult part of C/C++ package management. Or does Zig use some sort of package name prefixing?

[BraedonWooding]

@isaachier Well you can't have multiple definitions of a function in Zig, function overloads aren't a thing (intended).

You would import a package like;

const Json = @Import("JSON/index.zig");

fn main() void {
    Json.parse(...);
    // And whatever
}

When you 'include' things in your source Zig file they are exist under a variable kinda like a namespace (but simpler), this means that you should generally never run into multiple definitions :). If you want to 'use' an import like using in C++ you can do something like use Json; which will let you use the contents without having to refer to Json for example in the above example it would just be parse(...) instead of Json.parse(...) if you used use, you still can't use private functions however.

If for some reason you 'use' two 'libraries' that have a dual function definition you'll get an error and will most likely have to put one under a namespace/variable, very rarely should you use use :).

[isaachier]

I don't expect a clash in the language necessarily, but in the linker aren't there duplicate definitions for parse if multiple packages define it? Or is it automatically made into Json_parse?

[Hejsil]

@isaachier If you don't define your functions as export fn a() void, then Zig is allowed to rename the functions to avoid collisions.

[isaachier]

OK that makes sense. About package managers, I'm sure I'm dealing with experts here 😄, but wanted to make sure a few points are addressed for completeness.

• Should multiple versions of the same library be allowed? This can occur when library A relies on libraries B and C. A needs C version 2 and B needs C version 1. How do you handle that scenario? I'm not sure about the symbol exports, but that might be an issue if you intend to link in both versions.
• Are the packages downloaded independently for each project or cached on the local disk (like maven and Hunter). In the latter case, you have to consider the use of build flags and their effect on the shared build.

[andrewrk]

These are important questions.

The first question brings up an even more fundamental question which we have to ask ourselves if we go down the decentralized package route: how do you even know that a given package is the same one as another version?

For example, if FancyPantsJson library is mirrored on GitHub and BitBucket, and you have this:

// in main package
exe.addGitPackage("fancypantsjson", "https://github.com/mrfancypants/zig-fancypantsjson",
    "1.0.1", "76c50794004b5300a620ed71ef58e4444455fd72e7f7e8f70b7d930a040210ff");

// in a nested package
exe.addGitPackage("fancypantsjson", "https://bitbucket.org/mirrors-r-us/zig-fancypants.git",
    "1.0.1", "76c50794004b5300a620ed71ef58e4444455fd72e7f7e8f70b7d930a040210ff");

Here, we know that the library is the same because the sha-256 matches, and that means we can use the same code for both dependencies. However, consider if one was on a slightly newer version:

// in main package
exe.addGitPackage("fancypantsjson", "https://github.com/mrfancypants/zig-fancypantsjson",
    "1.0.2", "dea956b9f5f44e38342ee1dff85fb5fc8c7a604a7143521f3130a6337ed90708");

// in a nested package
exe.addGitPackage("fancypantsjson", "https://bitbucket.org/mirrors-r-us/zig-fancypants.git",
    "1.0.1", "76c50794004b5300a620ed71ef58e4444455fd72e7f7e8f70b7d930a040210ff");

Because this is decentralized, the name "fancypantsjson" does not uniquely identify the package. It's just a name mapped to code so that you can do @import("fancypantsjson") inside the package that depends on it.

But we want to know if this situation occurs. Here's my proposal for how this will work:

comptime {
    // these are random bytes to uniquely identify this package
    // developers compute these once when they create a new package and then
    // never change it
    const package_id = "\xfb\xaf\x7f\x45\x86\x08\x10\xec\xdb\x3c\xea\xb4\xb3\x66\xf9\x47";

    const package_info = @declarePackage(package_id, builtin.SemVer {
        .major = 1,
        .minor = 0,
        .revision = 1,
    });

    // these are the other packages that were not even analyzed because they
    // called @declarePackage with an older, but API-compatible version number.
    for (package_info.superseded) |ver| {
        @compileLog("using 1.0.1 instead of", ver.major, ver.minor, ver.revision);
    }

    // these are the other packages that have matching package ids, but
    // will additionally be compiled in because they do not have compatible
    // APIs according to semver
    for (package_info.coexisting) |pkg| {
        @compileLog("in addition to 1.0.1 this version is present",
            pkg.sem_ver.major, pkg.sem_ver.minor, pkg.sem_ver.revision);
    }
}

The prototype of this function would be:

// thes structs declared in @import("builtin");
pub const SemVer = struct {
    major: @typeOf(1),
    minor: @typeOf(1),
    revision: @typeOf(1),
};
const Namespace = @typeOf(this);
pub const Package = struct {
    namespace: Namespace,
    sem_ver: SemVer,
};
pub const PackageUsage = struct {
    /// This is the list of packages that have declared an older,
    /// but API-compatible version number. So zig stopped analyzing
    /// these versions when it hit the @declarePackage.
    superseded: []SemVer,

    /// This is the list of packages that share a package id, but
    /// due to incompatible versions, will coexist with this version.
    coexisting: []Package,
};

@declarePackage(comptime package_id: [16]u8, comptime version: &const SemVer) PackageUsage

Packages would be free to omit a package declaration. In this case, multiple copies of the
package would always coexist, and zig package manager would be providing no more than
automatic downloading of a resource, verification of its checksum, and caching.

Multiple package declarations would be a compile error, as well as @declarePackage somewhere
other than the first Top Level Declaration in a Namespace.

Let us consider for a moment, that one programmer could use someone else's package id, and then
use a minor version greater than the existing one. Via indirect dependency, they could "hijack"
the other package because theirs would supersede it.

At first this may seem like a problem, but consider:

• Most importantly, when a Zig programmer adds a dependency on a package and verifies the checksum,
  they are trusting that version of that package. So the hijacker has been approved.
• If the hijacker provides a compatible API, they are intentionally trying to create a drop-in replacement
  of the package, which may be a reasonable thing to do in some cases. An open source maintainer
  can essentially take over maintenance of an abandoned project, without permission, and offer
  an upgrade path to downstream users as simple as changing their dependency URL (and checksum).
• In practice, if they fail to provide a compatible API, runtime errors and compile time errors
  will point back to the hijacker's code, not the superseded code.

Really, I think this is a benefit of a decentralized approach.

Going back to the API of @declarePackage, here's an example of power this proposal gives you:

const encoding_table = blk: {
    const package_id = "\xfb\xaf\x7f\x45\x86\x08\x10\xec\xdb\x3c\xea\xb4\xb3\x66\xf9\x47";

    const package_info = @declarePackage(package_id, builtin.SemVer {
        .major = 2,
        .minor = 0,
        .revision = 0,
    });

    for (package_info.coexisting) |pkg| {
        if (pkg.sem_ver.major == 1) {
            break :blk pkg.namespace.FLAC_ENCODING_TABLE;
        }
    }

    break :blk @import("flac.zig").ENCODING_TABLE;
};

// ...

pub fn lookup(i: usize) u32 {
    return encoding_table[i];
}

Here, even though we have bumped the major version of this package from 1 to 2, we know that the FLAC ENCODING TABLE is unchanged, and perhaps it is 32 MB of data, so best to not duplicate it unnecessarily. Now even versions 1 and 2 which coexist, at least share this table.

You could also use this to do something such as:

if (package_info.coexisting.len != 0) {
    @compileError("this package does not support coexisting with other versions of itself");
}

And then users would be forced to upgrade some of their dependencies until they could all agree on a compatible version.

However for this particular use case it would be usually recommended to not do this, since there would be a general Zig command line option to make all coexisting libraries a compile error, for those who want a squeaky clean dependency chain. ReleaseSmall would probably turn this flag on by default.

---

As for your second question,

Are the packages downloaded independently for each project or cached on the local disk (like maven and Hunter). In the latter case, you have to consider the use of build flags and their effect on the shared build.

Package caching will happen like this:

• Caching of the source download (e.g. this .tar.gz has this sha-256, so we can skip downloading it if we already have it)
• The same binary caching strategy that zig uses for every file in every project

Caching is an important topic in the near future of zig, but it does not yet exist in any form. Rest assured that we will not get caching wrong. My goal is: 0 bugs filed in the lifetime of zig's existence where the cause was a false positive cache usage.

[andrewrk]

One more note I want to make:

In the example above I have:

exe.addGitPackage("fancypantsjson", "https://github.com/mrfancypants/zig-fancypantsjson",
    "1.0.2", "dea956b9f5f44e38342ee1dff85fb5fc8c7a604a7143521f3130a6337ed90708");

Note however that the "1.0.2" only tells Zig how to download from a git repository ("download the commit referenced by1.0.2"). The actual version you are depending on is the one that is set with @declarePackage in the code that matches the SHA-256.

So the package dependency can be satisfied by any semver-compatible version indirectly or directly depended on.

With that in mind, this decentralized strategy with @declarePackage even works if you do any of the following things:

• use a git submodule for the package
  • you can use addDirPackage("name", "/path/to/dir", "a3951217c609a5a9c5a100e5f3c37a4e8b14796642138ee613db46daca7d43c7").
• just copy+paste the package into your own source code
  • same thing, use addDirPackage

You can also force your dependency's dependency's dependency (and so on) to upgrade, simply by adding a direct dependency on the same package id with a minor or revision bump.

And to top it off you can purposefully inject code into your dependency's dependency's dependency (and so on), by:

• forking or otherwise copy pasting the package in question
• bumping the minor version
• adding a direct dependency on your fork
• do your code edits in the fork

This strategy could be used, for example, to add @optimizeFor(.Debug) in some tricky areas you're trying to troubleshoot in a third party library, or perhaps you found a bottleneck in a third party library and you want to add @optimizeFor(.ReleaseFast) to disable safety in the bottleneck. Or maybe you want to apply a patch while you're waiting for upstream to review and accept it, or a patch that will be coming out in the next version but isn't released yet.

[andrewrk]

Another note: this proposal does not actually depend on the self hosted compiler. There is nothing big blocking us from starting to implement it. It looks like:

• Implementation of @declarePackage in the c++ compiler. At this point you could test package management with the CLI.
• Add the API to zig build system, e.g. addDirPackage. At this point we have a working package manager that you could use with git submodule or copy+pasting the package you want to depend on into your codebase. (But read on - this can be further improved)
• Networking in the standard library. Probably best to use coroutines and async I/O for this. Related: http server in the standard library #910. I also need to document coroutines (basic documentation for everything #367)
• Add support for .tar.gz .tar.xz, .zip, etc to standard library
• Now we can implement addUrlPackage
• Add support for downloading a particular revision via git / svn
• Now we can implement addGitPackage / addSvnPackage

[clownpriest]

maybe worth considering p2p distribution and content addressing with ipfs?

see https://github.com/whyrusleeping/gx for example

just a thought

[costincaraivan]

One important thing to note, especially for adoption by larger organization: think about a packaging format and a repo structure that is proxy/caching/mirroring friendly and that also allows an offline mode.

That way the organization can easily centralize their dependencies instead of having everyone going everywhere on the internet (a big no-no for places such as banks).

Play around a bit with Maven and Artifactory/Nexus if you haven't already 😉

[andrewrk]

The decentralized proposal I made above is especially friendly to p2p distribution, ipfs, offline modes, mirroring, and all that stuff. The sha-256 hash ensures that software is built according to expectations, and the matter of where to fetch the resources can be provided by any number of "plugins" for how to download something:

• http (don't even need https because of the sha-256)
• git
• svn
• torrent
• ipfs
• nfs
• some custom thing, it doesn't matter, as long as the bytes can be delivered

[costincaraivan]

Looks good but I'd have to try it out in practice before I can say for sure 😄

I'd have one suggestion: for naming purposes, maybe it would be a good idea to also have a "group" or "groupId" concept?

In many situations it's useful to see the umbrella organization from which the dependency comes. Made up Java examples:

1. group: org.apache, name: httpclient.
2. group: org.apache, name: regexutils.

Otherwise what happens is that people basically overload the name to include the group, everyone in their own way (apache-httpclient, regexutils-apache). Or they just don't include it and you end up with super generic names (httpclient).

It also prevents or minimizes "name squatting". I.e. the first comers get the best names and then they abandon them...

[isaachier]

Structs provide the encapsulation you are looking for @costincaralvan. They seem to act as namespaces would in C++.

[demircancelebi]

I agree with @costincaraivan. npm has scoped packages for example: https://docs.npmjs.com/getting-started/scoped-packages.

In addition to minimizing name squatting and its practical usefulness (being able to more easily depend on a package if it is coming from an established organization or a well-known developer), honoring the creators of a package besides their creation sounds more respectful in general, and may incentivize people to publish more of their stuff :).

On the other hand, generic package names also come in handy because there is one less thing to remember when installing them.

[costincaraivan]

I didn't want to clutter the issue anymore but just today I bumped into something which is in my opinion relevant for the part I posted about groups (or scoped packages in NPM parlance):

http://bitprophet.org/blog/2012/06/07/on-vendorizing/

Look at their dilemma regarding the options, one of the solutions is forking the library:

Fork and release our own package on PyPI as e.g. fluidity-invoke.

• This works, but has many the drawbacks of the vendorizing option and offers few of the benefits.
• It also confuses things re: project ownership and who should receive/act on bug reports. Users new to the space might focus on your fork instead of upstream, forcing you to either handle their problems, or redirect them.

This would be easily solvable with another bit of metadata, the group. In Java world their issue would be solved by forking the library and then publishing it under the new group. Because of the group it's immediately obvious that the library was forked. Even easier to figure out in a repository browser of sorts since the original version would have presumably many versions while the fork will probably have 1 or 2.

[ElectricCoffee]

Small detail here 😊, but what about changing the wording to release.version.patch instead of major.minor.patch, I think it would be easy to grasp for beginners, and I kinda like it better

That makes it inconsistent with the terminology on https://semver.org/

[uael]

I listened to the recent conf at Milan and wanted to share some thoughts about the package manager milestone.

In short: what about nix flakes?
In more details:

• having a package manager for a language is likely mandatory today, but using an in house one sounds more like a trend.
  Back in the days there was no other suitable solutions than developing a language specific one, today nix flakes seems to feel that hole pretty well. It sound like a more modern solution.
• zig is a modern language that fixes old systems languages, nix a modern package manager that fixes packaging. Each of them do their job very well, why reinventing the wheel ?
• integrate zig into nix is very easy compared to the other way around, we can have a package manager for zig in a few days, and a very good one!
• I would personally love to nix develop without any additional step required 😏

[fogti]

@uael although integration with nix might be a good idea (I use NixOS as my primary linux distro, preferred over Gentoo, Debian and Fedora), keep in mind that afaik primary reasons for more / language-specific package managers are:

1. system package manager only/mostly has outdated packages (happens afaik with e.g. homebrew, debian/ apt, etc.)
2. it targets microsoft windows (which doesn't has an established system package manager, and writing package managers for that platform is harder afaik because default file system options (e.g. default file open options cause most files to be locked, unable to be concurrently modified, running exeutables can't be deleted (they usually can be on unix), which makes upgrading stuff while the system is running relatively hard))

(1) gets solved by nix. ok (2) doesn't get solved by nix because nix has no windows support, and the file system interface and container interface is different enough from linux/unix that it would be a pretty big amount of work. (it was already tried, but afaik there was no obvious way forward) see also

For (2) it might be a good idea to consider integration with vcpkg and winget, but as I usually don't develop on windows, especially due to the aforementioned annoyances, I don't have a clear idea how that could work.

[adamcstephens]

Adding nix as a dependency for zig would be bad for multiple reasons, in my opinion:

1. Requiring nix would severely limit where zig programs could be built. As @zseri notes it is not supported everywhere. Even on non-NixOS Linux and Mac systems where it is possible to install nix it is not widely used. Installing nix on those systems is mildly invasive and would be a non-starter in any controlled environment.
2. Packaging zig programs for other distributions will likely be precluded as adding a nix build dependency would not be acceptable to debian and the others.
3. Nix is another language a user would have to learn in order to be proficient with such a packaging system. This would increase the learning curve of zig without the benefit of being used to actually write zig programs. While this is possibly true of any package manager syntax, I'd argue the nix language's complexity and uniqueness far outweighs the benefits it offers here.

I think it would be a great idea to plan for nix integration or support with any new package manager. Unfortunately, as just a lowly user I don't think it would be prudent to use nix as the zig package manager.

[xmnlab]

I think it would be a great idea to plan for nix integration or support with any new package manager. Unfortunately, as just a lowly user I don't think it would be prudent to use nix as the zig package manager.

that makes sense to me. maybe the same would apply to conda. currently we can use conda for packaging, as zig is already available on conda-forge .. also it could be possible to have an specific conda channel just for zig ...

but I guess that something that covers all needs for zig would be the best approach, but maybe it would take more time to have it ready for usage ...

is there any action in progress about this? deadlines? plan? I would love to be envolved in this in order to learn more about the topic and contribute.

[AndersonTorres]

In short: what about nix flakes? In more details:

Oh no no no. Bad idea, trust me.
(Disclaimer: yes I am a Nix user.)

The most obvious is sovereignty: Zig will become dependent on a third party to provide this service of package management. If Nix changes anything in its policy, from dropping a system approach to drop a whole platform (e.g. Nix is not made to run in Windows), Zig will automatically suffer the same.

Certainly, it would be fun to employ the approaches and algorithms from Nix project in order to provide the same or a similar experience on Zig, however making Nix a part of Zig is not a good idea.

The second one is consistency: Zig is planning to release a (mostly) LLVM-free experience in the future. It would be insane to become dependent on another project to do packaging stuff.

The third is that Nix, in and by itself, proposes to solve a larger problem than those smaller language-restricted package managers.

using an in house one sounds more like a trend.

Zig needs to be bootstrappable. Using an external package manager makes it harder.

zig is a modern language that fixes old systems languages, nix a modern package manager that fixes packaging. Each of them do their job very well, why reinventing the wheel ?

Because sometimes we need better wheels made of different materials.

I would personally love to nix develop without any additional step required

Nah. This is just a flake.nix distance away.

I would prefer to rewrite Nix in Zig instead of the reverse.

[motiejus]

What if, besides doing the package-manager things, the Zig package manager could help us vendor our dependencies to protect ourselves from things disappearing off the internet?

I chatted with Andrew in Milan about this (to be clear, the idea-at-the-time was dismissed after about 5 seconds). I thought about it more and wrote down: https://jakstys.lt/2022/smart-bundling/

[moosichu]

I don't think you want to tie a package manager to any specific version control software (downstream) - but, being able to have the package manager just work with you checking your dependencies into whatever vcs system you are using is really important (imo).

Being able to have the confidence of being able to grab a zipped up copy of a clean working directory of your project and knowing it will just build with the correct corresponding version of the zig compiler is crucial.

It's how I currently work with zig libs that I use (copy & paste them into a libs folder in my repo), if a package manager is just a simple tool that makes it easier to control the upgrade process for that kind of thing- that would be the dream. Although I do understand there are use cases where you would not want to do that.

[AndersonTorres]

What if, besides doing the package-manager things, the Zig package manager could help us vendor our dependencies to protect ourselves from things disappearing off the internet?

Well, sometimes I pick some projects in the wild and put them on Museoa

However it looks like a Software Heritage job

[sam0x17]

It's how I currently work with zig libs that I use (copy & paste them into a libs folder in my repo), if a package manager is just a simple tool that makes it easier to control the upgrade process for that kind of thing- that would be the dream. Although I do understand there are use cases where you would not want to do that.

Yes, sane opinionated defaults with full ability to customize is the best philsophy for this stuff imo

[kuon]

I'll add my thought on:

Enforcing Semver is an interesting concept, and projects like Elm have done it.

I used Elm extensively and I think this is the most appreciable feature concerning maintainability. But for zig it is a bit harder to define what would be considered a stable API. For example, consider this:

Library code:

const RTCConfig = struct { wakup_interval: u16 };

pub fn init_rtc(config: RTCConfig) void {
}

User code:

init_rtc(.{.wakeup_interval: 3})

Now a field is added to RTCConfig, like so:

const RTCConfig = struct { wakup_interval: u16, alarm_interval: u16 = 0};

Current zig code would need no change to work with the new library, and the behavior would not change either, so for me this could be a nudge from 1.2.3 to 1.2.4 or 1.2.3 to 1.4.0 if the option does impact behavior in significant way. But Elm policy would force a bump from 1.2.3 to 2.0.0. Which I would find way too drastic in this case.

I don't have the solution (doc attribute to mark as API...), and I am not saying that zig should implement it. But I want to emphasis how well it works for Elm and how incredible to never have a build break because of an upgrade, it is the absolute opposite of the JS experience.

It could also be an external tool.

[cweagans]

But Elm policy would force a bump from 1.2.3 to 2.0.0. Which I would find way too drastic in this case.

Why is it too drastic? It's exactly the point of semver: you can't upgrade to the next version without making changes to your calling code.

[leecannon]

@cweagans But the point being made is no calling code would need to change, in the example given its an API change that is 100% backwards compatible.

This is a consequence of zig being a source first language as the old calling code will be recompiled with the new library and as all previous usages of the API are still valid nothing breaks.

[kuon]

@cweagans But the point being made is no calling code would need to change, in the example given its an API change that is 100% backwards compatible.

Yes, and imagine the case where you fixed a critical security vulnerability in your lib and you clearly want a patch release, but because you added a padding field in a struct for the fix (or anything of the sort), the package manager forces a big version bump.

The difference with elm is that normally elm has no security implication because it runs in a sandbox, this makes the above scenario rare or even impossible (you'll have to fix the browsers).

Again, I don't know what should zig do, but I think, even with my comments on how different it is from elm, we should consider some sort of mechanism to check source compatibility.

The scenario that should be avoided is to upgrade dependencies and having to rewrite large chunk of codes and re-read documentation while doing so (hindering code re-use).

But, on the other hand, apps should be "source upgradable" easily, for example, imagine a library does a major rewrite to move from xorg to wayland, that would be clearly be a x.y.z to x+1.0.0, but, because abstraction was properly done, it is source compatible. The package manager should say "I kept XXX back, but it is source compatible and would still compile, are you happy to go to new version? The changelog is: print CHANGELOG".

[HugoFlorentino]

Dependency tracking could use a range, too.

[ifreund]

But for zig it is a bit harder to define what would be considered a stable API.

Related comment: #9909 (comment)

[kuon]

I had more thought about the semver problem, and here a few ideas:

• there should be an upgrade-auto command than when called, upgrade to the lastest "source compatible" version of the package, the upgrade can be between patch/minor/major version, as long as app code will compile after running the command.
• there should be a update command that only update package meta data, the most important feature of this is to detect packaged marked as insecure. For example, my app uses package SSL v1.1.0 which has been marked as insecure. After an update it should be a compile error to build against that package. Either upgrade the package or add a compile flag to force the compilation of an insecure package.
• finally, a upgrade-to-latest command which upgrade the packages to the latest version without considering source compatibility.

Source compatibility should be defined two ways. The first way is to download the new package version, try to build with it and if it does, then hurray, it's compatible. The second way is to ensure the new version has not been made "breaking". A "breaking" version is explicitly marked as incompatible, even if it is in fact source compatible. Both check should pass for upgrade-auto.

For me, the most important points are:

• security vulnerabilities in packages should be strong and visible
• upgrading to latest version should be easy and non breaking
• semver is not really relevant as it might be in fact more related to project management, even marketing, than technical compatibility

[sam0x17]

One thing to watch out for with that approach, I think it very quickly becomes an n^2 problem when you have N packages that need to be updated and any one of them could have an incompatibility with any other upon upgrading.

…

On Jul 19, 2022, at 11:42 AM, Nicolas Goy ***@***.***> wrote: I had more thought about the semver problem, and here a few ideas: there should be an upgrade-auto command than when called, upgrade to the lastest "source compatible" version of the package, the upgrade can be between patch/minor/major version, as long as app code will compile after running the command. there should be a update command that only update package meta data, the most important feature of this is to detect packaged marked as insecure. For example, my app uses package SSL v1.1.0 which has been marked as insecure. After an update it should be a compile error to build against that package. Either upgrade the package or add a compile flag to force the compilation of an insecure package. finally, a upgrade-to-latest command which upgrade the packages to the latest version without considering source compatibility. Source compatibility should be defined two ways. The first way is to download the new package version, try to build with it and if it does, then hurray, it's compatible. The second way is to ensure the new version has not been made "breaking". A "breaking" version is explicitly marked as incompatible, even if it is in fact source compatible. Both check should pass for upgrade-auto. For me, the most important points are: security vulnerabilities in packages should be strong and visible upgrading to latest version should be easy and non breaking semver is not really relevant as it might be in fact more related to project management, even marketing, than technical compatibility — Reply to this email directly, view it on GitHub <#943 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE4LJNNFPLHIXYOIDENY3VU3EGTANCNFSM4E34JUDQ>. You are receiving this because you were mentioned.

[kuon]

One thing to watch out for with that approach, I think it very quickly becomes an n^2 problem when you have N packages that need to be updated and any one of them could have an incompatibility with any other upon upgrading.

Yes you are right, but even if the feature would be limited to "upgrade all packages or nothing" it would still be very useful in many situations.

Also, there should be an incentive for library author to avoid breaking source compatibility.

[sam0x17]

Yes I totally agree it’s still something we should do. Just maybe something we should do using multiple threads and all CPU cores ;)

…

On Jul 19, 2022, at 11:58 AM, Nicolas Goy ***@***.***> wrote: One thing to watch out for with that approach, I think it very quickly becomes an n^2 problem when you have N packages that need to be updated and any one of them could have an incompatibility with any other upon upgrading. Yes you are right, but even if the feature would be limited to "upgrade all packages or nothing" it would still be very useful in many situations. Also, there should be an incentive for library author to avoid breaking source compatibility. — Reply to this email directly, view it on GitHub <#943 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE4LPYDZHS4JKTZS2AYIDVU3GDFANCNFSM4E34JUDQ>. You are receiving this because you were mentioned.

[PaperPrototype]

Something that is overlooked is the design of the package manager site itself. I am curious what yall think would be nice to have in the site?

On a side note. I think it would be cool if you could connect to GitHub and select a specific commit and save that as a new version?

[PaperPrototype]

I have GitHub connected to my site and I select a commit to make new versions of a course. Then I use the GitHub api to query the repo at that commit and get the markdown check it out https://sparker3d.com

[andrewrk]

Alright, I stopped reading this issue a long time ago. I'm going to lock the discussion. The next step will be for the Zig core team members to create a proof of concept, which we as a community can then use and discuss with more specific, targeted proposals to change specific things about status quo. That will happen in a few months from now.