Write meta attribute as JSON into the derivation

[domenkozar]

This allows to generate SBOMs and filter derivations based on license without relying on evaluation time information.

The downside is that changing meta attributes triggers a rebuild (which shouldn't have much impact with CA derivations).

Another downside is disk space usage, for example pkgs.git closure of .drv files goes from 3.8MB to 4.4MB.

This is part of a larger effort to bring security vulnerability notifications to https://cachix.org and anyone else that wants to have the automation.

[shlevy]

which shouldn't have much impact with CA derivations

Aren't CA derivations still an experimental feature? We can't rely on them like this.

Metadata changes should not cause rebuilds and should not be part of the output. That's not metadata, that's data. If the maintainers of glibc change we shouldn't rebuild the world. The proper fix here is for Nix to have some mechanism for attaching metadata to store objects that doesn't impact the hash algorithm.

[domenkozar]

Aren't CA derivations still an experimental feature? We can't rely on them like this.

They are of course, I meant that long term we have a solution for this issue so it's less impactful than it may seem.

Metadata changes should not cause rebuilds and should not be part of the output.

The way I see it is that CA derivations are exactly the idea behind this, trivial changes shouldn't impact the whole recompilation of the tree.

Metadata changes should not cause rebuilds and should not be part of the output. That's not metadata, that's data. If the maintainers of glibc change we shouldn't rebuild the world.

There are two conflicting shoulds:

1. we should have good automation for security vulnerabilities
2. metadata shoudn't impact rebuilds

I do think (2) brings so many good things that we can delay (1) should until CA derivations are stable.

[shlevy]

If this feature is important, implement it properly in Nix. No need to wait for CA derivations.

[domenkozar]

If this feature is important, implement it properly in Nix. No need to wait for CA derivations.

I like to split improvements into three phases:

1. make it work (here comes the benefit)
2. make it correct (fix the bugs)
3. make it fast (CA derivations / Nix allowing derivation inputs not impacting hashing)

[shlevy]

You're "making it work" by breaking the semantics of an existing working system for everyone. You're not building some prototype of some opt-in functionality. Again, why not do your step 1 by implementing a prototype of a metadata feature in Nix? In 30 seconds of thought here's an approach you could probably put together in a day or two:

1. Add a field to the db for metadata, just an arbitrary string optionally associated with any store path
2. Add a new builtin configurableDerivationStrict that takes a set containing { meta, drvArgs }, passes drvArgs to derivationStrict, and attaches meta to the .drv storepath
3. Have mkDerivation use configurableDerivationStrict if Nix supports it and there is metadata.

[domenkozar]

You're "making it work" by breaking the semantics of an existing working system for everyone. You're not building some prototype of some opt-in functionality. Again, why not do your step 1 by implementing a prototype of a metadata feature in Nix? In 30 seconds of thought here's an approach you could probably put together in a day or two:

1. Add a field to the db for metadata, just an arbitrary string optionally associated with any store path
2. Add a new builtin configurableDerivationStrict that takes a set containing { meta, drvArgs }, passes drvArgs to derivationStrict, and attaches meta to the .drv storepath
3. Have mkDerivation use configurableDerivationStrict if Nix supports it and there is metadata.

I'm not sure creating another type of derivation is the right approach here, given that this is a solved problem already as experimental feature in Nix for ~2 years.

Note that your suggestion breaks most likely the semantics of .drv files if I understand it correctly, for example for existing parsers.

[shlevy]

I'm not sure creating another type of derivation is the right approach here

I'm not creating another type of derivation. I'm suggesting simply a wrapper primop that adds a new field to the Nix db.

given that this is a solved problem already as experimental feature in Nix for ~2 years.

Do CA derivations work? Are they ready to be enabled by default? Are they stable? If so, then they should be unexperimental. If not, they cannot be relied upon. Even if they were non-experimental, they would not be a proper solution here: nothing should rebuild if you change metadata.

Note that your suggestion breaks most likely the semantics of .drv files if I understand it correctly, for example for existing parsers.

No, no data is stored in the drv. Just in the db associated with the drv path.

[YorikSar]

I think that this pulls in too much information into the build. While license is a part of sources, and license change wouldn't happen without needing a rebuild anyway, other pieces of information like current maintainer or level of brokenness on different platforms shouldn't affect the package.

Maybe we should tie license to the sources of the package and store that in derivations instead? Or maybe even keep it in "source" derivations and not in "package" derivations.

I'm not sure how this helps security vulnerability checks though. If we find a vulnerability in the package, we add it in meta and currently it doesn't trigger package rebuild, so you can compare store path and check if the package that you already have is now vulnerable. If you add this information into derivation, without CA you'll get a new output path and you won't be able to check it on already deployed systems.

[shlevy]

Or maybe even keep it in "source" derivations and not in "package" derivations.

Given the package drv, we can almost always identify the src, unpack it, and scan the result for licenses. No new drv needed, probably gets you 95% coverage with a simple script. Seems like a perfect fit for @domenkozar's "make it work" stage.

[YorikSar]

identify the src, unpack it, and scan the result for licenses

I think this is too much effort with a bunch of corner cases, so “caching” the result of such action is a good thing. I also understand the need for having this cached not just in code, but in derivations as often you don’t have sources for drvs that produced these outputs, but drvs themselves are quite easy to keep around with keep-derivations.

[flokli]

identify the src, unpack it, and scan the result for licenses

I think this is too much effort with a bunch of corner cases, so “caching” the result of such action is a good thing.

That's exactly what IFD can already do today. Just because it doesn't fly with Hydra and its scheduler doesn't mean it's a bad idea - meta isn't recursed AFAIK anyways. I still would very much prefer if we specified this stuff manually, rather than blindly trusting a script to gather the right data.

[domenkozar]

I've changed the implementation to be opt-in, so that we can develop the feature until it's ready to be on by default.

[SuperSandro2000]

Do CA derivations work? Are they ready to be enabled by default?

No, they are far from ready. At the moment you can't reliable convert a already built system output because each other referencing outputs create deadlocks. In the past multiple such deadlocks where already fixed but I can easily imagine that there are more to be found and edge cases to be fixed.
Also hydra support is not existing and the open PR for it is stale and I couldn't get it work.

[roberth]

This could be handled by improving the string context mechanism. That way we don't "pollute" the drv files.

• More context in context string nix#4677

[blaggacao]

After Domen posted this PR in #slsa:nixos.org a quick exploratory discussion ensued between @RaitoBezarius and myself the outcome of which I want to resume here:

• More than a question of metadata storage, it can be considered a question of cache storage (since the SBOM data will be reproducible)

• That renders the question addressed by this PR one of a cache storage implementation

• .drv for the reasons mentioned above and also from its principle design as "build recipes", may be a very unsuitable implementation of such a cache

• other options to implement such a cache would be:

  • the local cache in the sql database, as proposed by @shlevy
  • a remote cache, i.e. sbom.cachix.org / sbom.nixos.org

Using .drv as a cache implementation still seemed unsuitable to us even if using it could lead to a quick "make it work" improvement of ecosystem outcomes by potentially improving overall security.

The reasons it seemed unsuitable even under this angle mostly stems from:

• .drv was not designed to be a cache layer / metadata transport layer even if placing something into a drv has all the effects of a cache via Nix' build cache
• this may set a precedent that can lead the community in the wrong architectural direction from which it may be difficult to recover. Related questions may be asked, such as: Do we have the governance in place to follow up and correct the implementation and ensure proper migration later? We answered that question such that we remained in favor of not setting that precedent. We did not validate if similar precedents already exist, though.
• an independent (metadata) cache layer implementation may emerge in a sandbox (whether remote or in sqlight) to validate all assumptions and use cases and a later settling inside Nix or Nixpkgs is a reasonable alternative path forward

I hope I did capture the outcomes appropriatly and that our findings are well received.

[lheckemann]

NixOS/nix#8080 is somewhat related; the approach currently implemented there only applies metadata from build outputs, but I could see a future where tags can come from evaluation-time info that doesn't land in the derivations.

[Ericson2314]

Can't we just have mkDerivation in Nixpkgs write another file with the metadata which refers to the derivation? No Nix changes needed then.

[emilylange]

I'll mark this as draft, as it's clear this shouldn't be merged until multiple concerns are resolved, and I want to prevent this getting merged by someone by accident (well aware there is currently a merge conflict, that prevents it from being merged right now anyway).

[zimbatm]

I'm still thinking about this because it seems like such low-hanging fruit.

Storing the meta.license attribute would get us 90% there, create almost no rebuild past the initial one, and limit the size growth of the .drv. It seems such an obvious trade to me.

The proposed alternatives are interesting but require patching Nix, changing the binary cache protocol, or deploying new infrastructure. These ideas require one or two orders of magnitude more effort to get out there, and so far, I haven't seen anybody decide to tackle them.

This is the typical "perfect is the enemy of good" situation; everybody jumps in with their ideal opinions. Now, almost one year later, nothing happens. We could have benefited from the hacky solution already.

[infinisil]

Agreeing with you @zimbatm. I'd be fine with merging this in principle, but I'd like to see:

• Since this is clearly not something internal, we should have documentation and tests
• We should either mark this as experimental, to reserve the right to change it in the future, give some more thought on what the interface should be, and/or make sure the interface can be evolved in the future. In particular, it might make more sense to always give the licenses as a list of SPDX identifiers, instead of the somewhat arbitrary format of meta.license.

[roberth]

Please be aware the this is how bad design choices and complexity proliferate. Slippery slope and all that.
I'd much prefer that people work on solving root causes.

That said, a non-mass-rebuild solution such as NixOS/nix#10780 is not available, and Nixpkgs can make its own choices.

Let's make sure we mitigate the impact of this workaround and make sure we actually agree, specifically:

• @domenkozar, please only store the license. This is about SBOM and we probably still don't want the more frequent changes to maintainers, description, etc to have to go through staging.

• @alyssais, since you are the code owner of /lib/licenses.nix, your work will be affected, as more changes to package licensing and/or the general licensing infrastructure will have to go through staging. Are you ok with this?

[domenkozar]

I have more urgent matters to do, would be great if someone picks this up.

[alyssais]

• @alyssais, since you are the code owner of /lib/licenses.nix, your work will be affected, as more changes to package licensing and/or the general licensing infrastructure will have to go through staging. Are you ok with this?

That's fine by me. There's a lot to improve about licenses.nix, but that work isn't exactly happening fast, so it's fine for it to go through staging. I wouldn't want to commit to the format of licenses in drv files being stable though, unless all that's written is a SPDX identifier.