Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retain latest build deps #482

Open
roberth opened this issue Dec 26, 2022 · 3 comments
Open

Retain latest build deps #482

roberth opened this issue Dec 26, 2022 · 3 comments
Labels
enhancement New feature or request

Comments

@roberth
Copy link
Member

roberth commented Dec 26, 2022

Hercules CI matrix room:

@srid:
I thought of a cool feature. "Pin" the most recent successful HCI job, so that the automatic GC will garbage collect all but derivations built by those jobs.

Why would this be cool? Let's say I have been using HCI for https://github.com/srid/neuron - and neuron is now in maintenance mode. It receives no commits lately. But I still want people to make use my of Nix cache. If I run Nix's usual GC service to have it delete, say, everything older than 30d (mainly done to not run out of disk space left), it will surely delete all neuron derivations. Instead, with this HCI pinning feature, the build for the latest master branch commit will remain the Nix store because HCI pinned it.
This will allow me to use nix-serve-ng and the like to provide a cache server, without thinking about pushing to s3 or cachix (where you still have to worry about disk usage growth).
I call this "intelligent GC"

@roberth:
I was going to call it smart gc :D
My idea was to keep a list of roots, and more of a cache than a more "obligatory" archive, but what you're suggesting is much easier to implement

@roberth roberth added the enhancement New feature or request label Dec 26, 2022
@roberth
Copy link
Member Author

roberth commented Dec 26, 2022

Instead of the latest job, it should perhaps be the latest change in each attribute. (maybe that's what you meant? job is different in, say, hydra) That will work better for multi-package projects and commits that don't change any hashes (README updates, etc).

@srid
Copy link

srid commented Dec 26, 2022

There is no need to keep all attributes. For example, I might delete packages.foo down the line (it vanishes from master branch), so that can be GC'ed. https://github.com/srid/ema exposes attributes for two different GHC versions; if I delete one of them down the line (which I'll), I imagine that will free up lots of space.

Ultimately the goal is that running, say, nix run github:user/repo/master should pull everything from the cache (exposed from the Nix store of the HCI agent), without building anything locally, regardless of how old the HEAD commit of the master branch is. And everything else (not part of that build) should be allowed to be GC'ed.

@srid
Copy link

srid commented Dec 26, 2022

It might make sense to make this configurable to specify which branches should be pinned, eg:

{
  herculesCI.gcRoots.branches = 
    [ "master" "staging" "develop" "release/*" ]; # By default: [ "master" "main" ]
  herculesCI.gcRoots.attributes =
    [ "packages.*.default" ];  # By default: [ "*" ]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants