lib.sourceSubdirs: init

[alyssais]

Description of changes

Things done

This is really useful for using Nix in monorepo layouts, where you
might have a component build that needs access to multiple
subdirectories of the repository. Compared to settings "srcs", it's
more convenient, because it preserves the relative
structure (including parent directories), and it avoids unnecessary
rebuilds compared to not filtering for subdirectories.

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[infinisil]

Could you take a look at the file set combinators in #222981 (the best start is here and the reference it links to starting here)? That functionality is very widely applicable and composable. In particular it would be equivalent to

with lib.fileset;
unions [
  ./one/directory
  ./another/directory
]

Or if you want to also import it into the store:

with lib.fileset;
importToStore {
  entryPoint = ./.;
  fileset = unions [
    ./one/directory
    ./another/directory
  ];
}

It would be great to get feedback if this works for your use case. Or if you're seeing a problem with this interface.

[alyssais]

It would be great to get feedback if this works for your use case.

I was hoping you'd chime in with a better way! I tried it, but it doesn't look like it works:

error: the string 'gqwi6j6d4l9zs8svwhzwbxnglv3zwzl3-source' is not allowed to refer to a store path (such as '')

       … while evaluating anonymous lambda

       at /nix/store/2pplzbz7d5jmi4mnqmfi7lds5bz2z5hx-source/lib/filesystem.nix:40:6:

           39|     # Nix <2.14 compatibility shim
           40|     (path:
             |      ^
           41|       if ! pathExists path

       … from call site

       at /nix/store/2pplzbz7d5jmi4mnqmfi7lds5bz2z5hx-source/lib/fileset.nix:494:8:

          493|     in
          494|     if pathType entryPoint != "directory" then
             |        ^
          495|       # This would also be caught by the `may be influenced` condition further down, because of how files always set their containing directories as the base

       … while evaluating 'importToStore'

       at /nix/store/2pplzbz7d5jmi4mnqmfi7lds5bz2z5hx-source/lib/fileset.nix:453:19:

          452|   */
          453|   importToStore = { name ? "source", base ? entryPoint, entryPoint, fileset }:
             |                   ^
          454|     let

       … from call site

My entryPoint is the result of lib.cleanSource, so it's itself a store path. It seems like unlike cleanSource, the filesystem stuff doesn't compose — it should just merge the filters like cleanSource does, because otherwise even if you can fix the reference problem there'll be a redundant intermediate copy.

[infinisil]

@alyssais Thanks for the feedback!

On one hand, the lib.fileset functions should be able to fully replace lib.sources, but safer and with better composition! So the idea would be to instead write a lib.fileset.clean, and use that instead of lib.sources.cleanSource. I haven't pushed such a function to the PR yet, limiting it to just the core functionality for now, but it can very much be implemented and I should do that at some point.

However you also made be realize that it is very much possible to convert lib.sources-produced values to be used in lib.fileset (not the other way around though!), which I now implemented, including better error messages when appropriate.

Could you give the same code another try with the updated PR?

[infinisil]

lib.fileset.unions is now merged, so this is now supported:

lib.fileset.toSource {
  root = ./.;
  fileset = lib.fileset.unions [
    ./host/rootfs
    ./scripts
  ];
}

[alyssais]

@infinisil sorry I never got round to retesting, but I still don't think it works for what I want to do — unless I'm mistaken there's still no way to use filesets with the result of lib.cleanSource, is there?

[infinisil]

@alyssais Good point, indeed that's not yet possible. In the WIP PR #222981 there's a lib.fileset.fromSource which allows that. I'll add this to my TODO list.

Though note that for the future it would be great to have a fileset-based lib.cleanSource instead though.

[infinisil]

Now opened #261732 for this, I should be able to polish that soon