lib.fileset.trace, lib.fileset.traceVal: init

[infinisil]

Description of changes

Another split-off from the main file set combinators PR. This one adds two functions for debugging file sets:

• lib.fileset.trace: Trace a file set and return a value

  trace (unions [ ./Makefile ./src ./tests/run.sh ]) null

• lib.fileset.traceVal: Same, but it returns the file set itself

  toSource {
    root = ./.;
    fileset = traceVal (unions [
      ./Makefile
      ./src
      ./tests/run.sh
    ]);
  }

These both print the file set like this:

trace: /home/user/src/myProject
trace: - Makefile (regular)
trace: - src (all files in directory)
trace: - tests
trace:   - run.sh (regular)

This work is sponsored by Antithesis ✨

Things done

• Implementation
  • Tested
  • Commented
• Added user documentation

[infinisil]

Removed the seq foldl' hack now that #256544 is merged.

[fricklerhandwerk]

The weird thing is that the stack overflow avoidance tests fail on my NixOS 23.05 with Nix 2.13.3.

test case at lib/fileset/tests.sh:523 failed: toSource { root = ./.; fileset = unions (
mapAttrsToList (name: _: ./. + "/${name}/a") (builtins.readDir ./.)); } failed to evalu
ate, but it was expected to succeed

lib/fileset/tests.sh: line 124: 18590 Segmentation fault      (core dumped) nix-instant
iate --eval --show-trace --expr "$prefixExpression (trace $config $expr)" > /dev/null 2
> "$tmp"/stderrTrace
lib/fileset/tests.sh: line 124: 18596 Segmentation fault      (core dumped) nix-instant
iate --eval --show-trace --expr "$prefixExpression (trace $config $expr)" > /dev/null 2
> "$tmp"/stderrTraceVal
test case at lib/fileset/tests.sh:635 failed: ./. should have traced this:

/run/user/1000/tmp.Ue0Y2xun3k/work (all files in directory)

but this was actually traced:

[infinisil]

@fricklerhandwerk (using this thread to reply to #256417 (review))

The weird thing is that the stack overflow avoidance tests fail on my NixOS 23.05 with Nix 2.13.3.

Interesting, does it happen both with nix-build lib/tests/release.nix and lib/fileset/tests.sh?

I tried reproducing this with

diff --git a/lib/tests/release.nix b/lib/tests/release.nix
index c8d6b810122e..02067a79f9a6 100644
--- a/lib/tests/release.nix
+++ b/lib/tests/release.nix
@@ -2,7 +2,12 @@
   # Don't test properties of pkgs.lib, but rather the lib in the parent directory
   pkgs ? import ../.. {} // { lib = throw "pkgs.lib accessed, but the lib tests should use nixpkgs' lib path directly!"; },
   nix ? pkgs.nix,
-  nixVersions ? [ pkgs.nixVersions.minimum nix pkgs.nixVersions.unstable ],
+  nixVersions ? [
+    pkgs.nixVersions.minimum
+    pkgs.nixVersions.nix_2_13
+    nix
+    pkgs.nixVersions.unstable
+  ],
 }:
 
 let

and nix-build lib/tests/release.nix -A paths.1, but no luck. I also tried downgrading from the current 2.13.5 to 2.13.3 by reverting bcb02bd, but that also didn't cause it to break.

I can kind of see how it could happen though: It looks like Nix segfaults if there's too little stack space, though for me that only happens for <44*1024 bytes:

$ ulimit -s 44
$ nix-env --version
nix-env (Nix) 2.15.1
$ ulimit -s 43
$ nix-env --version
zsh: segmentation fault (core dumped)  nix-env --version

Could you test whether you can run the tests on master? Because there's already another test doing this too:

nixpkgs/lib/fileset/tests.sh

Lines 471 to 473 in efb617b

	# Locally limit the maximum stack size to 100 * 1024 bytes
	# If unions was implemented recursively, this would stack overflow
	ulimit -s 100

@roberth Testing the stack like this might be machine dependent I guess, might be better to drop it

[fricklerhandwerk]

@infinisil it doesn't fail with or without ulimit when using nix-build, but does the same as previously when running test script directly. Note though that the output store path when using nix-build is /nix/store/7gf7wnp28h77gl5y1k2pw3n2pmxg6b57-nixpkgs-lib-tests-nix-2.17.0.

[infinisil]

I was able to not make the new test here rely on ulimit -s, I also created #258080 to remove the other use of ulimit -s.

[roberth]

Probably needs adaptation for

• lib.fileset: Representation for empty file sets without a base path #257351

[infinisil]

Rebased to fix conflicts and removed the {} argument from trace/traceVal as suggested. This even makes this more consistent with builtins.trace and lib.debug.traceVal, as those also can't be customised.

Here's the easily-viewable diff of the changes: https://github.com/tweag/nixpkgs/compare/55c1b0b462d28d404f78695e7d8d8885f81fe080..53c053be347989bff8c08769f08561ced200b21f

[roberth]

Thanks for the diff link! That helps a lot.

I don't see a test case for trace (unions [ ]). Could you add it?

[infinisil]

@roberth Oh good catch, I didn't think of the interaction after #257351, it indeed didn't work at all and needed to be fixed! That's done now, diff here (this one GitHub also links to)