Zig documentation generation occasionally fails with Zig >=0.12.0

[aherrmann]

See fddc80d

(06:27:04) ERROR: /Users/runner/work/rules_zig/rules_zig/e2e/workspace/zig-docs/BUILD.bazel:39:9: Generating Zig documentation for zig-docs/main.zig in bazel-out/darwin_arm64-fastbuild/bin/zig-docs/test.docs failed: (Exit 1): zig failed: error executing command (from target //zig-docs:test) external/zig_0.12.0_aarch64-macos/zig test --test-no-exec '-femit-docs=bazel-out/darwin_arm64-fastbuild/bin/zig-docs/test.docs' -fno-emit-bin -fno-emit-implib --zig-lib-dir ... (remaining 20 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
error: unable to open directory '/var/tmp/zig-cache/b/cb9becf646f63b0941e2ca559689c01a/': FileNotFound

and

(06:28:49) ERROR: /Users/runner/work/rules_zig/rules_zig/zig/runfiles/BUILD.bazel:34:12: Generating Zig documentation for zig/runfiles/runfiles.zig in bazel-out/darwin_arm64-fastbuild/bin/zig/runfiles/lib.docs failed: (Exit 1): zig failed: error executing ZigBuildDocs command (from target //zig/runfiles:lib) external/_main~zig~zig_0.12.0_aarch64-macos/zig build-lib '-femit-docs=bazel-out/darwin_arm64-fastbuild/bin/zig/runfiles/lib.docs' -fno-emit-bin -fno-emit-implib --zig-lib-dir ... (remaining 14 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
error: unable to open directory '/var/tmp/zig-cache/b/d9a5ae9513d8cf07f558187b0c1f9a5d/': FileNotFound

[aherrmann]

I was not able to reproduce this in a tmate session on a MacOS CI node. It's possible that this issue is caused by an inhermeticity introduced by the global shared cache.

[aherrmann]

Another instance occurred after adding a Bazel 7.0.0 pipeline and dropping Bazel 6 support.

(15:49:25) ERROR: /private/var/tmp/_bazel_runner/5ca2a14b857dddeda41464da6fe32452/external/rules_zig~override/zig/runfiles/BUILD.bazel:34:12: Generating Zig documentation for external/rules_zig~override/zig/runfiles/runfiles.zig in bazel-out/darwin_arm64-fastbuild/bin/external/rules_zig~override/zig/runfiles/lib.docs failed: (Exit 1): zig failed: error executing ZigBuildDocs command (from target @@rules_zig~override//zig/runfiles:lib) external/rules_zig~override~zig~zig_0.12.0_aarch64-macos/zig build-lib '-femit-docs=bazel-out/darwin_arm64-fastbuild/bin/external/rules_zig~override/zig/runfiles/lib.docs' -fno-emit-bin ... (remaining 16 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
error: unable to open directory '/var/tmp/zig-cache/b/d9a5ae9513d8cf07f558187b0c1f9a5d/': FileNotFound

[aherrmann]

I have also observed this issue locally on Ubuntu 22.04

$ bazel clean --expunge; bazel build //... --noremote_accept_cached
...
ERROR: /home/aj/src/rules_zig/main/zig/runfiles/BUILD.bazel:34:12: Generating Zig documentation for zig/runfiles/runfiles.zig in zig/runfiles/lib.docs failed: (Exit 1): zig failed: error executing ZigBuildDocs command (from target //zig/runfiles:lib) external/_main~zig~zig_0.12.0_x86_64-linux/zig build-lib '-femit-docs=bazel-out/k8-fastbuild/bin/zig/runfiles/lib.docs' -fno-emit-bin -fno-emit-implib --zig-lib-dir ... (remaining 14 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
error: unable to open directory '/tmp/zig-cache/b/20c09a1aeb3eb72fef452c05789e32ea/': FileNotFound

The error may be coming from https://github.com/ziglang/zig/blob/a685ab1499d6560c523f0dbce2890dc140671e43/src/Compilation.zig#L3755. Perhaps a race between building the runfiles library and the runfiles library docs?

[aherrmann]

Latest encounter on BCR CI, see here.

[aherrmann]

Reproduced locally on Ubuntu 22.04 with the following command

$ cd e2e/workspace
$ bazel clean; bazel test --build_tests_only --sandbox_tmpfs_path=/tmp -- //zig-docs:zig-docs //zig-docs:test

In contrast, the following succeeded more than three times in a row

$ cd e2e/workspace
bazel clean; bazel test --build_tests_only -- //zig-docs:zig-docs //zig-docs:test

The issue may be caused by an interplay between --sandbox_tmpfs_path=/tmp and the --(global-)cache-dir setting and --sandbox_add_mount_pair=/tmp.

---

Additionally passing the following flags seems to resolve the issue:

--repo_env=RULES_ZIG_CACHE_PREFIX=/var/tmp/bazel-zig-cache --sandbox_add_mount_pair=/var/tmp --sandbox_writable_path=/var/tmp