std.Build.Cache.hit: more discipline in error handling

[andrewrk]

Previous commits

2b09299
4ea2f44

had this text:

There are no dir components, so you would think that this was
unreachable, however we have observed on macOS two processes racing to
do openat() with O_CREAT manifest in ENOENT.

This is indeed true, as verified by the snippet of code in a below comment on this PR description. However, ENOENT is also possible to be returned when a handle refers to a deleted directory. Thus, the previous workaround that unconditionally retried the file system operation is not sufficient. This patch changes the workaround to retry with O_EXCL, disambiguating which error condition has occurred.

This patch also adds an explicit error set to std.Build.Cache.hit as well as changing the failed_file_index to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache.

The equivalent failure when occurring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue.

Mitigates #18340 but does not solve it.

[andrewrk]

There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory.

Well, I think this is evidence:

error: cache directory '/Users/ci/zach2/_work/zig/zig/build-debug/zig-global-cache/z/' unexpectedly removed during compiler execution
error: cache directory '/Users/ci/zach2/_work/zig/zig/build-debug/zig-global-cache/z/' unexpectedly removed during compiler execution
error: cache directory '/Users/ci/zach2/_work/zig/zig/build-debug/zig-global-cache/z/' unexpectedly removed during compiler execution

Because the only place this directory is removed is in the same CI script which failed, which is currently running.

[andrewrk]

On aarch64-macos:

% cat repro.zig
const std = @import("std");
pub fn main() !void {
    const dir = try std.fs.cwd().makeOpenPath("dir", .{});
    try dir.deleteFile("file");
    const thread1 = try std.Thread.spawn(.{}, thread, .{dir});
    defer thread1.join();
    const thread2 = try std.Thread.spawn(.{}, thread, .{dir});
    defer thread2.join();
}
fn thread(dir: std.fs.Dir) void {
    const file = dir.createFile("file", .{}) catch |err| @panic(@errorName(err));
    defer file.close();
}
% zig run repro.zig
thread 2427287 panic: FileNotFound

So the originally reported issue (#12138) is indeed accurate about this macOS kernel behavior.

Given this, I can think of two workarounds:

• Handle error.FileNotFound on macOS by attempting to create a temporary test file in the directory using a random number, in order to disambiguate whether the error code comes from the directory no longer existing (ENOENT also occurs for creating the secondary test file), or from the file creation race (operation should be retried).
• On macOS, attempt the file open X times before concluding the directory has been removed.

[andrewrk]

@jacobly0 pointed out that with .exclusive = true you get EEXIST instead of ENOENT. This suggests another workaround: on macOS, retry with .exclusive = true and then handle EEXIST with a retry of the original syscall, and handle ENOENT as directory deletion.

Pseudocode:

loop:
  createFile exclusive=false
  ENOENT: (macOS)
    createFile exclusive=true
    EEXIST:
      goto loop
    ENOENT:
      directory deletion detected
  ENOENT: (Linux)
    directory deletion detected

Linux does not need this workaround since ENOENT always means the directory was deleted.

It is unclear whether other POSIX operating systems need this workaround. However, the path forward is clear: do not implement workarounds if we have the ability to pressure the faulty systems to improve. Apple doesn't give a flying fuck about Zig so we're stuck working around their crap, however there is a chance that not implementing this workaround for one of the BSDs, for instance, leads to a developer actually fixing the bug.