libindex: add fallback path for O_TMPFILE not being supported

[hdonnay]

No description provided.

[codecov]

Codecov Report

Attention: Patch coverage is 71.42857% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 55.87%. Comparing base (bc8d3a3) to head (fedb9d3).

Files	Patch %	Lines
libindex/tempfile_linux.go	75.00%	6 Missing and 1 partial ⚠️
libindex/tempdir_unix.go	50.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1292      +/-   ##
==========================================
+ Coverage   55.82%   55.87%   +0.04%     
==========================================
  Files         265      266       +1     
  Lines       16597    16627      +30     
==========================================
+ Hits         9266     9290      +24     
- Misses       6367     6374       +7     
+ Partials      964      963       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[crozzy]

Nice, just a couple of clarifying comments

[BradLugo]

Hey @hdonnay, do you mind elaborating on this or rephrasing the comment? I'm having a hard time grokking it.

... temporary files are not unlinked from the filesystem upon creation,
because an equivalent to Linux's "/proc/self/fd" doesn't seem to exist.

I couldn't find a reason "/proc/self/fd" was required for "unlinking" temporary files. I'm also not sure what unlinking in this context means; my assumption based on the word choice would be that the implementation does some sort of symlinking and we need to unlink them at some point, but I don't think that's the case.

temporary files are not unlinked from the filesystem upon creation

I'm particularly confused by this. If we're on Linux, do other temporary files get unlinked when we create our temporary files?

---

fwiw, I'm not sure if others do this, but I know Ross and I both run the ACS Scanner on macOS devices, so sorting this out would help us out a lot (in other words, I'm not trying to be pedantic with this comment 🙂)

[BradLugo]

(also happy to move this discussion to another PR or elsewhere if we want to unblock merging this)

[hdonnay]

No worries.

"Unlink" in this context is a filesystem/syscall term-of-art: it means making a name no longer visible. This isn't exactly deleting a file because the semantics change based on the the type of object, the underlying filesystem, and outstanding file descriptions. Out of a combination of battle scars and habit, I tend to say "unlink" when I'm specifically thinking of unlink(2).

With regards to /proc/self/fd, that's a directory containing magic symlinks (actual terminology, see symlink(7)) representing open file descriptors. Opening one of them creates a new file descriptor backed by a new file description (see the NOTES section of open(2)). This means that unlike dup(2), the uses of the two file descriptors are completely independent (e.g. seeking one will not change the offset on the other).

The Linux code uses this to open files that aren't linked into the filesystem (meaning they'll never outlast the process, because only the process has them open) but can still be reopened as many times as needed to create independent file descriptors.

AFAICT, there's not a way to do this on OSX. Looking at FreeBSD man pages, fdescfs(5) acts the same way but only if mounted with nodup, which I don't think is the default. There is mention on that manpage about a O_EMPTY_PATH to openat(2), but I can't find any information on whether that exists on OSX and don't have a machine to test with. To work around this, the code simply doesn't unlink the file and re-opens it by name as needed. The downside is that the files are only removed as long as the program shuts down cleanly.

[BradLugo]

Thank you for the detailed explanation! I need to dive deeper into all this info, so please feel free to consider this discussion non-blocking for the PR

[crozzy]

LGTM passing baton to @BradLugo

[hdonnay]

will merge with green CI

[hdonnay]

/fast-forward