immutable cache design #12716
Comments
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Also, #12719 (comment) provides data on just how bad the sandbox setup cost is (60-70% of execution time on sandbox setup). |
|
Yeah, if we have an option that is both less work and higher performance, then why not do that? Granted it's a mild concession of hermeticity, but I think one we can easily live with for now. |
…13352) Turns out that #13339 didn't actually work - we are redownloading the same modules several times with Go! Downloads happen when: 1. determining `GoModInfo` (once per `go.mod`) 2. `AllDownloadedModules` (once per `go.mod`) 3. Determining metadata for each third-party package (once per third-party module) 4. Determining metadata for each first-party package (once per first-party package/directory) This PR fixes it so that we only download modules a single time, once per `go.mod`. To fix this, we stop treating third-party modules like first-party modules, i.e. we stop `cd`-ing into its downloaded directory and running `go list` directly in it, using its own `go.mod` and `go.sum`. That requires that the chroot has all of the module's transitive dependencies present, and it also resulted in issues like #13138. Instead, the much simpler thing to do is run `go list '...'` to do all third-party analysis in a single swoop. That gets us all the analysis we need. We also extract the relevant `.go` files from all of the downloaded `GOPATH`, i.e. all the downloaded modules. For compilation, all we need is the `.go` files + the metadata we had earlier calculated. Compilation doesn't need access to anything else like other package's. For first-party analysis, we copy the whole `GOPATH` into the chroot. (This is really slow! We need something like #12716 to fix this.) ## Benchmark Running in https://github.com/toolchainlabs/remote-api-tools. Before: ``` ❯ hyperfine -r 5 './pants_from_sources --no-process-execution-local-cache --no-pantsd package ::' Time (mean ± σ): 36.467 s ± 0.603 s [User: 41.109 s, System: 38.095 s] Range (min … max): 35.518 s … 37.137 s 5 runs ``` Fixing only third-party analysis: ``` ❯ hyperfine -r 5 --show-output './pants_from_sources --no-process-execution-local-cache --no-pantsd package ::' Time (mean ± σ): 29.880 s ± 0.901 s [User: 29.564 s, System: 15.281 s] Range (min … max): 28.835 s … 31.312 s 5 runs ``` Fixing everything: ``` ❯ hyperfine -r 5 './pants_from_sources --no-process-execution-local-cache --no-pantsd package ::' Time (mean ± σ): 26.633 s ± 2.283 s [User: 24.115 s, System: 30.453 s] Range (min … max): 24.570 s … 30.037 s 5 runs ``` [ci skip-rust] [ci skip-build-wheels]
…antsbuild#13352) Turns out that pantsbuild#13339 didn't actually work - we are redownloading the same modules several times with Go! Downloads happen when: 1. determining `GoModInfo` (once per `go.mod`) 2. `AllDownloadedModules` (once per `go.mod`) 3. Determining metadata for each third-party package (once per third-party module) 4. Determining metadata for each first-party package (once per first-party package/directory) This PR fixes it so that we only download modules a single time, once per `go.mod`. To fix this, we stop treating third-party modules like first-party modules, i.e. we stop `cd`-ing into its downloaded directory and running `go list` directly in it, using its own `go.mod` and `go.sum`. That requires that the chroot has all of the module's transitive dependencies present, and it also resulted in issues like pantsbuild#13138. Instead, the much simpler thing to do is run `go list '...'` to do all third-party analysis in a single swoop. That gets us all the analysis we need. We also extract the relevant `.go` files from all of the downloaded `GOPATH`, i.e. all the downloaded modules. For compilation, all we need is the `.go` files + the metadata we had earlier calculated. Compilation doesn't need access to anything else like other package's. For first-party analysis, we copy the whole `GOPATH` into the chroot. (This is really slow! We need something like pantsbuild#12716 to fix this.) ## Benchmark Running in https://github.com/toolchainlabs/remote-api-tools. Before: ``` ❯ hyperfine -r 5 './pants_from_sources --no-process-execution-local-cache --no-pantsd package ::' Time (mean ± σ): 36.467 s ± 0.603 s [User: 41.109 s, System: 38.095 s] Range (min … max): 35.518 s … 37.137 s 5 runs ``` Fixing only third-party analysis: ``` ❯ hyperfine -r 5 --show-output './pants_from_sources --no-process-execution-local-cache --no-pantsd package ::' Time (mean ± σ): 29.880 s ± 0.901 s [User: 29.564 s, System: 15.281 s] Range (min … max): 28.835 s … 31.312 s 5 runs ``` Fixing everything: ``` ❯ hyperfine -r 5 './pants_from_sources --no-process-execution-local-cache --no-pantsd package ::' Time (mean ± σ): 26.633 s ± 2.283 s [User: 24.115 s, System: 30.453 s] Range (min … max): 24.570 s … 30.037 s 5 runs ``` [ci skip-rust] [ci skip-build-wheels]
…(Cherry-pick of #13352) (#13378) Turns out that #13339 didn't actually work - we are redownloading the same modules several times with Go! Downloads happen when: 1. determining `GoModInfo` (once per `go.mod`) 2. `AllDownloadedModules` (once per `go.mod`) 3. Determining metadata for each third-party package (once per third-party module) 4. Determining metadata for each first-party package (once per first-party package/directory) This PR fixes it so that we only download modules a single time, once per `go.mod`. To fix this, we stop treating third-party modules like first-party modules, i.e. we stop `cd`-ing into its downloaded directory and running `go list` directly in it, using its own `go.mod` and `go.sum`. That requires that the chroot has all of the module's transitive dependencies present, and it also resulted in issues like #13138. Instead, the much simpler thing to do is run `go list '...'` to do all third-party analysis in a single swoop. That gets us all the analysis we need. We also extract the relevant `.go` files from all of the downloaded `GOPATH`, i.e. all the downloaded modules. For compilation, all we need is the `.go` files + the metadata we had earlier calculated. Compilation doesn't need access to anything else like other package's. For first-party analysis, we copy the whole `GOPATH` into the chroot. (This is really slow! We need something like #12716 to fix this.) ## Benchmark Running in https://github.com/toolchainlabs/remote-api-tools. Before: ``` ❯ hyperfine -r 5 './pants_from_sources --no-process-execution-local-cache --no-pantsd package ::' Time (mean ± σ): 36.467 s ± 0.603 s [User: 41.109 s, System: 38.095 s] Range (min … max): 35.518 s … 37.137 s 5 runs ``` Fixing only third-party analysis: ``` ❯ hyperfine -r 5 --show-output './pants_from_sources --no-process-execution-local-cache --no-pantsd package ::' Time (mean ± σ): 29.880 s ± 0.901 s [User: 29.564 s, System: 15.281 s] Range (min … max): 28.835 s … 31.312 s 5 runs ``` Fixing everything: ``` ❯ hyperfine -r 5 './pants_from_sources --no-process-execution-local-cache --no-pantsd package ::' Time (mean ± σ): 26.633 s ± 2.283 s [User: 24.115 s, System: 30.453 s] Range (min … max): 24.570 s … 30.037 s 5 runs ``` [ci skip-rust] [ci skip-build-wheels]
|
We will need to weigh implementing either this or FUSE support for #13435, depending on user's constraints around enabling FUSE. EDIT: Fixed issue link. |
|
Isn't FUSE a bit of a no-go with the current status of Apple trying hard to kick kexts to the curb and the maintenance situation of the fuse project for Mac? |
My understanding is that although it is closed source (seemingly in order to prevent rebranded forks of the FUSE project), it is still actively maintained. We definitely would not be expecting to fork and rebrand, so I'm not sure whether that is a blocker or not. Would need to see how comfortable companies with large monorepos are with using it as is, and what platforms they primarily are on. |
kext's got replaced with "system extensions" which run in user space (and are probably leveraging Mach to some degree). macOS FUSE appears to use system extensions which is implied by the "system extension" showing up in some error message: osxfuse/osxfuse#828 |
|
I spent an hour looking at FUSE last night, and the experience was vaguely disconcerting on macOS: I updated to a modern fork of the It's very likely that the tests are not tearing down the library correctly, because no amount of using the But I wasn't quite able to benchmark |
|
Correction. I was able to get some very unscientific benchmarks by executing
Useful also would be measuring the total amount of IO: materialization involves reading+writing+reading, while The immutable cache approach would involve the materialization step (once), followed by the creation of a much smaller number of symlinks (proportional to the total number of |
|
One other general thought here: to avoid implementing unionfs support in Given that, and given the relative instability of FUSE, it seems like the first step forward should be a symlinked strategy with the destination initially being a lightly defended temp dir, as first proposed in this ticket. We can then later have the symlink destination be |
As discussed in #13787, the `input_files` digest must currently include the `use_nailgun` digest, and that means that the input files for the server are materialized for every run. * Add an `InputDigests` struct to encapsulate managing the collection of input digests for a `Process` (which will soon also include the immutable digests from #12716), and split the `use_nailgun` digest from the `input_files` digest. * Move nailgun spawning (which uses `std::process` and `std::fs`, and so is synchronous) onto the `Executor`. * Adjust the (still hardcoded) nailgun pool size to keep idle servers beyond the number that are currently active (e.g. to allow for idle `javac` processes while `scalac` processes are active). Collectively, these changes make compilation 40% faster. Fixes #13787.
As designed by @tdyas in #12716, this change adds `immutable_inputs` to `Process`, and adapts the JVM `@rules` to use them: for now, only for the tools themselves. A followup change will move all JVM classpath inputs to `immutable_inputs`. Nailgun support is now enabled by specifying the subset of the `immutable_inputs` which should be used to start the server: all others will be used to start the client. [ci skip-build-wheels]
|
It looks like there might now be a more stable FUSE backend for macOS: https://www.fuse-t.org/ |
This issue proposes adding an "immutable caches" feature to
Processinvocations to support cases where large archives (e.g, the Go distribution) would need to be unpacked in an input root. See the following design doc for the details: https://docs.google.com/document/d/1e4djdm2D9L6IeeJ8Q1M0_yJU-kF4tTHopBomwzdDRpE/edit#heading=h.n4lrkjmz3x2sThe text was updated successfully, but these errors were encountered: