Improve performance of sandbox creation for Scala #13435
Comments
|
#13779 showed that we will need to do some more work here, and that to reduce IO (and to dodge After implementing #12716, the last big performance unknown will be around using a JVM-per-core model. While JVM-per-core was implemented very intentionally (allows for precise memory usage, classpath isolation, cancellation, etc), and aligns with Bazel's default, Bazel experimentally supports sharing a worker via multiplex workers, which saves JIT warmup time. To prioritize that, we'd need to get a sense of how much CPU we're spending on JIT warmup. |
stuhood
changed the title
It's worth noting though that #13603 completely removed the need for transitive dependencies for Java, which is exciting. While it is likely that we can also lean in for Scala to add exported types to shrink sandboxes, I'm less certain of the timeline for that than I am of #12716 (which had a lot of groundwork laid in #13813). EDIT: Opened #13836 for this. |
|
After #13862, the average sandbox creation time in our test repo is around 40 ms, so from that perspective, I think that we can call this issue closed. The overall performance still needs some work (we're about half as fast as I'd like us to be), but that might have to go on the back burner until we have a bit more feature work completed. |
…3862) As described in #13435, we expect large sandboxes for JVM compiles (for Scala in particular). #13848 added support for symlinking immutable inputs into sandboxes, and used it for tools. This change uses it for JVM classpaths as well. In a test repo, this reduces the total amount of time taken to create sandboxes for compilation by ~30% (from 14s to 10s), with an average sandbox creation time of 40ms (including amortized materialization of the symlink destinations). [ci skip-rust] [ci skip-build-wheels]
Unless Scala compiler errors have dramatically improved (when transitive dependencies are not present, the errors in
2.11at least tended to look like compiler crashes, but also did not give great information about what was missing), compilation support is likely to require transitive dependencies.That will mean very large sandboxes, and will likely require either #12716 or potentially FUSE support.
The text was updated successfully, but these errors were encountered: