Stop copying jar files #1074
Stop copying jar files #1074
Conversation
POM target shouldn't get a http_file target at all, they don't have a a file or a sha256, there's nothing to download
If the file is None then that means we shouldn't pass this to the http_file call, most likely this entries also have a None sha256 sum
manuelnaranjo
requested review from
jin,
Wyverald,
shs96c and
cheister
as code owners
manuelnaranjo
force-pushed
the
mnaranjo/no-copy
branch
from
3c1a089
to
8abcc5f
Compare
In Java 13 -noverify got marked as deprecated and may be dropped in the future
Since we have downloaded_file_path that points to the actual jar name in our http_file invocations we can directly consume from those targets instead of copying, this has an impact not just in IO as we reduce copying, but also reduces the amount of targets bazel is aware as we don't need an extra node, reducing the amount of RAM needed for rules_jvm_external slightly. In large maven repositories we use at Booking.com we saw a decrease in the disk cache from 2.4GB to 300MB, which aligns with a repository_cache of roughly 2.1GB. The time it takes to do a `bazel build @maven//...` went from 120 seconds to 50 seconds on a i9-13900H using 75% of the cores.
manuelnaranjo
force-pushed
the
mnaranjo/no-copy
branch
from
8abcc5f
to
d88ca8e
Compare
|
@manuelnaranjo, the idea seems like a good one, but I note that none of the CI tests are working yet. You can run these locally using |
|
@manuelnaranjo, do you need a hand getting this one over the line? It'd be nice to get it into the tree. If you do, just LMK. |
|
@shs96c I need to rebase the branch and get back into it, now I have a macbook available as well, so setting up Android locally would be easier, it's mostly lack of time that prevents me to look into this, I could use a hand, but I can't say when I'll have time to look into it |
|
@manuelnaranjo, I tend to lurk on the Bazel slack (both the "general" and the "java" channels) so feel free to ping me there, but will be on holiday for the next two weeks, so there's no hurry :) |
#1070
Stop making use of a genrule that copies the already downloaded jar handled by http_file file into the target repository and instead make use of an alias for proxying the artifact into the target repository.