Make maven_jar and friends smarter by re-using previously fetched artifacts across different projects

[davido]

Current maven_jar() implementation is limited to only using the fetched artifact for specific project. It doesn't provide solution for very basic requirements (that native Maven would provide):

• previously fetched artifacts should survive entire clean of output base for specific project (bazel clean --expunge)
• previously fetched artifacts can be reused by other clones of the same project. Say i have separate clones for master and stable branch
• previously fetched artifacts can be re-used by different projects. Say Gerrit Code Review, JGit, Gitiles and 42 Gerrit plugins (standalone build mode) are cloned and built on the same machine. Their prerequisites are almost the same

See Gerrit Code Review maven_jar Bucklet: [1] implementation how to get it right. The implementation is putting all fetched artifact in project independent area and is linking the artifacts to the poject output: [2]. More context is here: [3,4].

The bandwith is just too valuable resource to throw away (or ignore) previously downloaded artifacts and re-fetch the gigabytes of data again.

• [1] https://github.com/davido/bucklets/blob/master/maven_jar.bucklet
• [2] https://github.com/davido/bucklets/blob/master/tools/download_file.py
• [3] https://gerrit-review.googlesource.com/#/c/76253/1/WORKSPACE@6
• [4] make buck fetch smarter by keeping previous fetched artifacts facebook/buck#602

[kchodorow]

+1, it's super annoying to have these re-downloaded on every new workspace. This is a little tricky to implement, because we do need a way to clear the "master cache," whether that's to clear or update individual entries that might be corrupt/out of date or just avoid taking up all of the user's disk space. We'll also have to be careful about correctness, perhaps using the cache should require a hash.

(There were other requests for this a while ago, but now I can't find them. Will link if I come across them.)

[jin]

I've thought about these problems as well while re-implementing maven_jar() (#1410), especially while dealing with the location of local repositories. Will need to find a balance between the persistence of local repositories and Bazel's reproducibility and correctness ethos.

[kchodorow]

Aha, related to #1266.

[davido]

Thanks for the link. There is also similar feature request mentioned in #1266 on the dev mailing list.

As pointed out by @jin it would be trivial, to teach maven_jar reimplementation as Sykalrk rule to use alternative (project independent) directory, say ~/.bazel_external_repository: [1], right? Bonus point, to make this location configurable, so that we could just point to the Buck's download_artifact cache, or even teach the Skylark rule to hijack ~/.m2 for that purpose and re-use the downloaded artifact by Maven itself (for those of us who still have to use Maven for some other projects, that weren't ported to Buck or Bazel yet).

• [1] https://bazel-review.googlesource.com/5770

[jin]

Initial thoughts: a basic design uses a maven_local_repository rule, which is basically a thin wrapper around native local_repository. It only has one attribute, path, which lets you specify the absolute path of the system's local Maven repository.

load("@bazel_tools//tools/build_defs/repo:maven_rules.bzl", "maven_local_repository")
maven_local_repository(
    path = "/home/johndoe/.m2",
)

This folder will then be symlinked to each maven_jar subfolder in //external, hence allowing caching across Bazel builds, and reusability across build tools.

[kchodorow]

This would need to work for all repository rules, not just Maven. Right now we download/link stuff into output_base/externalreponame, we'd need to come up with a central location to download/link stuff, then add a step to create the symlink output_base/external/reponame -> $CENTRAL_CACHE/reponame.

[damienmg]

Caching the HttpDownloadValue is probably the easiest way to go forward. But we might want to offer that caching capabilities a bit more exposed directly (so execute result can also be cached)

[johnynek]

generally caching any download with a sha would be great. This is a major painpoint for our developers as we have a lot of downloads (many external repos: maven_jar + git_repository).

[jin]

0590483 now lets you use --experimental_repository_cache=$HOME/some/path to cache downloaded artifacts that have their SHA256 values specified. This cache will survive bazel clean --expunge. Works with artifacts downloaded with new_http_archive, http_archive, http_file, http_jar, Skylark's download and download_and_extract. Maven support coming up.

[jin]

With 38e54ac, maven_jar artifacts with the SHA1 value specified can now be cached using --experimental_repository_cache.

[davido]

Thanks. This is very much appreciated!

I have built from the tip of master and am trying to integrate this great feature in Gerrit Code Revew Bazel build and having some questions:

Neither ~/.gerritcodereview/bazel_repository_cache nor $HOME/.gerritcodereview/bazel_repository_cache seems to be supported. What we would like to do is to put these lines in .bazelrc in root of Gerrit project (this file is under GIT control, so we don't have the option to use resolved user name, only ~ or $HOME):

build --experimental_repository_cache=~/.gerritcodereview/bazel_repository_cache --workspace_status_command=./tools/workspace-status.sh --strategy=Javac=worker
fetch --experimental_repository_cache=~/.gerritcodereview/bazel_repository_cache

Note that Buck supports something similar, one can hijack the directory cache to user home directory (excerpt from .buckconfig):

[cache]
  mode = dir
  dir = ~/.gerritcodereview/buck-cache/locally-built-artifacts

I added this feature to Buck 2 years ago: facebook/buck@a1ba001

For now I tested it with --experimental_repository_cache=/home/davido/.gerritcodereview/bazel_repository_cache and it works as expected. I tried bazel fetch gerrit, then disconnected the wire and have rebuilt with bazel build gerrit without network connection. It has also survived bazel clean --expunge ;-). This is great!

Question: Can it be that the cached artifact are copied to external artifacts and not linked? i cannot see that symbolic links are used. Any particular reason to not use symbolic links for that?

[kchodorow]

Question: Can it be that the cached artifact are copied to external artifacts and not linked? i cannot see that symbolic links are used. Any particular reason to not use symbolic links for that?

We may change this in the future, but for now we decided to use copies to simplify cache cleanup.

Bazel options generally don't support ~ nor $HOME, I filed #2054 to gauge interest/have discussion.

[pwnall]

Asides from all the benefits mentioned above, I think that having a build cache makes it easy to use bazel repositories with package managers that do not allow (by policy) the build process to download anything on its own.

[dslomov]

We are deprecating native maven_jar.

[Vincent-M]

what is "native maven_jar" ? is this maven_jar(...) rule being deprecated? or will we just need to add a load(..., maven_jar at the top of the WORKSPACE file?

Regardless of the answer, the documentation says:

Prefer http_archive to git_repository, new_git_repository, and maven_jar

Could someone give an example on how to transform the following example:

maven_jar(
    name = "antlr27",
    artifact = "antlr:antlr:2.7.7",
    attach_source = False,
    sha1 = "83cd2cd674a217ade95a4bb83a8a14f351f48bd0",
)

to the http_library equivalent? (like let's say for instance that jar was not available in maven, but in some other place via a https url.)

My guess is in WORKSPACE :

http_archive(
    name = "antlr_lib",
    urls = [http://central.maven.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.jar"],
    sha256 = "<...>",
    build_file = "@//:antlr.BUILD",
)

but what to put in the antlr.BUILD file? a java_import?
Or is there a way to just have the http_archive like this:

http_archive(
    name = "antlr27",
    urls = [http://central.maven.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.jar"],
    sha256 = "<...>",
)

and direclty use the jar in the targets similar to how "@antlr27//jar" would be used out of maven_jar?