Make sure offline build can be achieved with vendor mode and the new lockfile format

[meteorcloudy]

Description of the feature request:

Vendor mode is being implemented at #19563, while the new lockfile format landed in #22154

We need to make sure vendor mode can actually vendor everything needed to achieve offline build easily. The new lockfile format contains less information than before and requires us to vendor more things.

Which category does this issue belong to?

No response

What underlying problem are you trying to solve with this feature?

With the new lockfile format, the bazel module graph is no longer stored in the lockfile, which means we need to do module resolution every time after a server restart. Although the resolution is fast, but it means we'll have to access the internet to fetch MODULE.bazel and source.json files even with required repositories being vendored.

To make sure a clean build without any download cache works offline, we'll need to vendor the registry files under the vendor directory besides the external repo sources.

Proposed solutions

1. Vendor with the similar layout of mirror.bazel.build

So a registry URL like

https://my.company.com/registry/modules/foo/1.0/MODULE.bazel

is stored at a path like

vendor_dir/registry_cache/my.company.com/registry/modules/foo/1.0/MODULE.bazel

However, there could be special characters in the URL:

• There might be port number in the URL, eg. my.company.com:8080, which is not a valid directory name on Windows.
• There might be upper cases in the path, eg. my.company.com/REGISTRY vs my.company.com/registry. This might cause conflicts on case-insensitive file systems.

2. Hash the registry URL

We could use a truncated sha256 hash as directory name for each registry, since we can be sure the normal registry paths like modules/foo/1.0/MODULE.bazel won't have above problems. We could add a README.md file to indicate which registry was cached.

The vendored source would look like

pcloudy@pcloudy:~/workspace/my_tests/test/vendor_src/registry_cache
$ tree .
.
├── 16b35d4f
│   ├── bazel_registry.json
│   ├── modules
│   │   └── foo
│   │       └── 1.0
│   │           ├── MODULE.bazel
│   │           └── source.json
│   └── README.md
└── 312d7aa6
    ├── bazel_registry.json
    ├── modules
    │   └── bar
    │       └── 2.1
    │           ├── MODULE.bazel
    │           └── source.json
    └── README.md

9 directories, 8 files
pcloudy@pcloudy:~/workspace/my_tests/test/vendor_src/registry_cache
$ cat 16b35d4f/README.md
This is a cache for registry: https://my.company.com/registry

Which operating system are you running Bazel on?

linux, macos, windows

What is the output of bazel info release?

HEAD

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[meteorcloudy]

/cc @Wyverald @fmeum

[meteorcloudy]

Another question, should we vendor local registry at files:///user/home/my/registry? Probably not?

[fmeum]

I find the hash scheme to be unnecessarily opaque as most registries will probably be hosted under purely lowercase URLs. I would be in favor of a scheme that preserve lowercase URLs as they are, similar to the Go module path escape scheme which escapes BazelCon as !bazel!con.

Since unconstrained URL hosts and paths can contain exclamation marks, we could use another character prohibited by https://www.ietf.org/rfc/rfc3986.txt but widely supported according to https://en.wikipedia.org/wiki/Filename#Problematic_characters, which leaves one of #@[].

Another question, should we vendor local registry at files:///user/home/my/registry? Probably not?

I would say no. If users want to vendor them, they can move them to a workspace relative path.

[meteorcloudy]

I find the hash scheme to be unnecessarily opaque as most registries will probably be hosted under purely lowercase URLs.

Indeed it's opaque, but I think we'll have upper case in registry URL more often than we hope, for example, they can show up in GitHub org name or repo name.

However, I do think having conflicts due to case or port number will be very rare overall. How about we just throw an error and let the user deal with it? If my.company.com/Registry and my.company.com/registry both show up, I would say it's more likely a typo than an actually intended use case?

I would say no. If users want to vendor them, they can move them to a workspace relative path.

SGTM

[fmeum]

I agree, in the case of a public suffix domain its single owner can avoid collisions and in the case of a content hoster such as GitHub, collisions are more likely to arise due to typo squatting than intentionally.