Reducing the memory/bandwidth footprint of submodules with ./x.py?

[gendx]

I recently tried to compile rust by using the ./x.py script. I noticed that the first thing that happens is a full clone of all the git submodules (see https://github.com/rust-lang/rust/blob/master/src/bootstrap/bootstrap.py#L679-L687).

Cloning the whole history of all the submodules is generally of little interest to compile the current version of rust, but it has a non-negligible memory footprint. Today, the largest submodules are:

• src/llvm-project with 696.10 MiB of history
• src/doc/rust-by-example with 136.06 MiB of history

Downloading so much data can be detrimental for several reasons:

• This increases the latency of the build, especially for people with a slow downloading bandwidth.
• The remote git server takes some time to compress the objects before sending them, also increasing the latency for everyone.
• The history takes space on disk.

I've tried to tweak the lines in the above bootstrap.py script to limit the depth of the submodule update. Unfortunately, using --depth=1 doesn't work, because the submodule commit referenced here may not be the latest in the submodule repo, and apparently git cannot clone at a specific commit.

However, I found that --depth=20 (and --depth=1 for llvm-project) was working on today's state. Overall, this reduced the overhead of submodule updates quite significantly.

Of course, this isn't bullet-proof:

• Until git supports cloning a repo at a specific commit, the heuristic depth of 20 may not be enough in the future.
• I only tested this on a fresh clone of rust-lang/rust. Updating from an existing clone may not work.

Nonetheless, would it make sense to add an option (CLI flag to ./x.py for example) that allows to manually reduce the cloning depth?

[ghost]

apparently git cannot clone at a specific commit
Until git supports cloning a repo at a specific commit

AFAIK git is able to do this, but github isn't allowing it.

Probably needs a different solution 😞

[jyn514]

This is closely related to #76653.