Cache submodules between different git checkouts

[jyn514]

What does this PR try to resolve?

Rather than letting git manage each submodule checkout individually, this uses a shared bare submodule directory in target/git/checkouts/submodules and makes a checkout of the shared directory in the relative path.

This caches based on the URL, but base64-encodes the URL to avoid errors about invalid FS paths. I'm hesitant to cache based on the relative path in the git checkout because it could use a different remote between commits of the parent.

Fixes #7987.

How should we test and review this PR?

I tested like so:

$ cargo init
$ echo 'boring = { git = "https://github.com/cloudflare/boring", branch = "master" }' >> Cargo.toml
$ /home/jnelson/rust-lang/cargo/target/debug/cargo update -p boring --precise 46787b7b6909cadf81cf3a8cd9dc351c9efdfdbd
$ /home/jnelson/rust-lang/cargo/target/debug/cargo update -p boring --precise c037a438f8d7b91533524570237afcfeffffe496

and confirmed that the time to do the second update is negligible. I'm not sure how to test this automatically using the cargo_test framework since cargo still prints "Updating git submodule" on the second checkout (because it has to create the symlink).

[rust-highfive]

r? @alexcrichton

(rust-highfive has picked a reviewer for you, use r? to override)

[jyn514]

cc @ehuss

[ehuss]

I haven't had a chance to review this, but I did want to mention a few things before you spend too much time on them.

I don't think we can rely on symlinks on windows. Older versions do not support them.

I don't think links are necessary, though. I would expect this to use the db directory to store a bare clone of the submodule, and to then just check out the specific revision inside the parent project (git/checkouts/foo-abcdef/ghdefk/submodule-path).

[jyn514]

I don't think links are necessary, though. I would expect this to use the db directory to store a bare clone of the submodule, and to then just check out the specific revision inside the parent project (git/checkouts/foo-abcdef/ghdefk/submodule-path).

Ok, done :)

[alexcrichton]

Reading over this I don't think that this is the best approach to solve this problem. Naively the "fix" I would expect for this issue is what @ehuss described. Checkouts of submodules would perform basically the exact same process as the rest of git checkouts. There's a shared db for each submodule url, and within that db we make resolve the submodule git commit or otherwise fetch contents to try to resolve it. Afterwards we'd then perform a checkout from the bare db repository into the checkout of the outer git checkout. Most of this probably wouldn't actually use libgit2's submodule support other than simply iterating, otherwise we'd be managing submodules in a custom fashion.

[jyn514]

@alexcrichton Sorry, I forgot to update the PR description - this now uses @ehuss's suggestion of bare checkouts.

[alexcrichton]

While that may be the case, I had other suggestions in my previous comment I don't think should be ignored simply because bare checkouts are used somewhere now.

[bors]

☔ The latest upstream changes (presumably #10296) made this pull request unmergeable. Please resolve the merge conflicts.

[jyn514]

There's a shared db for each submodule url, and within that db we make resolve the submodule git commit or otherwise fetch contents to try to resolve it. Afterwards we'd then perform a checkout from the bare db repository into the checkout of the outer git checkout.

Sorry, I'm not sure I understand what I haven't done from that list.

• There's a shared db for each submodule url: https://github.com/rust-lang/cargo/blob/2a2a94a2068bb235c1eac704c95e9e8ed6612b8a/src/cargo/sources/git/utils.rs#L347-L353
• Within that db we resolve the commit or otherwise fetch it: https://github.com/rust-lang/cargo/blob/2a2a94a2068bb235c1eac704c95e9e8ed6612b8a/src/cargo/sources/git/utils.rs#L435-L441
• Afterwards, we perform a checkout from the bare repository: https://github.com/rust-lang/cargo/blob/2a2a94a2068bb235c1eac704c95e9e8ed6612b8a/src/cargo/sources/git/utils.rs#L443-L444

Is there something I missed?

[alexcrichton]

Cargo has existing infrastructure for a global database of git repos and a global database of git checkouts. You've bypassed all that infrastructure and invented your own scheme. My comment is that you should be using what Cargo already has instead of inventing something new.

[alexcrichton]

Thanks for the PR, but I'm going to be stepping down from the Cargo team so I'm going to un-assign myself from this. The Cargo team will help review this when they get a chance.

[jyn514]

I am not quite sure how to proceed here; I'll open a new PR if I figure it out.