Make a shallow clone of git submodules

[yberreby]

When building from source, the build system makes a deep clone of LLVM and other dependencies, downloading several hundred megabytes, which takes a significant time over a slow connection. A shallow clone (done with git submodule update --depth 1, see http://stackoverflow.com/a/17692710/2754323 ) would only download the latest revision of the dependencies, since the rest of the commit history is not needed, and save bandwidth and space at no (as far as I know) cost.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. The way Github handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see CONTRIBUTING.md for more information.

[yberreby]

r? @steveklabnik

[rprichard]

I tried running git submodule init and git submodule update --depth 1 in a new Rust checkout, and it didn't work. The update command downloaded the master branch of https://github.com/rust-lang/llvm.git (commit 2e9f0d21fe321849a4759a01fc28eae82ef196d6), which is two years out of date. Then update complained that it couldn't find the commits it expected:

rprichard@ryan:~/work/rust-test$ git submodule update --depth 1
Cloning into 'src/compiler-rt'...
remote: Counting objects: 1405, done.
remote: Compressing objects: 100% (1226/1226), done.
remote: Total 1405 (delta 289), reused 845 (delta 152), pack-reused 0
Receiving objects: 100% (1405/1405), 2.02 MiB | 3.67 MiB/s, done.
Resolving deltas: 100% (289/289), done.
Checking connectivity... done.
fatal: reference is not a tree: 58ab642c30d9f97735d5745b5d01781ee199c6ae
Cloning into 'src/jemalloc'...
remote: Counting objects: 164, done.
remote: Compressing objects: 100% (145/145), done.
remote: Total 164 (delta 17), reused 80 (delta 14), pack-reused 0
Receiving objects: 100% (164/164), 354.07 KiB | 0 bytes/s, done.
Resolving deltas: 100% (17/17), done.
Checking connectivity... done.
fatal: reference is not a tree: e24a1a025a1f214e40eedafe3b9c7b1d69937922
Cloning into 'src/llvm'...
remote: Counting objects: 10756, done.
remote: Compressing objects: 100% (10239/10239), done.
remote: Total 10756 (delta 1194), reused 4076 (delta 433), pack-reused 0
Receiving objects: 100% (10756/10756), 14.39 MiB | 6.03 MiB/s, done.
Resolving deltas: 100% (1194/1194), done.
Checking connectivity... done.
fatal: reference is not a tree: bff69076975642c64e76dbeaa53476bfa7212086
Cloning into 'src/rt/hoedown'...
remote: Counting objects: 83, done.
remote: Compressing objects: 100% (81/81), done.
remote: Total 83 (delta 1), reused 62 (delta 1), pack-reused 0
Unpacking objects: 100% (83/83), done.
Checking connectivity... done.
fatal: reference is not a tree: 238c4d57cce10d33b05cf52a91fc62a09f31ffbb
Cloning into 'src/rust-installer'...
remote: Counting objects: 27, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 27 (delta 2), reused 18 (delta 0), pack-reused 0
Unpacking objects: 100% (27/27), done.
Checking connectivity... done.
fatal: reference is not a tree: ebc6b04c29591108d3f28e724b4b9b74cd1232e6
Unable to checkout '58ab642c30d9f97735d5745b5d01781ee199c6ae' in submodule path 'src/compiler-rt'
Unable to checkout 'e24a1a025a1f214e40eedafe3b9c7b1d69937922' in submodule path 'src/jemalloc'
Unable to checkout 'bff69076975642c64e76dbeaa53476bfa7212086' in submodule path 'src/llvm'
Unable to checkout '238c4d57cce10d33b05cf52a91fc62a09f31ffbb' in submodule path 'src/rt/hoedown'
Unable to checkout 'ebc6b04c29591108d3f28e724b4b9b74cd1232e6' in submodule path 'src/rust-installer'

The current branch is rust-llvm-2015-02-19, but .gitmodules specifies the master branch instead.

I'm using git version 1.9.1 on Xubuntu 14.04. Maybe that matters?

[pnkfelix]

@rprichard I think what you describe is consistent with @brson's findings when he investigated trying to do this, based on a recent conversation I heard him having.

[yberreby]

After research, this seems to be a common issue with submodules and low values of depth:
CocoaPods/cocoapods-downloader#32
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/EJtj5W-60-Y
http://www.tech.theplayhub.com/git_submodule_head_reference_is_not_a_tree_error/

[alexcrichton]

I believe a bare --depth 1 means that git will fetch the master branch with a depth of 1, but the LLVM repo that we're using has a branch for each rev that we're pinned to. This means that the clone will have to specify both the branch and the depth, I believe.

[brson]

As others have said, Rust submodules are not using the master branch of their repos, so a shallow clone of master will not work.

The best way I can think to get to a point where this works would be to set all our submodules to a stable named branch and state that branch in .gitmodules, make it a policy matter that submodules always use that name.

Getting there will require further planning, so closing this PR.

[brson]

If you want to pursue this further I might suggest opening up a thread on internals.rust-lang.org laying out a plan.

[brson]

Oh, the plan I stated previously doesn't work because historic checkouts don't work after submodule master branches have changed.