Clone the repo faster in Buildkite

[mokagio]

By default, Buildkite runs a git clone that fetches the entire repository history, for the main repo as well as the submodules (e.g. git clone -v -- git@github.com:wordpress-mobile/gutenberg-mobile.git .)

This approach is too slow for a huge repository like ours. A faster strategy is to run a shallow clone, git clone --depth=1 --shallows-submodules. Buildkite allows doing this via environment variables and the environment hook, but our CI configuration blocks that for security reason.

I'm not sure what is the safest and more robust way to move forward. In the meantime, I found a plugin that allows setting those environment variables and copied the approach into a plugin under our org. The reason for having our own plugin instead of relying and contributing to a community one is that it seems a safer approach. Hat tip @crazytonyli for pointing this out.

The screenshot shows a gain of more than 1 minute during checkout.

Testing

Green CI is enough here.

Alternatives

I guess something we could do is make the behavior implemented in the plugin the default in our Buildkite agents setup. Maybe this has never come up because the git checkout has never been a big problem in other repos.

In the meantime, this does the job in the context of this pipeline.

---

PR submission checklist:

• I have considered adding unit tests where possible. – N.A.
• I have considered if this change warrants user-facing release notes more info and have added them to RELEASE-NOTES.txt if necessary. – N.A.

[peril-wordpress-mobile]

Wanna run full suite of Android and iOS UI tests? Click here and 'Approve' CI job!

[SiobhyB]

@mokagio, 👋 , following the ping here, I wanted to provide some further context. We noticed that the Build iOS RN XCFramework & Publish to S3 tests began to fail after this PR was merged.

• Avoid navigation container dismissing the keyboard [test reference]:

Error log here ⤵️

$ cd /usr/local/var/buildkite-agent/plugins/github-com-automattic-git-shallow-clone-buildkite-plugin-trunk3830653496
--
  | $ git clone -v -- https://github.com/automattic/git-shallow-clone-buildkite-plugin .
  | xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
  | $ git clone -v -- https://github.com/automattic/git-shallow-clone-buildkite-plugin .
  | xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
  | $ git clone -v -- https://github.com/automattic/git-shallow-clone-buildkite-plugin .
  | xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
  | $ cd /usr/local/var/buildkite-agent/builds
  | $ cd /usr/local/var/buildkite-agent/builds/MV-MKE-X64-014/automattic/gutenberg-mobile
  | 🚨 Error: Failed to checkout plugin git-shallow-clone: exit status 1

• Fix iOS Focus loop for RichText components [test reference]

Error log here ⤵️

$ cd /usr/local/var/buildkite-agent/plugins/github-com-automattic-a8c-ci-toolkit-buildkite-plugin-2-18-2994260596
--
  | $ git clone -v -- https://github.com/automattic/a8c-ci-toolkit-buildkite-plugin .
  | xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
  | $ git clone -v -- https://github.com/automattic/a8c-ci-toolkit-buildkite-plugin .
  | xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
  | $ git clone -v -- https://github.com/automattic/a8c-ci-toolkit-buildkite-plugin .
  | xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
  | $ cd /usr/local/var/buildkite-agent/builds
  | $ cd /usr/local/var/buildkite-agent/builds/MV-MKE-X64-018/automattic/gutenberg-mobile
  | 🚨 Error: Failed to checkout plugin a8c-ci-toolkit: exit status 1

Following discussion at p1691052039405189/1691046441.873259-slack-C0126N6JH7Z, we decided to temporarily revert for the release branch (but not merge the revert to trunk) as we didn't want to block the release. We'd be grateful if you could help us look into why the tests began to fail so we can get a more permanent fix 🙇‍♀️

[AliSoftware]

@mokagio For the record, I would not recommend to use shallow clones, as from experience it often leads to edge cases in some git operations that become hard to debug. Which might explain the errors that Siobhan is seeing above.

---

What I would recommend instead to speed up git clones is to use our git cache mechanism that we use for our other repos, i.e.:

• Have a scheduled build set up via the pipeline's .tf file to run cache_repo via a dedicated yaml on a daily basis, so that the git cache is reconstructed and uploaded regularly in S3.
• Then use our git-s3-cache plugin on the steps of the repo's pipeline.yml so that Buildkite uses that git cache as a --reference and starting point when doing the cloning, instead of having to clone the whole repo from scratch.

[mokagio]

🤦‍♂️ the git-s3-cache setup was so good and invisible that, despite having dealt to my fair share of shallow checkout issues in the CircleCI days, I totally forgot about it.

Thank you for the suggestion @AliSoftware. See

• Add cache-builder.yml pipeline #6048
• https://github.com/Automattic/buildkite-ci/pull/246