Timeout while cloning terraform repo during nightly tests

[kmoe]

Example job.

The git clone code lives in hc-install. Despite the increase to 5min timeout in hc-install (hashicorp/hc-install/issues/62) for git clone operations, we still see the following during terraform-exec tests:

=== RUN   TestApply
=== RUN   TestApply/basic-refs/heads/main
    util_test.go:112: caching exec "gitref:refs/heads/main" in dir "/tmp/tfinstall117829236/gitref-refs-heads-main"
2022/07/05 00:29:09 git_revision.go:88: running terraform pre-clone check (timeout: 1m0s)
2022/07/05 00:29:09 git_revision.go:94: terraform pre-clone check finished
2022/07/05 00:29:09 git_revision.go:123: cloning terraform repository from https://github.com/hashicorp/terraform.git to /tmp/hc-install-build-terraform4102761777 (timeout: 5m0s)
--- FAIL: TestApply (370.72s)
    --- FAIL: TestApply/basic-refs/heads/main (370.72s)
panic: error installing terraform "gitref:refs/heads/main": unable to clone terraform from "https://github.com/hashicorp/terraform.git": context deadline exceeded [recovered]
	panic: error installing terraform "gitref:refs/heads/main": unable to clone terraform from "https://github.com/hashicorp/terraform.git": context deadline exceeded

goroutine 22 [running]:
testing.tRunner.func1.2({0xc43a20, 0xc002431fe0})
	/usr/local/go/src/testing/testing.go:1209 +0x36c
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1212 +0x3b6
panic({0xc43a20, 0xc002431fe0})
	/usr/local/go/src/runtime/panic.go:1047 +0x266
github.com/hashicorp/terraform-exec/tfexec/internal/testutil.(*TFCache).find(0xc000298160, 0xc0001f31e0, {0xc0001aea68, 0x16}, 0xc000131cf0)
	/home/circleci/project/tfexec/internal/testutil/tfcache_find.go:47 +0x55a
github.com/hashicorp/terraform-exec/tfexec/internal/testutil.(*TFCache).GitRef(0xd184c7, 0xc0001f31e0, {0xc00002a078, 0xf})
	/home/circleci/project/tfexec/internal/testutil/tfcache.go:45 +0xb1
github.com/hashicorp/terraform-exec/tfexec/internal/e2etest.runTestVersions.func1(0xc0001f31e0)
	/home/circleci/project/tfexec/internal/e2etest/util_test.go:112 +0x3b6
testing.tRunner(0xc0001f31e0, 0xc00027f4a0)
	/usr/local/go/src/testing/testing.go:1259 +0x230
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1306 +0x727
FAIL	github.com/hashicorp/terraform-exec/tfexec/internal/e2etest	371.016s
FAIL

At the moment this is intermittent, happening on perhaps 10% of runs. Workaround is to rerun the job in the morning.

It may be worth migrating to GHA (#220) first to see if it solves this. Perhaps the CircleCI infra causes these delays. It should not take 5mins to shallow-clone the Terraform repo.

We could also just build the Terraform binary outside the test code itself (#318), which would work around this issue without necessarily addressing the root cause.