flake: TestProvision/returns-modules

[spikecurtis]

    provision_test.go:803: log: [DEBUG] Initializing the backend...
    provision_test.go:803: log: [DEBUG] Initializing modules...
    provision_test.go:803: log: [DEBUG] - hello in module
    provision_test.go:803: log: [DEBUG] - hello.there in module/inner_module
    provision_test.go:803: log: [DEBUG] Initializing provider plugins...
    provision_test.go:803: log: [DEBUG] - Finding latest version of hashicorp/null...
Error: vision_test.go:803: log: [ERROR] Error: Failed to query available provider packages
Error: vision_test.go:803: log: [ERROR] Could not retrieve the list of available versions for provider
Error: vision_test.go:803: log: [ERROR] hashicorp/null: could not connect to registry.terraform.io: failed to request
Error: vision_test.go:803: log: [ERROR] discovery document: Get
Error: vision_test.go:803: log: [ERROR] "https://registry.terraform.io/.well-known/terraform.json": context deadline
Error: vision_test.go:803: log: [ERROR] exceeded
    t.go:106: 2025-01-27 07:07:03.674 [debu]  executor: command done  args="[init -no-color -input=false]"  error="exit status 1"
    t.go:106: 2025-01-27 07:07:03.674 [debu]  executor: closing writers  error="exit status 1"
    t.go:106: 2025-01-27 07:07:03.674 [debu]  init failed  error="exit status 1"
    t.go:106: 2025-01-27 07:07:03.674 [debu]  canceledOrComplete closed
    t.go:106: 2025-01-27 07:07:03.674 [debu]  executor: kill context ended  args="[/Users/runner/work/_temp/04f2bf78-299d-4436-8e4e-d964e2f76538/terraform init -no-color -input=false]"
    provision_test.go:819: 
        	Error Trace:	/Users/runner/work/coder/coder/provisioner/terraform/provision_test.go:819
        	Error:      	Not equal: 
        	            	expected: ""
        	            	actual  : "initialize terraform: exit status 1"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-
        	            	+initialize terraform: exit status 1
        	Test:       	TestProvision/returns-modules

https://github.com/coder/coder/actions/runs/12983674870/job/36205319350?pr=16265

Looks like this test requires reaching out to registry.terraform.io, which is generally a bad idea in unit tests, since it subjects us to network issues, throttling/limits, etc.

[spikecurtis]

https://github.com/coder/coder/actions/runs/14304126462/job/40084286292

Another example

[hugodutka]

Sorry for the delay, I completely forgot about this one.

The root cause of the issue seems to be terraform downloading providers over and over again. In the first error a connection to registry.terraform.io failed, in the second - a connection to github.com.

I think we could instead download the providers once and cache them between CI runs. I'd enforce that terraform init only uses a local registry mirror by using a provider_installation block in the Terraform CLI config for the tests:

provider_installation {
  filesystem_mirror {
    path    = "/path/to/local/providers"
    include = ["*/*"]
  }
  direct {
    exclude = ["*/*"]
  }
}

Each test would try to add its providers to the local mirror if it hasn't already while holding a system-wide lock with this command:

terraform providers mirror /path/to/local/providers

This approach doesn't completely remove the dependency on the external registry, but it dramatically reduces the number of network requests required. In fact, if terraform tests don't change between runs, no new network requests should be made at all. Do you see any issues with this approach, @spikecurtis? If not, I'll implement it this week.

[spikecurtis]

Is there any way we can completely remove the dependency on the external registry? Unit tests like this should be fast, self-contained, and bullet proof.

[hugodutka]

@spikecurtis From what I can see many of the TestProvider tests depend on the coder provider and check whether it behaves correctly. I don't think there's a way to remove the dependency without deleting the tests completely.

[spikecurtis]

Let's start with caching between runs in CI, and see if the flakes go away.

[hugodutka]

I've implemented the coder-side caching logic in coder/coder#17373. I'm still working out the caching strategy in GitHub Actions. I'll have an update soon, and then the PR will be ready for review.

[hugodutka]

Closing the issue since coder/coder#17373 should address it. If the problem resurfaces, we can reopen it.