Tests are executed 5 times each time

[gerbil]

terraform-plugin-testing version

v1.5.1

Relevant provider source code

package opswatProvider

import (
	"github.com/hashicorp/terraform-plugin-testing/helper/resource"
	"regexp"
	"testing"
)

func TestAccFileSyncDataSource(t *testing.T) {
	resource.Test(t, resource.TestCase{
		ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
		Steps: []resource.TestStep{
			// Read testing
			{
				Config: providerConfig + configFileSyncDataSource,
				Check: resource.ComposeTestCheckFunc(
					// Verify value returned
					resource.TestMatchResourceAttr("data.opswat_file_sync.test", "timeout", regexp.MustCompile("[0-9]+")),
				),
			},
		},
	})
}

const configFileSyncDataSource = `
data "opswat_file_sync" "test" {}
output "opswat_file_sync" {
  value = data.opswat_file_sync.test
}
`

Terraform Configuration Files

data "XXX" "test" {}
output XXX" {
  value = data.XXX.test
}

Expected Behavior

Test should be executed only once

Actual Behavior

Test was executed 5 times (based on fmt.Println in GetGlobalSync() func)

$ TF_ACC=1 go test ./internal/test/ -v
=== RUN TestAccFileSyncDataSource
request URL: https://xxx/admin/config/file/sync
request URL: https://xxx/admin/config/file/sync
request URL: https://xxx/admin/config/file/sync
request URL: https://xxx/admin/config/file/sync
request URL: https://xxx/admin/config/file/sync
--- PASS: TestAccFileSyncDataSource (21.23s)
PASS
ok terraform-provider-opswat/internal/test 23.468s

Steps to Reproduce

git clone https://github.com/gerbil/terraform-provider-opswat
$ TF_ACC=1 go test ./internal/test/ -v

[gerbil]

Btw applicable only to datasource tests

[bflad]

Hi @gerbil 👋 Thank you for submitting this issue.

This testing logic behavior is certainly confusing as the testing logic currently abstracts a few different Terraform commands per test step. It is not that the test itself is running multiple times, but the data source read is being invoked multiple times. This design choice was mainly to cover provider developer gotchas that were present in older versions of Terraform itself and due to unfortunate behaviors caused simply by using the prior provider development SDK, terraform-plugin-sdk. Essentially a Config* test step today runs the following commands: refresh, plan -refresh=false, apply -refresh=false, plan -refresh=false, plan. Modern versions of Terraform may be seeing the data source being referenced in the output value and forcing the data source refresh. You could try removing the output to see if it reduces the number of data source read invocations, as you can just directly reference data source attributes for value assertions.

One thing we could potentially do right now is remove the explicit refresh command invocations, since Terraform's recommendations and implementation details have shifted to recommend planning with refresh directly. We might also be able to introduce some special configuration-based heuristics to see if a configuration contains no managed resources. Given that the current behavior is potentially relied on by existing provider testing, we would need to be careful to not break those tests though. In the next major version, we may look at paring down or removing those abstractions so its more obvious what the testing logic is going to do as well.

Thanks again for the report.

[bflad]

Hi again @gerbil 👋

terraform-plugin-testing v1.6.0 removed separate terraform refresh invocations so you should notice less refresh calls to your data source. As mentioned, removing the explicit output configuration block might also help skip calls to your data source with certain versions of Terraform.

Beyond this, any more changes within the testing logic to help with this would likely be fairly complex to implement since it means parsing Terraform configurations, such as detecting data source only configurations. Are you okay with the current situation and explanations to close this?

[gerbil]

Sure, thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.