SDKv2 - Providers must only be specified either at the TestCase or TestStep level

[rios0rios0]

Module version

github.com/hashicorp/terraform-plugin-docs v0.19.4
github.com/hashicorp/terraform-plugin-framework v1.13.0
github.com/hashicorp/terraform-plugin-go v0.25.0
github.com/hashicorp/terraform-plugin-log v0.9.0
github.com/hashicorp/terraform-plugin-testing v1.10.0
github.com/ohler55/ojg v1.25.0
github.com/stretchr/testify v1.8.3

Relevant provider source code

var (
	/*
		Is this factory is barely used to create the block "terraform.required_providers" in the Terraform configuration??? It doesn't need for anything else?
	*/
	testAccProtoV6ProviderFactories = map[string]func() (tfprotov6.ProviderServer, error){
		"http": providerserver.NewProtocol6WithError(New("test")()),
	}
)

func TestHTTPProvider(t *testing.T) {
	t.Parallel()

	t.Run("should return an error when the URL is empty", func(t *testing.T) {
		resource.UnitTest(t, resource.TestCase{
			PreCheck:                 func() { testAccPreCheck(t) },
			ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
			Steps: []resource.TestStep{
				{
					ConfigDirectory: config.TestNameDirectory(),
					ExpectError: regexp.MustCompile("Get \"posts/1\": unsupported protocol scheme \"\""),
				},
			},
		})
	})
...

Terraform Configuration Files

File internal/provider/testdata/TestHTTPProvider/should_return_an_error_when_the_URL_is_empty/main.tf

resource "http_request" "test1" {
  method = "GET"
  path   = "/posts/1"
}

Debug Output

=== RUN   TestHTTPProvider
=== PAUSE TestHTTPProvider
=== CONT  TestHTTPProvider
=== RUN   TestHTTPProvider/should_return_an_error_when_the_URL_is_empty
    provider_test.go:47: Test validation error: TestStep 1/1 validation error: Providers must only be specified either at the TestCase or TestStep level
--- FAIL: TestHTTPProvider/should_return_an_error_when_the_URL_is_empty (13.88s)

--- FAIL: TestHTTPProvider (13.89s)

FAIL

Debugger finished with the exit code 0

Expected Behavior

The factory should be creating the provider's block, according to the Framework SDK? I'm the developer of this provider, and on the internet, there's no example. Even in the HashiCorp docs, even in the HashiCups project, and don't even in the Scaffolding. So, I'm not sure if I'm doing it rightly, but it's very inconsistent behavior. The final behavior should be creating the provider automatically trying to apply the resource and giving me the error of missing URL. Interesting that, if I use just Config parameter, I can bypass this "check".

Actual Behavior

It just trigger the error presented. And if I remove the provider factory, it'll give another error.

Steps to Reproduce

Just run the tests, any tests with the examples online using the structure I suggested. Doesn't matter to be honest, if you use the ConfigDirectory parameter, you are not going to be able to run the tests. You have to have another kind of provider factory which I don't know what is that.

References

• https://github.com/hashicorp/terraform-provider-scaffolding-framework/blob/main/internal/provider/provider_test.go#L18
• https://github.com/rios0rios0/terraform-provider-http/blob/main/internal/provider/provider_test.go#L51
• https://developer.hashicorp.com/terraform/plugin/framework/acctests#protocol-version-6
• https://developer.hashicorp.com/terraform/plugin/sdkv2/testing/acceptance-tests
• https://developer.hashicorp.com/terraform/plugin/sdkv2

[austinvalle]

Hey there @rios0rios0 👋🏻 , thanks for reporting the issue and sorry you're running into trouble here.

The main usage of the provider factories (like ProtoV6ProviderFactories) is to spin up your local provider server and setup the reattach configuration that the testing framework uses when executing terraform commands. Unfortunately, as you noted, there is some inconsistent behavior we have to keep for compatibility, where provider factories will also add the terraform / required_providers block, as well as the provider block itself in some instances (like when you're using the Config field).

I think the validation stopping you from using TestCase.ProtoV6ProviderFactories seems a little too strict, especially since it's only being triggered because of the presence of the provider block.... but I can't recall at the moment if that was introduced purposefully in #153 . IMO, we should be able to relax that, but will need to dig in deeper.

In the meantime, you should be able to move the provider factories to your test steps to proceed:

func TestHTTPProvider(t *testing.T) {
    t.Parallel()

    t.Run("should return an error when the URL is empty", func(t *testing.T) {
        resource.UnitTest(t, resource.TestCase{
            PreCheck: func() { testAccPreCheck(t) },
            Steps: []resource.TestStep{
                {
                    // => moved from TestCase
                    ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
                    ConfigDirectory:          config.TestNameDirectory(),
                    ExpectError:              regexp.MustCompile("Unknown URL for HTTP client"),
                },
            },
        })
    })

    // .. remainder of test
}

You'll also need to include the required_providers block in your configuration, so the final config at internal/provider/testdata/TestHTTPProvider/should_return_an_error_when_the_URL_is_empty/main.tf would look like:

terraform {
  required_providers {
    http = {
      source = "rios0rios0/http"
    }
  }
}

provider "http" {
  url = ""
}

resource "http_request" "test1" {
  method = "GET"
  path   = "/posts/1"
}