New data source 'azurerm_resources' #3529

tiwood · 2019-05-27T13:47:30Z

This PR introduces the new data source 'azurerm_resources'.

This data source can be used to get a list of resources which match specified criteria.

Get resources from a specific group
Get resources with one or more specific tags
Get all resources from a subscription

All attributes can be combined to fine tune the results.

The resulting list can be used in subsequent resources.

New Data source 'azurerm_resourcew'
Add tests
Add documentation

Example Usage

// Get Resources from a Resource Group
data "azurerm_resources" "test" {
  resource_group_name = "myResourceGroup"
}

// Get Resources with specific Tags
data "azurerm_resources" "test" {
  required_tags {
    environment = "production"
    role        = "webserver"
  }
}

Argument Reference

name - (Optional) The name of the Resource.
resource_group_name - (Optional) The name of the Resource group where the Resources are located.
type - (Optional) The Resource Type of the Resources you want to list (e.g. Microsoft.Network/virtualNetworks). A full list of available Resource Types can be found here.
required_tags - (Optional) A mapping of tags which the Resource has to have in order to be included in the result.

Attributes Reference

resources - One or more resource blocks as defined below.

The resource block contains:

name - The name of this resource.
id - The Resource ID of this resource.
type - The type of this resoource.
location - The location of this resource.
tags - The type of resource that this is, such as Microsoft.Network/virtualNetworks.

tiwood · 2019-05-27T15:48:55Z

It seems tests sometimes fail because the resources I try to get with the data sources are deleted before the data source ran or the data source ran before the resources are created. Any ideas why that happens?

tiwood · 2019-10-08T15:40:23Z

Tests pass.

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor'|grep -v 'examples') -v -run=TestAccDataSourceAzureRMResource_* -timeout 180m -count 1 -ldflags="-X=github.com/terraform-providers/terraform-provider-azurerm/version.ProviderVersion=acc"
?   	github.com/terraform-providers/terraform-provider-azurerm	[no test files]
=== RUN   TestAccDataSourceAzureRMResourceGroup_basic
=== PAUSE TestAccDataSourceAzureRMResourceGroup_basic
=== RUN   TestAccDataSourceAzureRMResource_ByResourceID
=== PAUSE TestAccDataSourceAzureRMResource_ByResourceID
=== RUN   TestAccDataSourceAzureRMResource_ByName
=== PAUSE TestAccDataSourceAzureRMResource_ByName
=== RUN   TestAccDataSourceAzureRMResource_ByResourceGroup
=== PAUSE TestAccDataSourceAzureRMResource_ByResourceGroup
=== RUN   TestAccDataSourceAzureRMResource_ByResourceType
--- PASS: TestAccDataSourceAzureRMResource_ByResourceType (118.21s)
=== RUN   TestAccDataSourceAzureRMResource_FilteredByTags
=== PAUSE TestAccDataSourceAzureRMResource_FilteredByTags
=== CONT  TestAccDataSourceAzureRMResourceGroup_basic
=== CONT  TestAccDataSourceAzureRMResource_ByResourceGroup
=== CONT  TestAccDataSourceAzureRMResource_ByResourceID
=== CONT  TestAccDataSourceAzureRMResource_ByName
=== CONT  TestAccDataSourceAzureRMResource_FilteredByTags
--- PASS: TestAccDataSourceAzureRMResourceGroup_basic (86.27s)
--- PASS: TestAccDataSourceAzureRMResource_FilteredByTags (107.27s)
--- PASS: TestAccDataSourceAzureRMResource_ByResourceGroup (109.61s)
--- PASS: TestAccDataSourceAzureRMResource_ByName (112.08s)
--- PASS: TestAccDataSourceAzureRMResource_ByResourceID (119.29s)
PASS
ok

@tombuildsstuff @katbyte, can you review this PR?

tombuildsstuff

hey @tiwood

Thanks for this PR - apologies for the delayed review here, we've been chatting about this internally trying to work out what's needed to get this in.

On the whole this looks pretty good, however there's a few things that need adjusting before we can merge this; in particular Terraform Core has a limit on the size of a Resource's State, as such we'll need to ensure we filter to a subset of the resources in the subscription, rather than storing the lot to ensure we don't hit this limit. That said, if we can fix those up this is otherwise looking good to me 👍

Thanks!

azurerm/data_source_resource.go

website/docs/d/resource.html.markdown

azurerm/data_source_resource.go

azurerm/provider.go

…ed docs

tiwood · 2019-10-09T15:23:00Z

@tombuildsstuff, thank you for the review.
I've updated the code as requested.

tombuildsstuff

hey @tiwood

Thanks for pushing those changes - taking a look through I noticed a few more minor issues but this otherwise LGTM 👍

Thanks!

azurerm/data_source_resources.go

website/docs/d/resources.html.markdown

katbyte

Thanks for the changes @tiwood,

However running it in our CI system it does seem like all the tests fail:

------- Stdout: -------
=== RUN   TestAccDataSourceAzureRMResources_ByName
=== PAUSE TestAccDataSourceAzureRMResources_ByName
=== CONT  TestAccDataSourceAzureRMResources_ByName
--- FAIL: TestAccDataSourceAzureRMResources_ByName (101.02s)
    testing.go:569: Step 0 error: Check failed: Check 1/1 error: data.azurerm_resources.test: Attribute 'resources.#' expected "1", got "0"
FAIL

------- Stderr: -------
2019/10/09 17:03:03 [DEBUG] Registering Data Sources for "Compute"..
2019/10/09 17:03:03 [DEBUG] Registering Resources for "Compute"..

tiwood · 2019-10-09T18:15:36Z

Thanks for the changes @tiwood,

However running it in our CI system it does seem like all the tests fail:

------- Stdout: -------
=== RUN   TestAccDataSourceAzureRMResources_ByName
=== PAUSE TestAccDataSourceAzureRMResources_ByName
=== CONT  TestAccDataSourceAzureRMResources_ByName
--- FAIL: TestAccDataSourceAzureRMResources_ByName (101.02s)
    testing.go:569: Step 0 error: Check failed: Check 1/1 error: data.azurerm_resources.test: Attribute 'resources.#' expected "1", got "0"
FAIL

------- Stderr: -------
2019/10/09 17:03:03 [DEBUG] Registering Data Sources for "Compute"..
2019/10/09 17:03:03 [DEBUG] Registering Resources for "Compute"..

I've just seen this happening on my side too. It seems we have a replication lag issue here, the API sometimes returns no results.

I can add a retry function to the data source or we can add a sleep to the test case (somehow?), but that dosen't sound right to me.

katbyte · 2019-10-09T18:20:24Z

🤔 given its a search index probably updating a retry in the resource makes less sense then a delay in the test steps.

tiwood · 2019-10-09T19:46:03Z

🤔 given its a search index probably updating a retry in the resource makes less sense then a delay in the test steps.

Any pointers how I could do that?

I've added some code to account for the replication lag/indexing, which seems to work now:

stateConf := &resource.StateChangeConf{
		Pending:                   []string{"ResourcesNotFound"},
		Target:                    []string{"ResourcesFound"},
		Refresh:                   dataArmResourcesStateStatusCodeRefreshFunc(ctx, meta, filter),
		Timeout:                   2 * time.Minute,
		MinTimeout:                30 * time.Second,
		ContinuousTargetOccurence: 2,
}

func dataArmResourcesStateStatusCodeRefreshFunc(ctx context.Context, meta interface{}, filter string) resource.StateRefreshFunc {
	return func() (interface{}, string, error) {
		resourcesClient := meta.(*ArmClient).Resource.ResourcesClient
		resp, err := resourcesClient.ListComplete(ctx, filter, "", nil)

		if err != nil {
			return nil, "", fmt.Errorf("Error while waiting for the resources to become available: %+v", err)
		}

		if resp.Value().ID == nil {
			return resp, "ResourcesNotFound", nil
		}

		return resp, "ResourcesFound", nil
	}
}

katbyte · 2019-10-09T23:08:53Z

Thanks! Give me a bit to discuss the best way forward internally and we'll get back to you 🙂

…ount for replication lag.

tiwood · 2019-10-11T15:15:11Z

I've pushed the changes which include waitForState() and the changes requested by @tombuildsstuff.

Please let me know if you want to put a delay in the tests rather than wait for state in the data source itself.

tombuildsstuff · 2019-10-14T07:11:30Z

hey @tiwood

Thanks for pushing those changes.

Taking a look through it appears the "replication issue" referred to above is a known issue in the test framework; where the Data Source will be called before the Resource has finished provisioning, unless there's an explicit depends_on. To work around this we're using two steps for provisioning the tests in the data sources, for example:

func TestAccDataSourceAzureRMResources_ByName(t *testing.T) {
	dataSourceName := "data.azurerm_resources.test"
	ri := tf.AccRandTimeInt()
	rs := acctest.RandString(4)
	location := testLocation()

	resource.ParallelTest(t, resource.TestCase{
		PreCheck:  func() { testAccPreCheck(t) },
		Providers: testAccProviders,
		Steps: []resource.TestStep{
			{
				Config: testAccDataSourceAzureRMStorageAccount_basic(ri, rs, location),
			},
			{
				Config: testAccDataSourceAzureRMResources_ByName(ri, rs, location),
				Check: resource.ComposeTestCheckFunc(
					resource.TestCheckResourceAttr(dataSourceName, "resources.#", "1"),
				),
			},
		},
	})
}

which means that the Resources will exist before the Data Source looks them up (we could also use a depends_on, but we're intentionally not since this is a test framework specific issue)

When updating the code to account for this (and removing the WaitForState check) the tests pass:

 $ acctests azurerm TestAccDataSourceAzureRMResources_ByName
=== RUN   TestAccDataSourceAzureRMResources_ByName
=== PAUSE TestAccDataSourceAzureRMResources_ByName
=== CONT  TestAccDataSourceAzureRMResources_ByName
--- PASS: TestAccDataSourceAzureRMResources_ByName (134.06s)
PASS
ok  	github.com/terraform-providers/terraform-provider-azurerm/azurerm	134.198s

Since I've made these changes locally to be able to confirm this I hope you don't mind but I'm going to push a commit with said change so that we can get this merged - but this otherwise LGTM 👍

Thanks!

dismissing since changes have been pushed

tombuildsstuff · 2019-10-14T07:40:39Z

Tests pass:

ghost · 2019-10-29T10:36:34Z

This has been released in version 1.36.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.36.0"
}
# ... other configuration ...

ghost · 2019-11-13T14:50:16Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

tiwood added 2 commits May 27, 2019 15:38

New data source azurerm_resource

99be441

Update provder conf to include new data source azurerm_resource

d96c682

ghost added size/XL documentation labels May 27, 2019

Refined the documentation to include some more info.

5247ec4

tiwood added 2 commits October 8, 2019 14:23

Merge branch 'master' into ds_arm_resource

e9640b3

Clean up after merge, updated code for Terraform 0.12

34da89e

Remove unused code; Formating

0237e63

tombuildsstuff self-assigned this Oct 9, 2019

tombuildsstuff suggested changes Oct 9, 2019

View reviewed changes

tombuildsstuff added new-data-source service/resources labels Oct 9, 2019

tombuildsstuff added this to the v1.36.0 milestone Oct 9, 2019

tombuildsstuff added the waiting-response label Oct 9, 2019

tombuildsstuff removed their assignment Oct 9, 2019

tiwood added 2 commits October 9, 2019 17:16

Added additional nil checks, renamed ds to 'azurerm_resources', updat…

988d40e

…ed docs

renamed files

01f1a60

tiwood changed the title ~~New data source 'azurerm_resource'~~ New data source 'azurerm_resources' Oct 9, 2019

ghost removed the waiting-response label Oct 9, 2019

tiwood requested a review from tombuildsstuff October 9, 2019 15:28

renamed files

86188a7

tombuildsstuff approved these changes Oct 9, 2019

View reviewed changes

katbyte previously requested changes Oct 9, 2019

View reviewed changes

Documentation changes; Added waitForState() to the data source to acc…

f1e5ef0

…ount for replication lag.

tombuildsstuff self-assigned this Oct 14, 2019

d/resources: provisioning the resources prior to the data source test

d248849

tombuildsstuff merged commit dddeb70 into hashicorp:master Oct 14, 2019

tombuildsstuff added a commit that referenced this pull request Oct 14, 2019

updating to include #3529

d4d6ed5

ghost locked and limited conversation to collaborators Nov 13, 2019

tiwood deleted the ds_arm_resource branch November 13, 2019 19:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New data source 'azurerm_resources' #3529

New data source 'azurerm_resources' #3529

tiwood commented May 27, 2019 •

edited

Loading

tiwood commented May 27, 2019 •

edited

Loading

tiwood commented Oct 8, 2019

tombuildsstuff left a comment

tiwood commented Oct 9, 2019

tombuildsstuff left a comment

katbyte left a comment

tiwood commented Oct 9, 2019

katbyte commented Oct 9, 2019

tiwood commented Oct 9, 2019

katbyte commented Oct 9, 2019

tiwood commented Oct 11, 2019 •

edited

Loading

tombuildsstuff commented Oct 14, 2019

tombuildsstuff commented Oct 14, 2019

ghost commented Oct 29, 2019

ghost commented Nov 13, 2019

New data source 'azurerm_resources' #3529

New data source 'azurerm_resources' #3529

Conversation

tiwood commented May 27, 2019 • edited Loading

Example Usage

Argument Reference

Attributes Reference

tiwood commented May 27, 2019 • edited Loading

tiwood commented Oct 8, 2019

tombuildsstuff left a comment

Choose a reason for hiding this comment

tiwood commented Oct 9, 2019

tombuildsstuff left a comment

Choose a reason for hiding this comment

katbyte left a comment

Choose a reason for hiding this comment

tiwood commented Oct 9, 2019

katbyte commented Oct 9, 2019

tiwood commented Oct 9, 2019

katbyte commented Oct 9, 2019

tiwood commented Oct 11, 2019 • edited Loading

tombuildsstuff commented Oct 14, 2019

tombuildsstuff commented Oct 14, 2019

ghost commented Oct 29, 2019

ghost commented Nov 13, 2019

tiwood commented May 27, 2019 •

edited

Loading

tiwood commented May 27, 2019 •

edited

Loading

tiwood commented Oct 11, 2019 •

edited

Loading