Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ec2: fix unretriable error when creating vpc, while reading vpc attributes #31877

Merged
merged 4 commits into from
Jun 12, 2023

Conversation

FabianPonce
Copy link
Contributor

@FabianPonce FabianPonce commented Jun 9, 2023

Description

We run Terraform in CI/CD and there appears to be a race condition in VPC creation. While the provider correctly waits up to 2 minutes for the VPC to be created and return an existant resource, the VPC attributes are assumed to return successfully. I regularly see the following output when creating VPCs:

Error: reading EC2 VPC (vpc-abc123) Attribute (enableDnsSupport): couldn't find resource

Re-attempting the Terraform run fixes the issue, but this is suboptimal. This PR treats the VPC attributes to the same retry logic as the VPC reading.

Relations

References

Output from Acceptance Testing

make testacc TESTS=TestAccVPC_ PKG=ec2                     SIGINT(2) ↵  5315  13:46:28 
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/ec2/... -v -count 1 -parallel 20 -run='TestAccVPC_'  -timeout 180m
=== RUN   TestAccVPC_basic
=== PAUSE TestAccVPC_basic
=== RUN   TestAccVPC_disappears
=== PAUSE TestAccVPC_disappears
=== RUN   TestAccVPC_tags
=== PAUSE TestAccVPC_tags
=== RUN   TestAccVPC_tags_computed
=== PAUSE TestAccVPC_tags_computed
=== RUN   TestAccVPC_tags_null
=== PAUSE TestAccVPC_tags_null
=== RUN   TestAccVPC_DefaultTags_zeroValue
=== PAUSE TestAccVPC_DefaultTags_zeroValue
=== RUN   TestAccVPC_DefaultTags_providerOnlyTestAccVPC_DefaultTags_providerOnly
=== PAUSE TestAccVPC_DefaultTags_providerOnlyTestAccVPC_DefaultTags_providerOnly
=== RUN   TestAccVPC_DefaultTags_updateToProviderOnly
=== PAUSE TestAccVPC_DefaultTags_updateToProviderOnly
=== RUN   TestAccVPC_DefaultTags_updateToResourceOnly
=== PAUSE TestAccVPC_DefaultTags_updateToResourceOnly
=== RUN   TestAccVPC_DefaultTagsProviderAndResource_nonOverlappingTag
=== PAUSE TestAccVPC_DefaultTagsProviderAndResource_nonOverlappingTag
=== RUN   TestAccVPC_DefaultTagsProviderAndResource_overlappingTag
=== PAUSE TestAccVPC_DefaultTagsProviderAndResource_overlappingTag
=== RUN   TestAccVPC_DefaultTagsProviderAndResource_duplicateTag
=== PAUSE TestAccVPC_DefaultTagsProviderAndResource_duplicateTag
=== RUN   TestAccVPC_DefaultTagsProviderAndResource_moveDuplicateTags
=== PAUSE TestAccVPC_DefaultTagsProviderAndResource_moveDuplicateTags
=== RUN   TestAccVPC_DynamicResourceTagsMergedWithLocals_ignoreChanges
=== PAUSE TestAccVPC_DynamicResourceTagsMergedWithLocals_ignoreChanges
=== RUN   TestAccVPC_DynamicResourceTags_ignoreChanges
=== PAUSE TestAccVPC_DynamicResourceTags_ignoreChanges
=== RUN   TestAccVPC_defaultAndIgnoreTags
=== PAUSE TestAccVPC_defaultAndIgnoreTags
=== RUN   TestAccVPC_ignoreTags
=== PAUSE TestAccVPC_ignoreTags
=== RUN   TestAccVPC_tenancy
=== PAUSE TestAccVPC_tenancy
=== RUN   TestAccVPC_updateDNSHostnames
=== PAUSE TestAccVPC_updateDNSHostnames
=== RUN   TestAccVPC_bothDNSOptionsSet
=== PAUSE TestAccVPC_bothDNSOptionsSet
=== RUN   TestAccVPC_disabledDNSSupport
=== PAUSE TestAccVPC_disabledDNSSupport
=== RUN   TestAccVPC_enableNetworkAddressUsageMetrics
=== PAUSE TestAccVPC_enableNetworkAddressUsageMetrics
=== RUN   TestAccVPC_assignGeneratedIPv6CIDRBlock
=== PAUSE TestAccVPC_assignGeneratedIPv6CIDRBlock
=== RUN   TestAccVPC_assignGeneratedIPv6CIDRBlockWithNetworkBorderGroup
=== PAUSE TestAccVPC_assignGeneratedIPv6CIDRBlockWithNetworkBorderGroup
=== RUN   TestAccVPC_IPAMIPv4BasicNetmask
=== PAUSE TestAccVPC_IPAMIPv4BasicNetmask
=== RUN   TestAccVPC_IPAMIPv4BasicExplicitCIDR
=== PAUSE TestAccVPC_IPAMIPv4BasicExplicitCIDR
=== RUN   TestAccVPC_IPAMIPv6
=== PAUSE TestAccVPC_IPAMIPv6
=== CONT  TestAccVPC_basic
=== CONT  TestAccVPC_DynamicResourceTags_ignoreChanges
=== CONT  TestAccVPC_IPAMIPv6
=== CONT  TestAccVPC_DefaultTags_updateToProviderOnly
=== CONT  TestAccVPC_updateDNSHostnames
=== CONT  TestAccVPC_IPAMIPv4BasicNetmask
=== CONT  TestAccVPC_IPAMIPv4BasicExplicitCIDR
=== CONT  TestAccVPC_DefaultTags_providerOnlyTestAccVPC_DefaultTags_providerOnly
=== CONT  TestAccVPC_ignoreTags
=== CONT  TestAccVPC_DefaultTagsProviderAndResource_nonOverlappingTag
=== CONT  TestAccVPC_DefaultTagsProviderAndResource_duplicateTag
=== CONT  TestAccVPC_DynamicResourceTagsMergedWithLocals_ignoreChanges
=== CONT  TestAccVPC_DefaultTagsProviderAndResource_moveDuplicateTags
=== CONT  TestAccVPC_defaultAndIgnoreTags
=== CONT  TestAccVPC_disabledDNSSupport
=== CONT  TestAccVPC_tags_computed
=== CONT  TestAccVPC_DefaultTags_zeroValue
=== CONT  TestAccVPC_enableNetworkAddressUsageMetrics
=== CONT  TestAccVPC_DefaultTagsProviderAndResource_overlappingTag
=== CONT  TestAccVPC_tenancy
=== NAME  TestAccVPC_IPAMIPv6
    vpc_test.go:1079: Step 1/1 error: Error running apply: exit status 1
        
        Error: creating IPAM: ResourceLimitExceeded: You've reached the limit for ipams. You have created 3 ipams and you are limited to 1.
        	status code: 400, request id: 82f2130c-d6db-4a04-8fa6-37c8a64629e8
        
          with aws_vpc_ipam.test,
          on terraform_plugin_test.tf line 4, in resource "aws_vpc_ipam" "test":
           4: resource "aws_vpc_ipam" "test" {
        
=== NAME  TestAccVPC_IPAMIPv4BasicNetmask
    vpc_test.go:1023: Step 1/1 error: Error running apply: exit status 1
        
        Error: creating IPAM: ResourceLimitExceeded: You've reached the limit for ipams. You have created 3 ipams and you are limited to 1.
        	status code: 400, request id: d5fdf049-42bc-4a33-b392-981110f14209
        
          with aws_vpc_ipam.test,
          on terraform_plugin_test.tf line 4, in resource "aws_vpc_ipam" "test":
           4: resource "aws_vpc_ipam" "test" {
        
=== NAME  TestAccVPC_IPAMIPv4BasicExplicitCIDR
    vpc_test.go:1051: Step 1/1 error: Error running apply: exit status 1
        
        Error: creating IPAM: ResourceLimitExceeded: You've reached the limit for ipams. You have created 3 ipams and you are limited to 1.
        	status code: 400, request id: ffbd9af2-91ad-43e0-ad13-c6a8ec64634b
        
          with aws_vpc_ipam.test,
          on terraform_plugin_test.tf line 4, in resource "aws_vpc_ipam" "test":
           4: resource "aws_vpc_ipam" "test" {
        
--- FAIL: TestAccVPC_IPAMIPv6 (16.82s)
=== CONT  TestAccVPC_tags_null
--- FAIL: TestAccVPC_IPAMIPv4BasicNetmask (18.55s)
=== CONT  TestAccVPC_bothDNSOptionsSet
--- FAIL: TestAccVPC_IPAMIPv4BasicExplicitCIDR (19.72s)
=== CONT  TestAccVPC_DefaultTags_updateToResourceOnly
--- PASS: TestAccVPC_DefaultTagsProviderAndResource_duplicateTag (38.36s)
=== CONT  TestAccVPC_assignGeneratedIPv6CIDRBlockWithNetworkBorderGroup
    acctest.go:848: skipping tests; AWS_DEFAULT_REGION (us-east-1) not supported. Supported: [us-west-2]
--- SKIP: TestAccVPC_assignGeneratedIPv6CIDRBlockWithNetworkBorderGroup (0.00s)
=== CONT  TestAccVPC_assignGeneratedIPv6CIDRBlock
--- PASS: TestAccVPC_tags_computed (39.47s)
=== CONT  TestAccVPC_tags
--- PASS: TestAccVPC_basic (46.06s)
=== CONT  TestAccVPC_disappears
--- PASS: TestAccVPC_tags_null (33.99s)
--- PASS: TestAccVPC_enableNetworkAddressUsageMetrics (55.63s)
--- PASS: TestAccVPC_disabledDNSSupport (55.72s)
--- PASS: TestAccVPC_DynamicResourceTags_ignoreChanges (64.24s)
--- PASS: TestAccVPC_DynamicResourceTagsMergedWithLocals_ignoreChanges (65.12s)
--- PASS: TestAccVPC_ignoreTags (67.90s)
--- PASS: TestAccVPC_bothDNSOptionsSet (49.81s)
--- PASS: TestAccVPC_updateDNSHostnames (69.03s)
--- PASS: TestAccVPC_DefaultTags_updateToProviderOnly (69.68s)
--- PASS: TestAccVPC_defaultAndIgnoreTags (70.57s)
--- PASS: TestAccVPC_disappears (25.62s)
--- PASS: TestAccVPC_DefaultTags_updateToResourceOnly (57.92s)
--- PASS: TestAccVPC_DefaultTagsProviderAndResource_moveDuplicateTags (79.28s)
--- PASS: TestAccVPC_DefaultTags_zeroValue (80.41s)
--- PASS: TestAccVPC_DefaultTagsProviderAndResource_overlappingTag (82.74s)
--- PASS: TestAccVPC_DefaultTagsProviderAndResource_nonOverlappingTag (83.11s)
--- PASS: TestAccVPC_DefaultTags_providerOnlyTestAccVPC_DefaultTags_providerOnly (84.22s)
--- PASS: TestAccVPC_tenancy (85.15s)
--- PASS: TestAccVPC_tags (58.36s)
--- PASS: TestAccVPC_assignGeneratedIPv6CIDRBlock (82.98s)
FAIL
FAIL	github.com/hashicorp/terraform-provider-aws/internal/service/ec2	121.617s
FAIL
make: *** [testacc] Error 1

I will re-edit the above tests once I have time to run in sequence since the maximum IPAMs is 1 per region.

@github-actions
Copy link

github-actions bot commented Jun 9, 2023

Community Note

Voting for Prioritization

  • Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
  • Please see our prioritization guide for information on how we prioritize.
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

  • Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
  • For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
  • Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

@github-actions github-actions bot added size/XS Managed by automation to categorize the size of a PR. service/vpc Issues and PRs that pertain to the vpc service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. needs-triage Waiting for first response or review from a maintainer. labels Jun 9, 2023

Verified

This commit was signed with the committer’s verified signature.
ewbankkit Kit Ewbank
@justinretzolk justinretzolk added bug Addresses a defect in current functionality. and removed needs-triage Waiting for first response or review from a maintainer. labels Jun 9, 2023
@ewbankkit ewbankkit added the eventual-consistency Pertains to eventual consistency issues. label Jun 12, 2023

Verified

This commit was signed with the committer’s verified signature.
ewbankkit Kit Ewbank

Verified

This commit was signed with the committer’s verified signature.
ewbankkit Kit Ewbank
@github-actions github-actions bot removed the tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. label Jun 12, 2023
Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀.

% make testacc TESTARGS='-short -run=TestAccVPC_' PKG=ec2 ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/ec2/... -v -count 1 -parallel 2  -short -run=TestAccVPC_ -timeout 180m
=== RUN   TestAccVPC_basic
=== PAUSE TestAccVPC_basic
=== RUN   TestAccVPC_disappears
=== PAUSE TestAccVPC_disappears
=== RUN   TestAccVPC_tags
=== PAUSE TestAccVPC_tags
=== RUN   TestAccVPC_tags_computed
=== PAUSE TestAccVPC_tags_computed
=== RUN   TestAccVPC_tags_null
=== PAUSE TestAccVPC_tags_null
=== RUN   TestAccVPC_DefaultTags_zeroValue
=== PAUSE TestAccVPC_DefaultTags_zeroValue
=== RUN   TestAccVPC_DefaultTags_providerOnlyTestAccVPC_DefaultTags_providerOnly
=== PAUSE TestAccVPC_DefaultTags_providerOnlyTestAccVPC_DefaultTags_providerOnly
=== RUN   TestAccVPC_DefaultTags_updateToProviderOnly
=== PAUSE TestAccVPC_DefaultTags_updateToProviderOnly
=== RUN   TestAccVPC_DefaultTags_updateToResourceOnly
=== PAUSE TestAccVPC_DefaultTags_updateToResourceOnly
=== RUN   TestAccVPC_DefaultTagsProviderAndResource_nonOverlappingTag
=== PAUSE TestAccVPC_DefaultTagsProviderAndResource_nonOverlappingTag
=== RUN   TestAccVPC_DefaultTagsProviderAndResource_overlappingTag
=== PAUSE TestAccVPC_DefaultTagsProviderAndResource_overlappingTag
=== RUN   TestAccVPC_DefaultTagsProviderAndResource_duplicateTag
=== PAUSE TestAccVPC_DefaultTagsProviderAndResource_duplicateTag
=== RUN   TestAccVPC_DefaultTagsProviderAndResource_moveDuplicateTags
=== PAUSE TestAccVPC_DefaultTagsProviderAndResource_moveDuplicateTags
=== RUN   TestAccVPC_DynamicResourceTagsMergedWithLocals_ignoreChanges
=== PAUSE TestAccVPC_DynamicResourceTagsMergedWithLocals_ignoreChanges
=== RUN   TestAccVPC_DynamicResourceTags_ignoreChanges
=== PAUSE TestAccVPC_DynamicResourceTags_ignoreChanges
=== RUN   TestAccVPC_defaultAndIgnoreTags
=== PAUSE TestAccVPC_defaultAndIgnoreTags
=== RUN   TestAccVPC_ignoreTags
=== PAUSE TestAccVPC_ignoreTags
=== RUN   TestAccVPC_tenancy
=== PAUSE TestAccVPC_tenancy
=== RUN   TestAccVPC_updateDNSHostnames
=== PAUSE TestAccVPC_updateDNSHostnames
=== RUN   TestAccVPC_bothDNSOptionsSet
=== PAUSE TestAccVPC_bothDNSOptionsSet
=== RUN   TestAccVPC_disabledDNSSupport
=== PAUSE TestAccVPC_disabledDNSSupport
=== RUN   TestAccVPC_enableNetworkAddressUsageMetrics
=== PAUSE TestAccVPC_enableNetworkAddressUsageMetrics
=== RUN   TestAccVPC_assignGeneratedIPv6CIDRBlock
=== PAUSE TestAccVPC_assignGeneratedIPv6CIDRBlock
=== RUN   TestAccVPC_assignGeneratedIPv6CIDRBlockWithNetworkBorderGroup
=== PAUSE TestAccVPC_assignGeneratedIPv6CIDRBlockWithNetworkBorderGroup
=== RUN   TestAccVPC_IPAMIPv4BasicNetmask
    vpc_test.go:1015: skipping long-running test in short mode
--- SKIP: TestAccVPC_IPAMIPv4BasicNetmask (0.00s)
=== RUN   TestAccVPC_IPAMIPv4BasicExplicitCIDR
    vpc_test.go:1042: skipping long-running test in short mode
--- SKIP: TestAccVPC_IPAMIPv4BasicExplicitCIDR (0.00s)
=== RUN   TestAccVPC_IPAMIPv6
    vpc_test.go:1070: skipping long-running test in short mode
--- SKIP: TestAccVPC_IPAMIPv6 (0.00s)
=== CONT  TestAccVPC_basic
=== CONT  TestAccVPC_DefaultTagsProviderAndResource_moveDuplicateTags
--- PASS: TestAccVPC_basic (27.67s)
=== CONT  TestAccVPC_updateDNSHostnames
--- PASS: TestAccVPC_DefaultTagsProviderAndResource_moveDuplicateTags (53.21s)
=== CONT  TestAccVPC_assignGeneratedIPv6CIDRBlockWithNetworkBorderGroup
--- PASS: TestAccVPC_updateDNSHostnames (48.53s)
=== CONT  TestAccVPC_assignGeneratedIPv6CIDRBlock
--- PASS: TestAccVPC_assignGeneratedIPv6CIDRBlockWithNetworkBorderGroup (76.46s)
=== CONT  TestAccVPC_enableNetworkAddressUsageMetrics
--- PASS: TestAccVPC_enableNetworkAddressUsageMetrics (36.60s)
=== CONT  TestAccVPC_disabledDNSSupport
--- PASS: TestAccVPC_assignGeneratedIPv6CIDRBlock (90.53s)
=== CONT  TestAccVPC_bothDNSOptionsSet
--- PASS: TestAccVPC_disabledDNSSupport (36.59s)
=== CONT  TestAccVPC_defaultAndIgnoreTags
--- PASS: TestAccVPC_bothDNSOptionsSet (36.85s)
=== CONT  TestAccVPC_tenancy
--- PASS: TestAccVPC_defaultAndIgnoreTags (44.51s)
=== CONT  TestAccVPC_ignoreTags
--- PASS: TestAccVPC_tenancy (63.69s)
=== CONT  TestAccVPC_DynamicResourceTags_ignoreChanges
--- PASS: TestAccVPC_ignoreTags (46.71s)
=== CONT  TestAccVPC_DefaultTags_providerOnlyTestAccVPC_DefaultTags_providerOnly
--- PASS: TestAccVPC_DynamicResourceTags_ignoreChanges (45.61s)
=== CONT  TestAccVPC_DefaultTagsProviderAndResource_duplicateTag
--- PASS: TestAccVPC_DefaultTagsProviderAndResource_duplicateTag (19.19s)
=== CONT  TestAccVPC_DefaultTagsProviderAndResource_overlappingTag
--- PASS: TestAccVPC_DefaultTags_providerOnlyTestAccVPC_DefaultTags_providerOnly (55.91s)
=== CONT  TestAccVPC_DefaultTagsProviderAndResource_nonOverlappingTag
--- PASS: TestAccVPC_DefaultTagsProviderAndResource_overlappingTag (56.82s)
=== CONT  TestAccVPC_DefaultTags_updateToResourceOnly
--- PASS: TestAccVPC_DefaultTagsProviderAndResource_nonOverlappingTag (59.13s)
=== CONT  TestAccVPC_DefaultTags_updateToProviderOnly
--- PASS: TestAccVPC_DefaultTags_updateToResourceOnly (42.87s)
=== CONT  TestAccVPC_DynamicResourceTagsMergedWithLocals_ignoreChanges
--- PASS: TestAccVPC_DefaultTags_updateToProviderOnly (46.63s)
=== CONT  TestAccVPC_tags_computed
--- PASS: TestAccVPC_DynamicResourceTagsMergedWithLocals_ignoreChanges (45.76s)
=== CONT  TestAccVPC_DefaultTags_zeroValue
--- PASS: TestAccVPC_tags_computed (24.80s)
=== CONT  TestAccVPC_tags_null
--- PASS: TestAccVPC_tags_null (21.69s)
=== CONT  TestAccVPC_tags
--- PASS: TestAccVPC_DefaultTags_zeroValue (50.86s)
=== CONT  TestAccVPC_disappears
--- PASS: TestAccVPC_disappears (17.79s)
--- PASS: TestAccVPC_tags (59.24s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/ec2	567.455s

@ewbankkit
Copy link
Contributor

@FabianPonce Thanks for the contribution 🎉 👏.
We had been seeing those errors occasionally in our own CI.

@ewbankkit ewbankkit merged commit 7eb5e04 into hashicorp:main Jun 12, 2023
@github-actions github-actions bot added this to the v5.3.0 milestone Jun 12, 2023
@github-actions
Copy link

This functionality has been released in v5.3.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 14, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. eventual-consistency Pertains to eventual consistency issues. service/vpc Issues and PRs that pertain to the vpc service. size/XS Managed by automation to categorize the size of a PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants