aws_sqs_queue times out on policy whitespace changes #22505

mentos1386 · 2022-01-10T17:37:08Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform version: 1.1.3

Provider versions

v3.71.0 - Timeout
v3.69.0 - Timeout
v3.68.0 - Works
v3.67.0 - Works
v3.66.0 - Works
v3.65.0 - Works

Affected Resource(s)

aws_sqs_queue
aws_sqs_queue_policy

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_sqs_queue" "queue" {
  name = var.name
  tags = var.tags
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dead_letter.arn
    maxReceiveCount     = 4
  })
  // sqs_managed_sse_enabled = true
  policy = jsonencode({
    Version = "2012-10-17",
    Id      = "sqspolicy",
    Statement = [
      {
        Sid       = "AllowSNSSubscriptions",
        Effect    = "Allow",
        Principal = "*",
        Action    = "sqs:SendMessage",
        Resource  = "arn:aws:sqs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:${var.name}",
        Condition = {
          ArnEquals = {
            "aws:SourceArn" = []
          }
        }
      }
    ]
  })
}

Which produces plan:

  ~ resource "aws_sqs_queue" "queue" {
        id                                = "https://sqs.us-east-1.amazonaws.com/redacted/redacted"
        name                              = "redacted"
      ~ policy                            = jsonencode( # whitespace changes
            {
                Id        = "sqspolicy"
                Statement = [
                    {
                        Action    = "sqs:SendMessage"
                        Condition = {
                            ArnEquals = {
                                aws:SourceArn = []
                            }
                        }
                        Effect    = "Allow"
                        Principal = "*"
                        Resource  = "redacted"
                        Sid       = "AllowSNSSubscriptions"
                    },
                ]
                Version   = "2012-10-17"
            }
        )
        tags                              = {
        // redacted
        }
        # (13 unchanged attributes hidden)
    }

Debug Output

  2022-01-10T17:13:23.022Z [INFO]  provider.terraform-provider-aws_v3.71.0_x5: 2022/01/10 17:13:23 [DEBUG] [aws-sdk-go] DEBUG: Response sqs/GetQueueAttributes Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 2028
Content-Type: text/xml
Date: Mon, 10 Jan 2022 17:13:23 GMT
X-Amzn-Requestid: 2baab053-89f2-57b1-b253-ea05e2806dcd

 2022-01-10T17:13:33.057Z [INFO]  provider.terraform-provider-aws_v3.71.0_x5: 2022/01/10 17:13:33 [DEBUG] [aws-sdk-go] <?xml version="1.0"?><GetQueueAttributesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><GetQueueAttributesResult><Attribute><Name>QueueArn</Name><Value>arn:aws:sqs:***:redacted:redacted</Value></Attribute><Attribute><Name>ApproximateNumberOfMessages</Name><Value>0</Value></Attribute><Attribute><Name>ApproximateNumberOfMessagesNotVisible</Name><Value>0</Value></Attribute><Attribute><Name>ApproximateNumberOfMessagesDelayed</Name><Value>0</Value></Attribute><Attribute><Name>CreatedTimestamp</Name><Value>1631028296</Value></Attribute><Attribute><Name>LastModifiedTimestamp</Name><Value>1641833218</Value></Attribute><Attribute><Name>VisibilityTimeout</Name><Value>30</Value></Attribute><Attribute><Name>MaximumMessageSize</Name><Value>262144</Value></Attribute><Attribute><Name>MessageRetentionPeriod</Name><Value>345600</Value></Attribute><Attribute><Name>DelaySeconds</Name><Value>0</Value></Attribute><Attribute><Name>Policy</Name><Value>{&quot;Version&quot;:&quot;2012-10-17&quot;,&quot;Id&quot;:&quot;sqspolicy&quot;,&quot;Statement&quot;:[{&quot;Sid&quot;:&quot;AllowSNSSubscriptions&quot;,&quot;Effect&quot;:&quot;Allow&quot;,&quot;Principal&quot;:&quot;*&quot;,&quot;Action&quot;:&quot;sqs:SendMessage&quot;,&quot;Resource&quot;:&quot;arn:aws:sqs:***:redacted:redacted&quot;,&quot;Condition&quot;:{&quot;ArnEquals&quot;:{&quot;aws:SourceArn&quot;:[]}}}]}</Value></Attribute><Attribute><Name>RedrivePolicy</Name><Value>{&quot;deadLetterTargetArn&quot;:&quot;arn:aws:sqs:***:redacted:redacted&quot;,&quot;maxReceiveCount&quot;:4}</Value></Attribute><Attribute><Name>ReceiveMessageWaitTimeSeconds</Name><Value>0</Value></Attribute><Attribute><Name>SqsManagedSseEnabled</Name><Value>true</Value></Attribute></GetQueueAttributesResult><ResponseMetadata><RequestId>324b8dff-35b1-5c1c-bbaf-d7a95ecc8563</RequestId></ResponseMetadata></GetQueueAttributesResponse>: timestamp=2022-01-10T17:13:33.057Z

 2022-01-10T17:13:23.023Z [INFO]  provider.terraform-provider-aws_v3.71.0_x5: 2022/01/10 17:13:23 [TRACE] Waiting 10s before next try: timestamp=2022-01-10T17:13:23.022Z
redacted.aws_sqs_queue.queue: Still modifying... [id=https://sqs.***.amazonaws.com/redacted/redacted, 1m50s elapsed]

Formatted xml

<?xml version="1.0" encoding="UTF-8"?>
<GetQueueAttributesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/">
   <GetQueueAttributesResult>
      <Attribute>
         <Name>QueueArn</Name>
         <Value>arn:aws:sqs:***:redacted:redacted</Value>
      </Attribute>
      <Attribute>
         <Name>ApproximateNumberOfMessages</Name>
         <Value>0</Value>
      </Attribute>
      <Attribute>
         <Name>ApproximateNumberOfMessagesNotVisible</Name>
         <Value>0</Value>
      </Attribute>
      <Attribute>
         <Name>ApproximateNumberOfMessagesDelayed</Name>
         <Value>0</Value>
      </Attribute>
      <Attribute>
         <Name>CreatedTimestamp</Name>
         <Value>1631028296</Value>
      </Attribute>
      <Attribute>
         <Name>LastModifiedTimestamp</Name>
         <Value>1641833218</Value>
      </Attribute>
      <Attribute>
         <Name>VisibilityTimeout</Name>
         <Value>30</Value>
      </Attribute>
      <Attribute>
         <Name>MaximumMessageSize</Name>
         <Value>262144</Value>
      </Attribute>
      <Attribute>
         <Name>MessageRetentionPeriod</Name>
         <Value>345600</Value>
      </Attribute>
      <Attribute>
         <Name>DelaySeconds</Name>
         <Value>0</Value>
      </Attribute>
      <Attribute>
         <Name>Policy</Name>
         <Value>{"Version":"2012-10-17","Id":"sqspolicy","Statement":[{"Sid":"AllowSNSSubscriptions","Effect":"Allow","Principal":"*","Action":"sqs:SendMessage","Resource":"arn:aws:sqs:***:redacted:redacted","Condition":{"ArnEquals":{"aws:SourceArn":[]}}}]}</Value>
      </Attribute>
      <Attribute>
         <Name>RedrivePolicy</Name>
         <Value>{"deadLetterTargetArn":"arn:aws:sqs:***:redacted:redacted","maxReceiveCount":4}</Value>
      </Attribute>
      <Attribute>
         <Name>ReceiveMessageWaitTimeSeconds</Name>
         <Value>0</Value>
      </Attribute>
      <Attribute>
         <Name>SqsManagedSseEnabled</Name>
         <Value>true</Value>
      </Attribute>
   </GetQueueAttributesResult>
   <ResponseMetadata>
      <RequestId>324b8dff-35b1-5c1c-bbaf-d7a95ecc8563</RequestId>
   </ResponseMetadata>
</GetQueueAttributesResponse>

 {
      "module": "redacted",
      "mode": "managed",
      "type": "aws_sqs_queue",
      "name": "queue",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "arn": "arn:aws:sqs:***:redacted:redacted",
            "content_based_deduplication": false,
            "deduplication_scope": "",
            "delay_seconds": 0,
            "fifo_queue": false,
            "fifo_throughput_limit": "",
            "id": "https://sqs.***.amazonaws.com/redacted/redacted",
            "kms_data_key_reuse_period_seconds": 300,
            "kms_master_key_id": "",
            "max_message_size": 262144,
            "message_retention_seconds": 345600,
            "name": "redacted",
            "name_prefix": "",
            "policy": "{\"Id\":\"sqspolicy\",\"Statement\":[{\"Action\":\"sqs:SendMessage\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":[]}},\"Effect\":\"Allow\",\"Principal\":\"*\",\"Resource\":\"arn:aws:sqs:***:redacted:redacted\",\"Sid\":\"AllowSNSSubscriptions\"}],\"Version\":\"2012-10-17\"}",
            "receive_wait_time_seconds": 0,
            "redrive_policy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:***:redacted:redacted\",\"maxReceiveCount\":4}",
            "sqs_managed_sse_enabled": true,
            "tags": {
             // redacted
            },
            "tags_all": {
             // redacted
            },
            "url": "https://sqs.***.amazonaws.com/redacted/redacted",
            "visibility_timeout_seconds": 30
          },
          "sensitive_attributes": [],
          "private": "redacted",
          "dependencies": [
            // redacted
          ]
        }

Expected Behavior

The plan should be applied.

Actual Behavior

The plan timeout with "notequal" state.

v3.71.0

 Error: error waiting for SQS Queue (https://sqs.us-east-1.amazonaws.com/redacted/redacted) attributes to update: timeout while waiting for state to become 'equal' (last state: 'notequal', timeout: 2m0s)

  with redacted.module.queue.aws_sqs_queue.queue,
  on /tmp/terraform-data-dir/modules/redacted/main.tf line 15, in resource "aws_sqs_queue" "queue":
  15: resource "aws_sqs_queue" "queue" {

v3.69.0

 Error: error waiting for SQS Queue (https://sqs.***.amazonaws.com/redacted/redacted) attributes to update: SQS Queue policies are not equivalent

  with redacted.aws_sqs_queue.queue,
  on /tmp/terraform-data-dir/modules/redacted/main.tf line 15, in resource "aws_sqs_queue" "queue":
  15: resource "aws_sqs_queue" "queue" {

Steps to Reproduce

terraform plan
Observe the "whitespace" changes.
terraform apply
Observe the timeout.

Important Factoids

It worked before (november 2021 ish). I'll try to run with older versions and report the results.

References

Might be related with AssumeRolePolicy seems to be incorrectly normalized, resulting in re-planning for "whitespace changes" #21016

The text was updated successfully, but these errors were encountered:

mentos1386 · 2022-01-11T09:13:06Z

I have run some more tests today. And it seems that there was a regression with v3.69.0 release. Versions before that seem to work fine.

justinretzolk · 2022-01-11T20:46:34Z

Hey @mentos1386 👋 Thank you for taking the time to file this, and for the great information and digging you've already done. While I wasn't quite able to determine what may have caused the change in behavior between v3.68.0 and v3.69.0, I did find at least some information on an initial pass at looking at this.

Based on the error that you pasted over from v3.69.0, I believe that was fixed with #22194. In that pull request, it was noted that:

NOTE: These changes will make SQS configurations take a little longer to apply.

Based on this, I think this may ultimately be the cause of the timeouts in v3.71.0 (the changes we actually released in v3.70.0, so I suspect you'd see similar issues in that version). I'm going to mark this as a bug so that the team can take a look as soon as time allows.

github-actions · 2022-01-14T00:01:38Z

This functionality has been released in v3.72.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2022-05-20T02:31:20Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

mentos1386 changed the title ~~SQS Times out on policy whitespace changes~~ aws_sqs_queue times out on policy whitespace changes Jan 10, 2022

github-actions bot added needs-triage Waiting for first response or review from a maintainer. service/sqs Issues and PRs that pertain to the sqs service. labels Jan 10, 2022

justinretzolk added bug Addresses a defect in current functionality. regression Pertains to a degraded workflow resulting from an upstream patch or internal enhancement. and removed needs-triage Waiting for first response or review from a maintainer. labels Jan 11, 2022

This was referenced Jan 12, 2022

Treat Conditions with empty array values as empty string values hashicorp/awspolicyequivalence#8

Merged

Bump github.com/hashicorp/awspolicyequivalence from 1.4.0 to 1.5.0 #22547

Merged

ewbankkit closed this as completed in #22547 Jan 12, 2022

github-actions bot added this to the v3.72.0 milestone Jan 12, 2022

github-actions bot locked as resolved and limited conversation to collaborators May 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws_sqs_queue times out on policy whitespace changes #22505

aws_sqs_queue times out on policy whitespace changes #22505

mentos1386 commented Jan 10, 2022 •

edited

Loading

mentos1386 commented Jan 11, 2022

justinretzolk commented Jan 11, 2022

github-actions bot commented Jan 14, 2022

github-actions bot commented May 20, 2022

aws_sqs_queue times out on policy whitespace changes #22505

aws_sqs_queue times out on policy whitespace changes #22505

Comments

mentos1386 commented Jan 10, 2022 • edited Loading

Community Note

Terraform CLI and Terraform AWS Provider Version

Provider versions

Affected Resource(s)

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

References

mentos1386 commented Jan 11, 2022

justinretzolk commented Jan 11, 2022

github-actions bot commented Jan 14, 2022

github-actions bot commented May 20, 2022

mentos1386 commented Jan 10, 2022 •

edited

Loading