DynamoDB Non-Key Attributes Ordering

[bacoboy]

I believe this is a bug of some kind due to the refactoring in #3136 or maybe it was part of the create logic and has been there all along.

When specifying multiple non-key attributes the order specified in terraform isn't how the table is created and an immediate terraform plan shows changes are needed.

Start with this table resource:

resource "aws_dynamodb_table" "AAA" {
  name           = "AAA"
  read_capacity  = "1"
  write_capacity = "1"
  hash_key       = "AAA"

  attribute {
    name = "AAA"
    type = "S"
  }

  attribute {
    name = "BBB"
    type = "S"
  }

  global_secondary_index {
    name               = "BBB"
    hash_key           = "BBB"
    read_capacity      = "1"
    write_capacity     = "1"
    projection_type    = "INCLUDE"
    non_key_attributes = ["foo", "bar", "baz"]
  }

}

As you can see, foo, bar and baz are the non key attributes. Now run terraform apply.

$ terraform apply
aws_dynamodb_table.AAA: Creating...
  arn:                                                   "" => "<computed>"
  attribute.#:                                           "" => "2"
  attribute.3741918594.name:                             "" => "AAA"
  attribute.3741918594.type:                             "" => "S"
  attribute.3839322102.name:                             "" => "BBB"
  attribute.3839322102.type:                             "" => "S"
  global_secondary_index.#:                              "" => "1"
  global_secondary_index.109633842.hash_key:             "" => "BBB"
  global_secondary_index.109633842.name:                 "" => "BBB"
  global_secondary_index.109633842.non_key_attributes.#: "" => "3"
  global_secondary_index.109633842.non_key_attributes.0: "" => "foo"
  global_secondary_index.109633842.non_key_attributes.1: "" => "bar"
  global_secondary_index.109633842.non_key_attributes.2: "" => "baz"
  global_secondary_index.109633842.projection_type:      "" => "INCLUDE"
  global_secondary_index.109633842.range_key:            "" => ""
  global_secondary_index.109633842.read_capacity:        "" => "1"
  global_secondary_index.109633842.write_capacity:       "" => "1"
  hash_key:                                              "" => "AAA"
  name:                                                  "" => "AAA"
  read_capacity:                                         "" => "1"
  server_side_encryption.#:                              "" => "<computed>"
  stream_arn:                                            "" => "<computed>"
  stream_label:                                          "" => "<computed>"
  stream_view_type:                                      "" => "<computed>"
  write_capacity:                                        "" => "1"
aws_dynamodb_table.AAA: Still creating... (10s elapsed)
[SNIP]
aws_dynamodb_table.AAA: Still creating... (4m20s elapsed)
aws_dynamodb_table.AAA: Creation complete after 4m27s (ID: AAA)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

After the table is created run terraform plan. I expected to see no changes, but got this instead:

Terraform will perform the following actions:

  ~ aws_dynamodb_table.AAA
      global_secondary_index.109633842.hash_key:              "" => "BBB"
      global_secondary_index.109633842.name:                  "" => "BBB"
      global_secondary_index.109633842.non_key_attributes.#:  "0" => "3"
      global_secondary_index.109633842.non_key_attributes.0:  "" => "foo"
      global_secondary_index.109633842.non_key_attributes.1:  "" => "bar"
      global_secondary_index.109633842.non_key_attributes.2:  "" => "baz"
      global_secondary_index.109633842.projection_type:       "" => "INCLUDE"
      global_secondary_index.109633842.range_key:             "" => ""
      global_secondary_index.109633842.read_capacity:         "" => "1"
      global_secondary_index.109633842.write_capacity:        "" => "1"
      global_secondary_index.4188367599.hash_key:             "BBB" => ""
      global_secondary_index.4188367599.name:                 "BBB" => ""
      global_secondary_index.4188367599.non_key_attributes.#: "3" => "0"
      global_secondary_index.4188367599.non_key_attributes.0: "bar" => ""
      global_secondary_index.4188367599.non_key_attributes.1: "baz" => ""
      global_secondary_index.4188367599.non_key_attributes.2: "foo" => ""
      global_secondary_index.4188367599.projection_type:      "INCLUDE" => ""
      global_secondary_index.4188367599.range_key:            "" => ""
      global_secondary_index.4188367599.read_capacity:        "1" => "0"
      global_secondary_index.4188367599.write_capacity:       "1" => "0"

Using the aws cli, I confirm the order AWS has them:

$ aws dynamodb describe-table --table-name AAA
{
    "Table": {
        "AttributeDefinitions": [
            {
                "AttributeName": "AAA",
                "AttributeType": "S"
            },
            {
                "AttributeName": "BBB",
                "AttributeType": "S"
            }
        ],
        "TableName": "AAA",
        "KeySchema": [
            {
                "AttributeName": "AAA",
                "KeyType": "HASH"
            }
        ],
        "TableStatus": "ACTIVE",
        "CreationDateTime": 1521204954.031,
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 1,
            "WriteCapacityUnits": 1
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:us-east-1:246062453161:table/AAA",
        "TableId": "3903d836-f14b-482f-a426-288c6b3b4766",
        "GlobalSecondaryIndexes": [
            {
                "IndexName": "BBB",
                "KeySchema": [
                    {
                        "AttributeName": "BBB",
                        "KeyType": "HASH"
                    }
                ],
                "Projection": {
                    "ProjectionType": "INCLUDE",
                    "NonKeyAttributes": [
                        "bar",
                        "baz",
                        "foo"
                    ]
                },
                "IndexStatus": "ACTIVE",
                "ProvisionedThroughput": {
                    "NumberOfDecreasesToday": 0,
                    "ReadCapacityUnits": 1,
                    "WriteCapacityUnits": 1
                },
                "IndexSizeBytes": 0,
                "ItemCount": 0,
                "IndexArn": "arn:aws:dynamodb:us-east-1:246062453161:table/AAA/index/BBB"
            }
        ]
    }
}

The order has changed from foo, bar, baz (see the order in the create-table operation) TO bar, baz, foo (the actual state in AWS and the terraform state)

Running terraform apply again will drop and recreated the BBB GSI, but then you are still in the same place:

$ terraform apply
aws_dynamodb_table.AAA: Refreshing state... (ID: AAA)
aws_dynamodb_table.AAA: Modifying... (ID: AAA)
  global_secondary_index.109633842.hash_key:              "" => "BBB"
  global_secondary_index.109633842.name:                  "" => "BBB"
  global_secondary_index.109633842.non_key_attributes.#:  "0" => "3"
  global_secondary_index.109633842.non_key_attributes.0:  "" => "foo"
  global_secondary_index.109633842.non_key_attributes.1:  "" => "bar"
  global_secondary_index.109633842.non_key_attributes.2:  "" => "baz"
  global_secondary_index.109633842.projection_type:       "" => "INCLUDE"
  global_secondary_index.109633842.range_key:             "" => ""
  global_secondary_index.109633842.read_capacity:         "" => "1"
  global_secondary_index.109633842.write_capacity:        "" => "1"
  global_secondary_index.4188367599.hash_key:             "BBB" => ""
  global_secondary_index.4188367599.name:                 "BBB" => ""
  global_secondary_index.4188367599.non_key_attributes.#: "3" => "0"
  global_secondary_index.4188367599.non_key_attributes.0: "bar" => ""
  global_secondary_index.4188367599.non_key_attributes.1: "baz" => ""
  global_secondary_index.4188367599.non_key_attributes.2: "foo" => ""
  global_secondary_index.4188367599.projection_type:      "INCLUDE" => ""
  global_secondary_index.4188367599.range_key:            "" => ""
  global_secondary_index.4188367599.read_capacity:        "1" => "0"
  global_secondary_index.4188367599.write_capacity:       "1" => "0"
aws_dynamodb_table.AAA: Still modifying... (ID: AAA, 10s elapsed)
[SNIP]
aws_dynamodb_table.AAA: Still modifying... (ID: AAA, 2m20s elapsed)
aws_dynamodb_table.AAA: Modifications complete after 2m30s (ID: AAA)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_dynamodb_table.AAA: Refreshing state... (ID: AAA)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aws_dynamodb_table.AAA
      global_secondary_index.109633842.hash_key:              "" => "BBB"
      global_secondary_index.109633842.name:                  "" => "BBB"
      global_secondary_index.109633842.non_key_attributes.#:  "0" => "3"
      global_secondary_index.109633842.non_key_attributes.0:  "" => "foo"
      global_secondary_index.109633842.non_key_attributes.1:  "" => "bar"
      global_secondary_index.109633842.non_key_attributes.2:  "" => "baz"
      global_secondary_index.109633842.projection_type:       "" => "INCLUDE"
      global_secondary_index.109633842.range_key:             "" => ""
      global_secondary_index.109633842.read_capacity:         "" => "1"
      global_secondary_index.109633842.write_capacity:        "" => "1"
      global_secondary_index.4188367599.hash_key:             "BBB" => ""
      global_secondary_index.4188367599.name:                 "BBB" => ""
      global_secondary_index.4188367599.non_key_attributes.#: "3" => "0"
      global_secondary_index.4188367599.non_key_attributes.0: "bar" => ""
      global_secondary_index.4188367599.non_key_attributes.1: "baz" => ""
      global_secondary_index.4188367599.non_key_attributes.2: "foo" => ""
      global_secondary_index.4188367599.projection_type:      "INCLUDE" => ""
      global_secondary_index.4188367599.range_key:            "" => ""
      global_secondary_index.4188367599.read_capacity:        "1" => "0"
      global_secondary_index.4188367599.write_capacity:       "1" => "0"


Plan: 0 to add, 1 to change, 0 to destroy.

The only way to fix this is to modify the terraform to match whatever ordering was picked the first time:

-    non_key_attributes = ["foo", "bar", "baz"]
+    non_key_attributes = ["bar", "baz", "foo" ]

Now the terraform plan shows no changes:

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_dynamodb_table.AAA: Refreshing state... (ID: AAA)

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Af first I thought the list might need to alphabetical, but my real table needed this modification to align:

-    non_key_attributes = ["CUUID", "AccountId", "CreateTimestamp"] # specified at create time
+    non_key_attributes = ["CreateTimestamp", "CUUID", "AccountId" ] # order needed to sync TF state

[nalindak]

Hey @bacoboy It worked! You have saved us! Thanks man, really appreciate it!

[davehowell]

Is this infernal reordering in the terraform state something that is just naturally fixed in terraform 0.12+ ?

[ennioj]

ordering the non-key_attribute works fine only if they are less than eleven, otherwise you will have always:
global_secondary_index.1718974420.name: "" => "extUserUUID-createdAt-index" global_secondary_index.1718974420.non_key_attributes.#: "0" => "11" global_secondary_index.1718974420.non_key_attributes.0: "" => "requestedFrom" global_secondary_index.1718974420.non_key_attributes.1: "" => "extStoreUUID" global_secondary_index.1718974420.non_key_attributes.10: "" => "uuid" global_secondary_index.1718974420.non_key_attributes.2: "" => "isAsap" global_secondary_index.1718974420.non_key_attributes.3: "" => "serialNumber" global_secondary_index.1718974420.non_key_attributes.4: "" => "updatedAt" global_secondary_index.1718974420.non_key_attributes.5: "" => "status" global_secondary_index.1718974420.non_key_attributes.6: "" => "bundles" global_secondary_index.1718974420.non_key_attributes.7: "" => "payment" global_secondary_index.1718974420.non_key_attributes.8: "" => "createdAt" global_secondary_index.1718974420.non_key_attributes.9: "" => "timezone" global_secondary_index.1718974420.projection_type: "" => "INCLUDE"
basically the one with index10 will be always counted as the third element

[flpStrri]

Is there any update on this?

[yannh]

I believe this is a duplicate of #3828 , and seems to have to do with ordering of non Key Attributes.

[bflad]

Hi folks 👋 The global_secondary_index configuration block non_key_attributes ordering issues were likely resolved with the resource update released in Terraform AWS Provider version 3.5.0. If you are still having trouble after this release, please open a new GitHub bug report and we will take a look. Thanks.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!