[Bug]: aws_msk_replicator not tracking correct state of detect_and_copy_new_topics

[cobbr2]

Terraform Core Version

1.5.7

AWS Provider Version

5.38.0

Affected Resource(s)

• aws_msk_replicator

Expected Behavior

Nothing. The state of the resource's detect_and_copy_new_topics attribute is already true

Actual Behavior

The plan shows that the provider wants to change that field from false to true. The state file also shows false, though the resource itself @ AWS shows true. Attempting to change it from its current "true" to Terraform's "true" generates AWS error message:

│ Error: updating MSK Replicator (arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:replicator/common-green-us-west-2-us-east-1/7e7885e6-0453-4623-9990-84759ba89d17-4): operation error Kafka: UpdateReplicationInfo, https response error StatusCode: 400, RequestID: 9f6ae9f2-d565-41f8-a4b1-1ef797d657a2, BadRequestException: The specified replication info is identical to the current value. Specify a different value, then try again.

State at AWS:

$ aws kafka describe-replicator --replicator-arn "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:replicator/common-green-us-west-2-us-east-1/7e7885e6-0453-4623-9990-84759ba89d17-4" --no-paginate
{
    "CreationTime": "2024-02-06T01:57:18.110000+00:00",
    "CurrentVersion": "CKEVKO7DR4RA",
    "IsReplicatorReference": false,
    "KafkaClusters": [
        {
            "AmazonMskCluster": {
                "MskClusterArn": "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:cluster/common-green-us-east-1/361a2e04-ff39-48d4-9ee0-f810cd197370-s2"
            },
            "KafkaClusterAlias": "common-green-us-east-1-f810cd197370",
            "VpcConfig": {
                "SecurityGroupIds": [
                    "sg-0553046cce9f25d4e"
                ],
                "SubnetIds": [
                    "subnet-0c3<REDACTED>",
                    "subnet-f83<REDACTED>",
                    "subnet-b74<REDACTED>"
                ]
            }
        },
        {
            "AmazonMskCluster": {
                "MskClusterArn": "arn:aws:kafka:us-west-2:ACCOUNT_REDACTED:cluster/common-green-us-west-2/dda06cc0-beac-4862-8cba-e18cd64c6075-s2"
            },
            "KafkaClusterAlias": "common-green-us-west-2-e18cd64c6075",
            "VpcConfig": {
                "SecurityGroupIds": [
                    "sg-0e96ecfe1cb5bf645"
                ],
                "SubnetIds": [
                    "subnet-037<REDACTED>",
                    "subnet-0a2<REDACTED>",
                    "subnet-00f<REDACTED>"
                ]
            }
        }
    ],
    "ReplicationInfoList": [
        {
            "ConsumerGroupReplication": {
                "ConsumerGroupsToExclude": [],
                "ConsumerGroupsToReplicate": [
                    ".*"
                ],
                "DetectAndCopyNewConsumerGroups": true,
                "SynchroniseConsumerGroupOffsets": true
            },
            "SourceKafkaClusterAlias": "common-green-us-west-2-e18cd64c6075",
            "TargetCompressionType": "NONE",
            "TargetKafkaClusterAlias": "common-green-us-east-1-f810cd197370",
            "TopicReplication": {
                "CopyAccessControlListsForTopics": false,
                "CopyTopicConfigurations": true,
                "DetectAndCopyNewTopics": true,
                "TopicsToExclude": [],
                "TopicsToReplicate": [
                    ".*"
                ]
            }
        }
    ],
    "ReplicatorArn": "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:replicator/common-green-us-west-2-us-east-1/7e7885e6-0453-4623-9990-84759ba89d17-4",
    "ReplicatorName": "common-green-us-west-2-us-east-1",
    "ReplicatorState": "RUNNING",
    "ServiceExecutionRoleArn": "arn:aws:iam::ACCOUNT_REDACTED:role/kafka-replicator-common-green",
    "Tags": {
        "Service": "shared-kafka",
        "Team": "developer-platform",
        "Module": "replicated_kafka",
        "ModuleVersion": "0.2",
        "SecurityLevel": "red"
    }
}

State in terraform show:

$ tf state show "module.common_cluster_green[0].aws_msk_replicator.replicator_region02_region01"
# module.common_cluster_green[0].aws_msk_replicator.replicator_region02_region01:
resource "aws_msk_replicator" "replicator_region02_region01" {
    arn                        = "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:replicator/common-green-us-west-2-us-east-1/7e7885e6-0453-4623-9990-84759ba89d17-4"
    current_version            = "CKEVKO7DR4RA"
    id                         = "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:replicator/common-green-us-west-2-us-east-1/7e7885e6-0453-4623-9990-84759ba89d17-4"
    replicator_name            = "common-green-us-west-2-us-east-1"
    service_execution_role_arn = "arn:aws:iam::ACCOUNT_REDACTED:role/kafka-replicator-common-green"
    tags                       = {
        "Module"        = "replicated_kafka"
        "ModuleVersion" = "0.2"
        "SecurityLevel" = "red"
    }
    tags_all                   = {
        "Module"        = "replicated_kafka"
        "ModuleVersion" = "0.2"
        "SecurityLevel" = "red"
        "Service"       = "shared-kafka"
        "Team"          = "developer-platform"
    }

    kafka_cluster {
        amazon_msk_cluster {
            msk_cluster_arn = "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:cluster/common-green-us-east-1/361a2e04-ff39-48d4-9ee0-f810cd197370-s2"
        }
        vpc_config {
            security_groups_ids = [
                "sg-0553046cce9f25d4e",
            ]
            subnet_ids          = (sensitive value)
        }
    }
    kafka_cluster {
        amazon_msk_cluster {
            msk_cluster_arn = "arn:aws:kafka:us-west-2:ACCOUNT_REDACTED:cluster/common-green-us-west-2/dda06cc0-beac-4862-8cba-e18cd64c6075-s2"
        }
        vpc_config {
            security_groups_ids = [
                "sg-0e96ecfe1cb5bf645",
            ]
            subnet_ids          = (sensitive value)
        }
    }

    replication_info_list {
        source_kafka_cluster_alias = "common-green-us-west-2-e18cd64c6075"
        source_kafka_cluster_arn   = "arn:aws:kafka:us-west-2:ACCOUNT_REDACTED:cluster/common-green-us-west-2/dda06cc0-beac-4862-8cba-e18cd64c6075-s2"
        target_compression_type    = "NONE"
        target_kafka_cluster_alias = "common-green-us-east-1-f810cd197370"
        target_kafka_cluster_arn   = "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:cluster/common-green-us-east-1/361a2e04-ff39-48d4-9ee0-f810cd197370-s2"

        consumer_group_replication {
            consumer_groups_to_exclude          = []
            consumer_groups_to_replicate        = [
                ".*",
            ]
            detect_and_copy_new_consumer_groups = true
            synchronise_consumer_group_offsets  = true
        }

        topic_replication {
            copy_access_control_lists_for_topics = false
            copy_topic_configurations            = true
            detect_and_copy_new_topics           = false
            topics_to_exclude                    = []
            topics_to_replicate                  = [
                ".*",
            ]
        }
    }
}

State after state rm and import is the same:

$ tf state rm "module.common_cluster_green[0].aws_msk_replicator.replicator_region02_region01"
Removed module.common_cluster_green[0].aws_msk_replicator.replicator_region02_region01
Successfully removed 1 resource instance(s).
Releasing state lock. This may take a few moments...

$ tf import "module.common_cluster_green[0].aws_msk_replicator.replicator_region02_region01" "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:replicator/common-green-us-west-2-us-east-1/7e7885e6-0453-4623-9990-84759ba89d17-4"
...
Import successful!
...
Releasing state lock. This may take a few moments...

$ tf state show "module.common_cluster_green[0].aws_msk_replicator.replicator_region02_region01"
# module.common_cluster_green[0].aws_msk_replicator.replicator_region02_region01:
resource "aws_msk_replicator" "replicator_region02_region01" {
    arn                        = "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:replicator/common-green-us-west-2-us-east-1/7e7885e6-0453-4623-9990-84759ba89d17-4"
    current_version            = "CKEVKO7DR4RA"
    id                         = "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:replicator/common-green-us-west-2-us-east-1/7e7885e6-0453-4623-9990-84759ba89d17-4"
    replicator_name            = "common-green-us-west-2-us-east-1"
    service_execution_role_arn = "arn:aws:iam::ACCOUNT_REDACTED:role/kafka-replicator-common-green"
    tags                       = {
        "Module"        = "replicated_kafka"
        "ModuleVersion" = "0.2"
        "SecurityLevel" = "red"
    }
    tags_all                   = {
        "Module"        = "replicated_kafka"
        "ModuleVersion" = "0.2"
        "SecurityLevel" = "red"
        "Service"       = "shared-kafka"
        "Team"          = "developer-platform"
    }

    kafka_cluster {
        amazon_msk_cluster {
            msk_cluster_arn = "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:cluster/common-green-us-east-1/361a2e04-ff39-48d4-9ee0-f810cd197370-s2"
        }
        vpc_config {
            security_groups_ids = [
                "sg-0553046cce9f25d4e",
            ]
            subnet_ids          = [
                "subnet-0c3<REDACTED>",
                "subnet-b74<REDACTED>",
                "subnet-f83<REDACTED>",
            ]
        }
    }
    kafka_cluster {
        amazon_msk_cluster {
            msk_cluster_arn = "arn:aws:kafka:us-west-2:ACCOUNT_REDACTED:cluster/common-green-us-west-2/dda06cc0-beac-4862-8cba-e18cd64c6075-s2"
        }
        vpc_config {
            security_groups_ids = [
                "sg-0e96ecfe1cb5bf645",
            ]
            subnet_ids          = [
                "subnet-00f<REDACTED>",
                "subnet-037<REDACTED>",
                "subnet-0a2<REDACTED>",
            ]
        }
    }

    replication_info_list {
        source_kafka_cluster_alias = "common-green-us-west-2-e18cd64c6075"
        source_kafka_cluster_arn   = "arn:aws:kafka:us-west-2:ACCOUNT_REDACTED:cluster/common-green-us-west-2/dda06cc0-beac-4862-8cba-e18cd64c6075-s2"
        target_compression_type    = "NONE"
        target_kafka_cluster_alias = "common-green-us-east-1-f810cd197370"
        target_kafka_cluster_arn   = "arn:aws:kafka:us-east-1:ACCOUNT_REDACTED:cluster/common-green-us-east-1/361a2e04-ff39-48d4-9ee0-f810cd197370-s2"

        consumer_group_replication {
            consumer_groups_to_exclude          = []
            consumer_groups_to_replicate        = [
                ".*",
            ]
            detect_and_copy_new_consumer_groups = true
            synchronise_consumer_group_offsets  = true
        }

        topic_replication {
            copy_access_control_lists_for_topics = false
            copy_topic_configurations            = true
            detect_and_copy_new_topics           = false
            topics_to_exclude                    = []
            topics_to_replicate                  = [
                ".*",
            ]
        }
    }
}

Relevant Error/Panic Output Snippet

│ Error: updating MSK Replicator (arn:aws:kafka:us-east-1:179874059503:replicator/common-green-us-west-2-us-east-1/7e7885e6-0453-4623-9990-84759ba89d17-4): operation error Kafka: UpdateReplicationInfo, https response error StatusCode: 400, RequestID: 9f6ae9f2-d565-41f8-a4b1-1ef797d657a2, BadRequestException: The specified replication info is identical to the current value. Specify a different value, then try again.

Terraform Configuration Files

I'll try to do this later in the day; I'll want to extract a single replicator and single cluster pair to replicate with. This particular root module builds 5 MSK clusters & 4 replicators, so it's a bit big for a test case.

Steps to Reproduce

terraform apply

Debug Output

See https://gist.github.com/cobbr2/da1e354a303b6c43705108e8c5c382f3

Panic Output

No response

Important Factoids

These replicators have been destroyed & re-created dozens of times; only once we started setting this parameter to true did we run into this problem. We never used the default settings (we can't, we use MSK Serverless so the ..._copy_acl... stuff has to be set to false).

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.