Unable to create MSK configuration with no kafka_versions

[mgusiew-guide]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

• Terraform v0.12.23
• provider.aws v2.56.0

Affected Resource(s)

• aws_msk_configuration

Terraform Configuration Files

#No kafka_versions property

resource aws_msk_configuration no_arg_config {
  name                     = "no_arg_config"
  server_properties = <<PROPERTIES
auto.create.topics.enable = true
delete.topic.enable = true
PROPERTIES
}

# kafka_versions null property

resource aws_msk_configuration null_kafka_versions_config {
  name              = "null_kafka_versions_config"
  kafka_versions    = null
  server_properties = <<PROPERTIES
auto.create.topics.enable = true
delete.topic.enable = true
PROPERTIES
}

Debug Output

#No kafka_versions property output

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
cd "msk-configs"; \
        terraform plan 
Error: Missing required argument
  on .terraform/modules/msk-configs/modules/msk-configs/main.tf line 23, in resource "aws_msk_configuration" "no_arg_config":
  23: resource aws_msk_configuration no_arg_config {
The argument "kafka_versions" is required, but no definition was found.

#kafka_versions null property output 

cd "msk-configs"; \
        terraform plan 
Error: "kafka_versions": required field is not set
  on .terraform/modules/msk-configs/modules/msk-configs/main.tf line 23, in resource "aws_msk_configuration" "null_kafka_versions_config":
  23: resource aws_msk_configuration null_kafka_versions_config {

Panic Output

N/A

Expected Behaviour

I expected to create MSK configuration without specifying any Kafka Versions (see actual behaviour) but this property is currently mandatory on Terraform level even though it is optional on SDK level.

FTR it is fine that empty list and/or null fail but I expected that at least unset kafka_versions would get through

Actual Behavior

AWS UI and CLI allow to create MSK configuration without specifying Kafka version

• AWS CLI see https://docs.aws.amazon.com/cli/latest/reference/kafka/create-configuration.html)
• UI console in AWS MSK service allows to specify None for Kafka versions

In CLI --kafka-versions is optional parameter however Terraform resource requires this parameter to be set (see https://www.terraform.io/docs/providers/aws/r/msk_configuration.html#kafka_versions)

I tried following settings on terraform level:

1. Leave kafka_versions unset in TF resource - this does not work, see first output above for details
2. Set kafka_versions to null - this does not work either, see second output
3. Set kafka_versions to empty list ([]) - unfortunately this does not work either. I checked the cloudtrails and it shows following trace:

"requestParameters": {
        "name": "ConfigEmptyVersion",
        "kafkaVersions": [],
        "serverProperties": "YXV0by5jcmVhdGUudG9waWNzLmVuYWJsZSA9IHRydWUKZGVsZXRlLnRvcGljLmVuYWJsZSA9IHRydWUK"
    },
    "responseElements": {
        "message": "The provided value is not valid.",
        "invalidParameter": "kafkaVersions"
    }

I also created configuration without versions using UI just to check how should the request look like and got following results:

"requestParameters": {
        "name": "NoneConfigTest",
        "description": "Some desc",
        "serverProperties": "YXV0by5jcmVhdGUudG9waWNzLmVuYWJsZT1mYWxzZQ=="
    },
    "responseElements": {
        "name": "NoneConfigTest",
        "creationTime": "2020-07-23T14:49:11.474Z",
        "arn": "arn:aws:kafka:us-west-2:942795021942:configuration/NoneConfigTest/d97b9c42-9500-4f45-906d-9e86fe8bc2d7-4",
        "latestRevision": {
            "creationTime": "2020-07-23T14:49:11.474Z",
            "revision": 1,
            "description": "Some desc"
        }
    },

So it looks like the request works as long as kafka_versions is not specified, however given that null, empty list and property omissions don't work I am not able to achieve that from Terraform level.

This is can be useful when creating cluster with some config and then upgrading it to higher version when it becomes available (config is not version specific so chances are high that it will apply to higher version as well).

I think kafka_versions property should be optional on Terraform level as it is on AWS SDK level.

Steps to Reproduce

Try to create aws_msk_configuration with either:

1. Unset kafka_versions
2. kafka_versions set to null
3. kafka_versions set to []

Given that kafka_versions is optional on SDK level I would expect that 1) should be supported. I am fine with 2) and 3) not working

Important Factoids

It looks like AWS is actively working on providing more features WRT MSK configurations so it may be that kafka_versions property was mandatory at some point

References

[mantkiewicz-gwre]

I did some digging around. In aws-sdk-go, the KafkaVersions used to be a required field up until May. Compare the schema changes:

• previous https://github.com/aws/aws-sdk-go/blame/cf7f7d1533e03dad0bfce6e6da8395a65bd2f3d4/service/kafka/api.go#L3483
• current https://github.com/aws/aws-sdk-go/blame/master/service/kafka/api.go#L3772

I think terraform-provider-aws should make that field optional too.

[mgusiew-guide]

Good point @mantkiewicz . I think they had to make it optional when introducing Kafka upgrade feature. If you upgrade Kafka from 2.3.1 to 2.4.1 and have config for 2.3.1 then things may become inconsistent (cluster could end up on 2.4.1 but config would be still for 2.3.1). Therefore possibility to have config without Kafka version may be useful. So this one is partly related to #13547

[ghost]

This has been released in version 3.43.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.