aws_kinesis_firehose_delivery_stream not updating on kinesis_source_configuration change

[srkiNZ84]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

terraform -v
Terraform v0.11.3
+ provider.aws v1.14.0

Affected Resource(s)

• aws_kinesis_firehose_delivery_stream

Terraform Configuration Files

resource "aws_kinesis_firehose_delivery_stream" "sync_firehose" {
  name        = "${var.environment}-sync_firehose"
  destination = "redshift"

  kinesis_source_configuration {
    kinesis_stream_arn = "arn:aws:kinesis:${var.region}:${var.accountid}:stream/${var.environment}_SourceOfChanges"

    role_arn = "${aws_iam_role.sync_firehose.arn}"
  }

  s3_configuration {
    role_arn   = "${aws_iam_role.sync_firehose.arn}"
    bucket_arn = "${aws_s3_bucket.sync_firehose_bucket.arn}"

    cloudwatch_logging_options {
      enabled         = true
      log_group_name  = "/aws/kinesisfirehose/${var.environment}-sync_firehose/s3delivery"
      log_stream_name = "S3Delivery"
    }
  }

  redshift_configuration {
    role_arn        = "${aws_iam_role.sync_firehose.arn}"
    cluster_jdbcurl = "jdbc:redshift://${var.redshift_endpoint}/${var.redshift_database_name}"
    username        = "sync_firehose_etl"
    password        = "REDACTED" 
    data_table_name = "foo.bar"
    copy_options    = "json 's3://${aws_s3_bucket.sync_firehose_bucket.id}/sync_firehose/jsonpaths.json' region '${var.region}' timeformat 'auto'"

    data_table_columns = "id, createdAt, updatedAt, foo, bar"
    retry_duration     = 60

    cloudwatch_logging_options {
      enabled         = true
      log_group_name  = "/aws/kinesisfirehose/${var.environment}-sync-firehose/redshiftdelivery"
      log_stream_name = "RedshiftDelivery"
    }
  }
}

Debug Output

Please let me know if this will be required

Panic Output

N/A

Expected Behavior

When I change the value of the kinesis_source_configuration.kinesis_stream_arn, Terraform should detect that there is a change to be applied. When running "terraform apply" it should also apply the changes, instead of returning a message that there are no changes to be applied.

NOTE: From the console, trying to "edit" the firehose object doesn't allow changing the Kinesis Source Stream value, so this may have to be a delete/re-create option.

Actual Behavior

After changing the Kinesis Source Stream ARN on the "Kinesis Firehose" object. Terraform responded that the changes had been applied successfully and then subsequently that there were no further changes to be applied. However, examining the Kinesis Firehose object in the AWS console and using the AWS CLI showed that the "Source Stream ARN value was still set to the previous value.

Steps to Reproduce

1. Create two Kinesis Data Steam objects in AWS (e.g. "A" and "B")

2. Run terraform apply to create a Kinesis Firehose object pointing at "Stream A"

3. Change the HCL to set the Firehose to point to "Stream B"

4. Run terraform plan and note the plan to change the Firehose object

5. Run terraform apply and note the "successful change" message

6. Check in AWS console/CLI and note that the Firehose "source" is still pointing to "Stream A"

7. Run terraform plan

8. Note that the output shows nothing to be applied.

Important Factoids

As mentioned, from the AWS console, trying to "edit" the firehose object doesn't allow changing the Kinesis Source Stream value, so this may have to be a delete/re-create option.

References

• Possibly related to aws_kinesis_firehose_delivery_stream ignores changes to role_arn #380

[bflad]

Since the Firehose API does not currently provide an update mechanism, both the Kinesis role and stream arguments have been marked as ForceNew: true in #5332 to recreate the resource instead of presenting updates that will never occur. This will release with version 1.29.0 of the AWS provider shortly.

[bflad]

This has been released in version 1.29.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!