aws_kinesis_firehose_delivery_stream still doesn't save processors configuration properly

[dvishniakov]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform: 0.14.11 (doesn't matter much because this is reproduced in other versions)
AWS Provider: 3.41

Affected Resource(s)

• aws_kinesis_firehose_delivery_stream

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream
resource "aws_kinesis_firehose_delivery_stream" "default" {
  name        = local.firehose_name
  tags        = var.tags
  destination = "extended_s3"

  extended_s3_configuration {
    role_arn   = var.firehose_role_arn
    bucket_arn = var.log_target_s3_bucket_arn

    prefix              = "prefix"
    error_output_prefix = "prefix"

    buffer_size        = var.buffer_size
    buffer_interval    = var.buffer_interval
    compression_format = var.compression_format
    kms_key_arn        = var.kms_key_arn

    # https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html
    processing_configuration {
      enabled = true

      processors {
        type = "Lambda"

        parameters {
          parameter_name  = "LambdaArn"
          parameter_value = var.log_processing_lambda_arn
        }

        # Up to 3 MB by default
        parameters {
          parameter_name  = "BufferSizeInMBs"
          parameter_value = "3"
        }

        parameters {
          parameter_name  = "BufferIntervalInSeconds"
          parameter_value = "300"
        }
      }
    }
  }

}

Debug Output

  ~ extended_s3_configuration {
        # (8 unchanged attributes hidden)


      ~ processing_configuration {
            # (1 unchanged attribute hidden)

          ~ processors {
                # (1 unchanged attribute hidden)

              ~ parameters {
                  ~ parameter_name  = "BufferIntervalInSeconds" -> "BufferSizeInMBs"
                  ~ parameter_value = "300" -> "3"
                }
              + parameters {
                  + parameter_name  = "BufferIntervalInSeconds"
                  + parameter_value = "300"
                }
                # (1 unchanged block hidden)
            }
        }
        # (1 unchanged block hidden)
    }

Panic Output

Expected Behavior

No changes in infra

Actual Behavior

Terraform adds BufferSizeInMBs each time.

Steps to Reproduce

1. terraform apply

Important Factoids

References

#4392

• See the previous to last comment there - seems didn't work after merge.

[tlkamp]

I am also experiencing this issue. It is also present when using nested processors blocks (i.e. under a splunk_configuration).

I did some investigating and found that the setting is being applied to the resource properly but the parameter is not being saved in the state file, resulting in the perpetual diff.

[ewbankkit]

Related:

• Kinesis firehose delivery stream, default lambda transform params not stored in state #9827

[dkujawski]

We are hitting this bug now.

I am pretty sure there is a bug in here where the default values are being evaluated. The logic needs to be reviewed, it doesn't seem like this will ever read the actual values from AWS.

	// It is necessary to explicitly filter this out
	// to prevent diffs during routine use and retain the ability
	// to show diffs if any field has drifted
	defaultLambdaParams := map[string]string{
		"NumberOfRetries":         "3",
		"RoleArn":                 roleArn,
		"BufferSizeInMBs":         "3",
		"BufferIntervalInSeconds": "60",
	}

	processors := make([]interface{}, len(pc.Processors))
	for i, p := range pc.Processors {
		t := aws.StringValue(p.Type)
		parameters := make([]interface{}, 0)

		for _, params := range p.Parameters {
			name := aws.StringValue(params.ParameterName)
			value := aws.StringValue(params.ParameterValue)

			if t == firehose.ProcessorTypeLambda {
				// Ignore defaults
				if v, ok := defaultLambdaParams[name]; ok && v == value {
					continue
				}
			}

[Iguanod]

@dkujawski the values are read from AWS, if you follow the function calls the pc parameter in that function comes from the call to FindDeliveryStreamByName in resourceDeliveryStreamRead.

The problem is not in removing the values if they match the defaults, the logic is correct and I assume it is necessary to do so according to the comment. Besides, that behaviour is documented in https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream (see the note in the parameters block configuration).

The issue here is that the default value for the buffer size is assumed to be 3MB, while in reality it is 1MB. I don't know if the default value was indeed 3MB at some point in the past, but right now new delivery streams are created with 1MB by default (as documented in https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html).

[github-actions]

This functionality has been released in v5.32.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.