processing_configuration not available for AWS Elasticsearch Firehose Delivery Stream

[hashibot]

This issue was originally opened by @jmlw as hashicorp/terraform#16963. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.11.1
* provider.aws: version = "~> 1.6"
* provider.template: version = "~> 1.0"

Terraform Configuration Files

resource "aws_kinesis_firehose_delivery_stream" "elasticsearch_ingestion_stream" {
  name        = "elasticsearch_ingestion_stream"
  destination = "elasticsearch"

  kinesis_source_configuration {
    kinesis_stream_arn = "kinesis-stream-arn"
    role_arn           = "firehose_iam_role"
  }

  s3_configuration {
    role_arn           = "firehose-iam-role"
    bucket_arn         = "arn:aws:s3:::bucket"
    prefix             = "logs/logs"
    buffer_size        = 10
    buffer_interval    = 400
    compression_format = "GZIP"
  }

  elasticsearch_configuration {
    domain_arn = "domain"
    role_arn   = "role"
    index_name = "index"
    type_name  = "doc_type"
    buffering_interval = 60
    index_rotation_period = "OneDay"
    retry_duration = 900
  }
}

resource "aws_kinesis_firehose_delivery_stream" "s3_ingestion_stream" {
  name        = "s3_ingestion_stream"
  destination = "extended_s3"

  kinesis_source_configuration {
    kinesis_stream_arn = "kinesis-stream-arn"
    role_arn           = "firehose_iam_role"
  }

  extended_s3_configuration {
    role_arn                 = "firehost_iam_role"
    bucket_arn               = "arn:aws:s3:::bucket"
    prefix                   = "logs/logs"
    processing_configuration = [
      {
        enabled = "true"
        processors = [
          {
            type = "Lambda"
            parameters = [
              {
                parameter_name  = "LambdaArn"
                parameter_value = "transformer_lambda"
              }
            ]
          }
        ]
      }
    ]
  }
}

Debug Output

Not applicable

Crash Output

Not Applicable

Expected Behavior

Should be able to define lambda transformer for Elasticsearch destination the same way a processor can be defined for extended s3 destinations

Actual Behavior

processing_configuration is not a valid key for elasticsearch_configuration and is not available under any other key

[knope]

My team is also hitting this issue now. The method of CWL -through firehose subscription-> AWSES requires at the very minimum a lambda processor in order to decompress the CWL data.

Edit: Word on the street is this may get worked in some time soon. 💯

[knope]

this looks relevant #3621 ❣️

and updated doc includes: https://www.terraform.io/docs/providers/aws/r/kinesis_firehose_delivery_stream.html

  elasticsearch_configuration {
    domain_arn = "${aws_elasticsearch_domain.test_cluster.arn}"
    role_arn   = "${aws_iam_role.firehose_role.arn}"
    index_name = "test"
    type_name  = "test"

    processing_configuration = [
      {
        enabled = "true"
        processors = [
          {
            type = "Lambda"
            parameters = [
              {
                parameter_name = "LambdaArn"
                parameter_value = "${aws_lambda_function.lambda_processor.arn}:$LATEST"
              }
            ]
          }
        ]
      }
    ]
  }

[bflad]

Thanks for the bump @knope! This was previously released in version 1.14.0 of the AWS provider and has been available in all releases since. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!