DMS Endpoint Re-design

[YakDriver]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Driving Force: AWS has hinted (in the Console) that it will be deprecating extra_connection_attributes. In addition, it is bad (see below) having two ways of providing the exact same information: 1) using a single long string providing many attributes (e.g., extra_connection_attributes = "bucketFolder=value;bucketName=value;...) that maps to 2) many Terraform arguments (e.g., s3_settings.0.bucket_folder = "value" and s3_settings.0.bucket_name = "value").

Problems:

• It causes many diffs as the values AWS wants vary slightly depending on which way you use (e.g., extra_connection_attributes wants dataFormat=PARQUET_2_0 while S3Settings.DataFormat wants parquet-2-0) but then the provider is trying to marry them up. In addition, e.g., AWS only accepts all caps for some values but returns all lowercase.
• Handling the complexities between extra_connection_attributes and individual arguments is error prone. In addition, the mess that comes with handling so many different endpoint types and arguments in one resource is error prone.
• This approach is confusing because it varies significantly from the normal practitioner experience in Terraform.

Previous Design: Before, we could fit a bunch of endpoints into the same resource because we did not have individual Terraform arguments corresponding to each of the extra_connection_attributes. Without extra_connection_attributes, there are about 259 endpoint-specific attributes (159 unique attributes).

New Design Plan:

1. We add new endpoint-specific resources (see below), each of which will have a reasonable number of arguments.
2. The new resources will not have extra_connection_attributes. This means each attribute will need a Terraform argument. This allows us to avoid the mapping back and forth and makes it easier to handle the AWS API inconsistencies (e.g., caps to lowercase).
3. The current endpoint (aws_dms_endpoint) will continue as before but we will deprecate these arguments: elasticsearch_settings, extra_connection_attributes, kafka_settings, kinesis_settings, mongodb_settings, and s3_settings.
4. The current endpoint resource will still be used for some endpoint types because they either have no corresponding extra_connection_attributes or use only the set of common arguments: aurora, azuredb, db2, dynamodb, mariadb, and sybase.

New or Affected Resource(s)

• aws_dms_dms_transfer_endpoint
• aws_dms_db2_endpoint
• aws_dms_docdb_endpoint
• aws_dms_kafka_endpoint
• aws_dms_kinesis_endpoint
• aws_dms_mongodb_endpoint
• aws_dms_mysql_endpoint
• aws_dms_neptune_endpoint
• aws_dms_opensearch_endpoint
• aws_dms_oracle_endpoint
• aws_dms_postgres_endpoint
• aws_dms_redis_endpoint
• aws_dms_redshift_endpoint
• aws_dms_s3_endpoint
• aws_dms_sqlserver_endpoint

Potential Terraform Configuration

resource "aws_dms_s3_endpoint" "test" {
  endpoint_id   = "example"
  endpoint_type = "target"
  ssl_mode      = "none"

  tags = {
    Name   = "example"
    Update = "to-update"
    Remove = "to-remove"
  }

  add_column_name                             = true
  bucket_folder                               = "folder"
  bucket_name                                 = "updated_name"
  canned_acl_for_objects                      = "private"
  cdc_inserts_and_updates                     = true
  cdc_max_batch_interval                      = 100
  cdc_min_file_size                           = 16
  cdc_path                                    = "cdc/path"
  compression_type                            = "GZIP"
  csv_delimiter                               = ";"
  csv_no_sup_value                            = "x"
  csv_null_value                              = "?"
  csv_row_delimiter                           = "\\r\\n"
  data_format                                 = "parquet"
  data_page_size                              = 1100000
  date_partition_delimiter                    = "SLASH"
  date_partition_enabled                      = true
  date_partition_sequence                     = "yyyymmddhh"
  date_partition_timezone                     = "America/Eastern"
  dict_page_size_limit                        = 1000000
  enable_statistics                           = false
  encoding_type                               = "plain"
  encryption_mode                             = "SSE_S3"
  external_table_definition                   = "etd"
  ignore_header_rows                          = 1
  include_op_for_full_load                    = true
  max_file_size                               = 1000000
  parquet_timestamp_in_millisecond            = true
  parquet_version                             = "parquet-2-0"
  preserve_transactions                       = false
  rfc_4180                                    = true
  row_group_length                            = 11000
  service_access_role_arn                     = aws_iam_role.iam_role.arn
  timestamp_column_name                       = "tx_commit_time"
  use_csv_no_sup_value                        = true
  use_task_start_time_for_full_load_timestamp = true

  depends_on = [aws_iam_role_policy.dms_s3_access]
}

References

• AWS DMS Endpoint - Support for redshift settings #13162
• feat(dms_endpoint): add DataFormat, EncryptionMode, and KmsKeyId to the S3 settings block #14340
• Add dms endpoint s3 attributes to keep format from changing to default on update #14620
• resource/dms_endpoint: add support for extraConnectionAttributes in create/read ops and handle diffs #16827
• Add S3 Connection Attributes to DMS Endpoint #17591
• Include additional kinesis-settings in dms endpoint #18750
• r/aws_dms_endpoint: diff in extra_connection_attributes present when configured with s3_settings #19501
• Resource aws_dms_endpoint - Conflict between extra_connection_attributes and s3_settings #19667
• Database Migration Service provider 'resource_aws_dms_endpoint.go' breaks/drops extra-connection-attributes due to conditional logic error #20290
• Add neptune_settings configuration block in aws_dms_endpoint #20393
• Add postgre_sql_settings configuration block in aws_dms_endpoint #20395
• Add my_sql_settings configuration block in aws_dms_endpoint #20396
• Add oracle_settings configuration block in aws_dms_endpoint #20397
• Add sybase_settings configuration block in aws_dms_endpoint #20400
• Add microsoft_sql_server_settings configuration block in aws_dms_endpoint #20401
• aws_dms_endpoint - Add support for secrets_manager_arn with engine_name sqlserver #22405
• resource/dms: Add ability to use secrets manager for engine sqlserver #22646
• aws_dms_endpoint - Add support for DMS Endpoint settings #23225
• Add support for AWS Secrets Manager when creating DMS endpoints for aurora-postgresql engine #23463
• r/aws_dms_endpoint: Secrets manager secret for aws_dms_endpoint with engine_name = 'aurora', 'mariadb', or 'mysql' #24845
• r/aws_dms_endpoint: Secrets manager secret for aws_dms_endpoint with engine_name = 'redshift'  #24911

[samoclay]

would be great to hear any news on when this could be available

[slords]

Any update on this? Oracle endpoints are currently broken as the extra_connection_atributes (at least for sources) doesn't get applied. #20397 was raised because this doesn't work but it was pointed here as the solution. As this hasn't been implemented yet there is no solution for oracle endpoints. Any update that would provide guidance for how to solve this today would be appreciated.

[evbo]

there is currently no way to set mysql settings, would be great to have this or @mdjward workaround released:

https://docs.aws.amazon.com/dms/latest/APIReference/API_MySQLSettings.html