dms: New resource aws_dms_s3_endpoint

[YakDriver]

Description

This PR addresses at least 3 challenges currently swirling around DMS endpoints.

1. Extra connection attributes aren't scaling well for AWS and all the endpoint types available. Correspondingly, the aws_dms_endpoint resource is not well suited to handling the many endpoint types, and their unique attributes, without the use of extra connection attributes. The resource is stretched thin (read it's a mess) already with attempts to handle so many endpoints and all the variious edge cases associated with each.
2. As AWS moves closer to deprecating extra connection attributes, it appears that they are dropping support for some attributes, like compression_type. You can still use compression_type but not through extra connection attributes.
3. AWS seems to have tightened up some validation rules. For example, previously, including data_partition_delimiter in a request to create a source endpoint did not cause problems. Now, it causes an error.

Relations

Closes #27852
Closes #27980
Relates #23507

References

Output from Acceptance Testing

$ make testacc TESTS=TestAccXXX PKG=ec2

...

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

• Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
• Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

[github-actions]

This functionality has been released in v4.45.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[sbuzan]

I tried the new aws_dms_s3_endpoint, and I am still getting the "compressionType...errror (see details below).

terraform version = 1.3.6
aws provider version = 4.45.0

main.tf =

`terraform {
required_version = ">= 1.3.6"

required_providers {
    aws      = ">= 4.45.0"
}

}

provider "aws" { region = "us-east-2" }

resource "aws_dms_s3_endpoint" "my_endpoint" {
endpoint_id = "my-test"
endpoint_type = "source"
bucket_folder = "my-folder/"
bucket_name ="my-bucket"
csv_delimiter =","
csv_row_delimiter ="\n"
date_partition_delimiter ="SLASH"
date_partition_enabled =false
date_partition_sequence ="YYYYMMDD"
service_access_role_arn = "arn:aws:iam::12345:role/my/role"
cdc_path = "my-cdc-path"
external_table_definition = file("./reader_table_definition.json")
} `

Console output:

aws_dms_s3_endpoint.my_endpoint: Creating... ╷ │ Error: creating AWS DMS (Database Migration) S3 Endpoint (my-test): InvalidParameterValueException: Unsupported value 'compressionType' for extra connection attributes │ status code: 400, request id: 10f9cc6d-59d7-4dbf-a90b-8afd327c5958 │ │ with aws_dms_s3_endpoint.my_endpoint, │ on main.tf line 11, in resource "aws_dms_s3_endpoint" "my_endpoint": │ 11: resource "aws_dms_s3_endpoint" "my_endpoint" {

[YakDriver]

@sbuzan @BrandonF8181 It turns out that the error is region specific. It works fine in us-west-2, where I tested it, but I'm also seeing it fail in us-east-2, but only for source endpoints. 😞

[YakDriver]

This should be resolved with #28220 @sbuzan @BrandonF8181. Let us know if not.

[narnaldo]

failing also on us-east-1

[sbuzan]

@YakDriver I tested 4.45.0 in us-west-2 and confirmed that the aws_dms_s3_endpoint deploys. However, the aws_dms_replication_task cannot read the new endpoint type as a valid source. Cdc tasks fail when validating the endpoint parameters. It is looking for "Parameter cdcPath." I believe that it is checking the extra connection attributes which do not exist on the new endpoint.

resource "aws_dms_s3_endpoint" "my_endpoint" {
endpoint_id = "my-test"
endpoint_type = "source"
bucket_folder = "my-folder/"
bucket_name ="my-bucket"
csv_delimiter =","
csv_row_delimiter ="\n"
date_partition_delimiter ="SLASH"
date_partition_enabled =false
date_partition_sequence ="YYYYMMDD"
service_access_role_arn = "arn:aws:iam::12345:role/my/role"
cdc_path = "my-cdc-path"
external_table_definition = file("./reader_table_definition.json")
} `

resource "aws_dms_replication_task" "reader_cdc_replication_task" { migration_type = "cdc" source_endpoint_arn = aws_dms_s3_endpoint.my_endpoint.endpoint_arn }

console output with task added:

aws_dms_s3_endpoint.source: Creation complete after 0s [id=dms-s3-source-endpoint-test] aws_dms_replication_task.reader_cdc_replication_task: Creating... ╷ │ Error: error creating DMS Replication Task (dms-test-cdc): **InvalidParameterValueException**: **Parameter cdcPath** is required when task is configured for ongoing replication with Amazon S3 as a source. │ status code: 400, request id: 736fc34f-10be-4250-8547-630827bf5cd6 │ │ with aws_dms_replication_task.reader_cdc_replication_task, │ on main.tf line 45, in resource "aws_dms_replication_task" "reader_cdc_replication_task": │ 45: resource "aws_dms_replication_task" "reader_cdc_replication_task" {

[sbuzan]

@YakDriver , I just tested 4.46.0. I get the same error above when attaching a "cdc" task to the endpoint. The the task appears to be looking for cdcPath in the ExtraConnectionAttributes, which we no longer populate. (I duplicated this behavior with a CloudFormation deployment).

Condition:

Deploy the following:
(abbreviated tag)
resource aws_dms_replication_task {
migration_type = "cdc"
source_endpoint_arn = aws_dms_s3_endpoint.source.endpoint_arn
}

console output:

aws_dms_replication_task.reader_cdc_replication_task: Creating...
╷
│ Error: error creating DMS Replication Task (dms-test-cdc): InvalidParameterValueException: Parameter cdcPath is required when task is configured for ongoing replication with Amazon S3 as a source.
│ status code: 400, request id: f4a8d0dc-decc-4450-b874-f4840a4db9a9

[thundermann]

I also noticed that documentation for the attribute ignore_headers_row in the new aws_dms_s3_endpoint is actually supposed to be named ignore_header_rows

[narnaldo]

for source type some of the parameters shouldn't be sent and not set with default value.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.