Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dms endpoint s3 attributes to keep format from changing to default on update #14620

Closed
wants to merge 2 commits into from

Conversation

Niarfe
Copy link

@Niarfe Niarfe commented Aug 13, 2020

This is a resubmission of #11576

Original description:

Extend DMS S3 settings to support updated API changes in aws-sdk-go.
This is so that we can avoid using extra_connection_attributes.

I've tested functionality to dms endpoint for the issue that I want to address.

Issue: Given a created dms migration with an s3 target endpoint in parquet format, if you updated and execute a terraform update the target endpoint format reverts to csv.

With current changes: Created a dms migration with an s3 target endpoint, then changed the name of the bucket, ran terraform apply and verified the migration still proceeded with original parquet format.

Relates to #11576

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Relates to #11576

Release note for CHANGELOG:

ENHANCEMENTS:

resource/aws_dms_endpoint: Added missing s3_settings attributes ServiceAccessRoleArn, ExternalTableDefinition, CsvRowDelimiter, CsvDelimiter, BucketFolder, BucketName, CompressionType, TimestampColumnName, DataFormat, ParquetVersion, EncryptionMode, ServerSideEncryptionKmsKeyId

Output from acceptance testing:

$ go version
go version go1.15 linux/amd64

$ make testacc TEST=./aws TESTARGS='-run=TestAccAwsDmsEndpoint'
...
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch (157.41s)    
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration (173.00s)    
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage (173.57s)    
--- PASS: TestAccAwsDmsEndpoint_Db2 (223.90s)    
--- PASS: TestAccAwsDmsEndpoint_DocDB (225.37s)    
--- PASS: TestAccAwsDmsEndpoint_Kafka_Broker (226.68s)    
--- PASS: TestAccAwsDmsEndpoint_basic (226.69s)    
--- PASS: TestAccAwsDmsEndpoint_Kafka_Topic (227.48s)    
--- PASS: TestAccAwsDmsEndpoint_MongoDb (228.72s)    
--- PASS: TestAccAwsDmsEndpoint_DynamoDb (238.63s)    
--- PASS: TestAccAwsDmsEndpoint_S3_Parquet (238.97s)    
--- PASS: TestAccAwsDmsEndpoint_S3_Csv (239.37s)    
--- PASS: TestAccAwsDmsEndpoint_Kinesis (251.62s)    
PASS    
ok      github.com/terraform-providers/terraform-provider-aws/aws   251.673s 

@Niarfe Niarfe requested a review from a team August 13, 2020 09:53
@ghost ghost added size/L Managed by automation to categorize the size of a PR. service/databasemigrationservice tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. needs-triage Waiting for first response or review from a maintainer. labels Aug 13, 2020
Base automatically changed from master to main January 23, 2021 00:58
@breathingdust breathingdust requested a review from a team as a code owner January 23, 2021 00:58
@azhurbilo
Copy link

@breathingdust could you check this MR please? we also use S3 for DMS as target and need setup "DataFormat=parquet;ParquetVersion=PARQUET_2_0"

@YakDriver
Copy link
Member

Related #14340
Related #17591
Related #18750

@breathingdust breathingdust added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Sep 16, 2021
@zhelding
Copy link
Contributor

Pull request #21306 has significantly refactored the AWS Provider codebase. As a result, most PRs opened prior to the refactor now have merge conflicts that must be resolved before proceeding.

Specifically, PR #21306 relocated the code for all AWS resources and data sources from a single aws directory to a large number of separate directories in internal/service, each corresponding to a particular AWS service. This separation of code has also allowed for us to simplify the names of underlying functions -- while still avoiding namespace collisions.

We recognize that many pull requests have been open for some time without yet being addressed by our maintainers. Therefore, we want to make it clear that resolving these conflicts in no way affects the prioritization of a particular pull request. Once a pull request has been prioritized for review, the necessary changes will be made by a maintainer -- either directly or in collaboration with the pull request author.

For a more complete description of this refactor, including examples of how old filepaths and function names correspond to their new counterparts: please refer to issue #20000.

For a quick guide on how to amend your pull request to resolve the merge conflicts resulting from this refactor and bring it in line with our new code patterns: please refer to our Service Package Refactor Pull Request Guide.

@YakDriver
Copy link
Member

Thanks so much for this submission! There's a lot of good work here. Unfortunately, many of the issues have been resolved apart from this PR. We value your time in submissions and do our best to make sure contributions don't go to waste. We're sorry for this not being used. Rest assured we are always seeking to improve our processes so that this doesn't happen again.

@YakDriver YakDriver closed this Jan 6, 2023
@github-actions
Copy link

github-actions bot commented Feb 6, 2023

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/dms Issues and PRs that pertain to the dms service. size/L Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants