Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow Elasticsearch to be used as an engine for DMS Endpoint #11792

Merged
merged 1 commit into from
Apr 15, 2020
Merged

Allow Elasticsearch to be used as an engine for DMS Endpoint #11792

merged 1 commit into from
Apr 15, 2020

Conversation

timrcoulson
Copy link
Contributor

@timrcoulson timrcoulson commented Jan 29, 2020 •
edited
Loading

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #9899

Release note for CHANGELOG:

resource_aws_dms_endpoint: allow `elasticsearch` to be used for the engine for DMS

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAwsDmsEndpoint_Elasticsearch'
?       github.com/terraform-providers/terraform-provider-aws   [no test files]
=== RUN   TestAccAwsDmsEndpoint_Elasticsearch
=== PAUSE TestAccAwsDmsEndpoint_Elasticsearch
=== CONT  TestAccAwsDmsEndpoint_Elasticsearch
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch (46.01s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       46.047s
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/flatmap      0.022s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/keyvaluetags 0.006s [no tests to run]

@timrcoulson timrcoulson requested a review from a team January 29, 2020 15:52
@ghost ghost added size/L Managed by automation to categorize the size of a PR. needs-triage Waiting for first response or review from a maintainer. documentation Introduces or discusses updates to documentation. service/databasemigrationservice tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Jan 29, 2020
@timrcoulson timrcoulson changed the title add dms elasticsearch target Allow elasticsearch to be used as an engine for DMS Endpoint Jan 29, 2020
@timrcoulson timrcoulson changed the title Allow elasticsearch to be used as an engine for DMS Endpoint Allow Elasticsearch to be used as an engine for DMS Endpoint Jan 29, 2020
johanwiren
johanwiren reviewed Mar 2, 2020
View reviewed changes
aws/resource_aws_dms_endpoint.go Outdated
"error_retry_duration": {
Type: schema.TypeInt,
Optional: true,
Default: "10",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Elasticsearch.html the default should be 300

johanwiren
johanwiren reviewed Mar 2, 2020
View reviewed changes
aws/resource_aws_dms_endpoint.go Outdated
"full_load_error_percentage": {
Type: schema.TypeInt,
Optional: true,
Default: "300",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this one should default to 10

johanwiren
johanwiren reviewed Mar 2, 2020
View reviewed changes
website/docs/r/dms_endpoint.html.markdown Outdated
@@ -65,6 +65,7 @@ The following arguments are supported:
* `service_access_role` - (Optional) The Amazon Resource Name (ARN) used by the service access IAM role for dynamodb endpoints.
* `mongodb_settings` - (Optional) Settings for the source MongoDB endpoint. Available settings are `auth_type` (default: `password`), `auth_mechanism` (default: `default`), `nesting_level` (default: `none`), `extract_doc_id` (default: `false`), `docs_to_investigate` (default: `1000`) and `auth_source` (default: `admin`). For more details, see [Using MongoDB as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MongoDB.html).
* `s3_settings` - (Optional) Settings for the target S3 endpoint. Available settings are `service_access_role_arn`, `external_table_definition`, `csv_row_delimiter` (default: `\\n`), `csv_delimiter` (default: `,`), `bucket_folder`, `bucket_name` and `compression_type` (default: `NONE`). For more details, see [Using Amazon S3 as a Target for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html).
* `elasticsearch_settings` - (Optional) Settings for the target Elasticsearch. Available settings are `service_access_role_arn`, `endpoint_uri`, `error_retry_duration` (default: `10`) and `full_load_error_percentage` (default: `300`). For more details, see [Using an Amazon Elasticsearch Service Cluster as a Target for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Elasticsearch.html).
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default values for error_retry_duration and full_load_error_percentage are swapped.

@timrcoulson
Copy link
Contributor Author

@johanwiren swapped the default values

@ShaneCal
Copy link

We urgently need support for this. We are planning on streaming data from our production databases to ELK in the next week and I really don't want to put this in through the console

@bflad bflad added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Apr 15, 2020
@bflad bflad self-assigned this Apr 15, 2020
@bflad bflad added this to the v2.58.0 milestone Apr 15, 2020
bflad
bflad approved these changes Apr 15, 2020
View reviewed changes
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @timrcoulson 👋 Thank you for submitting this. Looks good. I'll be merging this along with #8633 so you don't need to fix merge conflicts. 👍

Output from acceptance testing:

--- PASS: TestAccAwsDmsEndpoint_Basic (28.67s)
--- PASS: TestAccAwsDmsEndpoint_Db2 (29.07s)
--- PASS: TestAccAwsDmsEndpoint_DocDB (29.22s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration (35.89s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch (36.07s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage (36.14s)
--- PASS: TestAccAwsDmsEndpoint_MongoDb (36.37s)
--- PASS: TestAccAwsDmsEndpoint_DynamoDb (38.07s)
--- PASS: TestAccAwsDmsEndpoint_S3 (39.49s)
--- PASS: TestAccAwsDmsEndpoint_Kinesis (82.14s)

@bflad bflad merged commit 2fd64aa into hashicorp:master Apr 15, 2020
bflad added a commit that referenced this pull request Apr 15, 2020
@bflad
Update CHANGELOG for #8633 and #11792
03905ce
@ghost
Copy link

ghost commented Apr 17, 2020

This has been released in version 2.58.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented May 16, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators May 16, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@johanwiren johanwiren johanwiren left review comments

@bflad bflad bflad approved these changes

Assignees

@bflad bflad

Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. size/L Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v2.58.0
Development

Successfully merging this pull request may close these issues.

Add support for aws_dms_endpoint elasticsearch target engine type
4 participants
@timrcoulson @ShaneCal @bflad @johanwiren