Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DynamoDB V2019.11.21 #12342

Merged
merged 16 commits into from
Apr 14, 2020
Merged

DynamoDB V2019.11.21 #12342

merged 16 commits into from
Apr 14, 2020

Conversation

cmonty
Copy link
Contributor

@cmonty cmonty commented Mar 11, 2020

Attempts to add support for DynamoDB Global Table V2019.11.21. The PR introduces a new resource aws_dynamodb_table_2019 as initially discussed in #11096. After implementing though, it may be better to fold replica into aws_dynamodb_table as you can add replica to existing tables (just not existing tables already replicated with V2017).

I've decided to change the initial PR to add replica to aws_dynamodb_table. I think there are benefits if you have an existing DynamoDB table without any GlobalTables, being able to add replica to the existing resource and having it add the new version of GlobalTables. Otherwise, you'd need to recreate the resource which would require a new table name or deleting an existing one.

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #11096

Release note for CHANGELOG:

* Adds support for DynamoDB Global Table V2019.11.21

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSDynamoDbTable2019_basic'
--- PASS: TestAccAWSDynamoDbTable2019_basic (127.79s)                                                     
PASS                                                                                              
ok      github.com/terraform-providers/terraform-provider-aws/aws       127.814s                                                                                                                         

@ghost ghost added size/XL Managed by automation to categorize the size of a PR. needs-triage Waiting for first response or review from a maintainer. provider Pertains to the provider itself, rather than any interaction with AWS. service/dynamodb Issues and PRs that pertain to the dynamodb service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Mar 11, 2020
@stolfi
Copy link
Contributor

stolfi commented Mar 18, 2020

Have you tried importing an existing 2019 replicated table?

Can you drop a sample bit of TF code as an example of the configuration?

@cmonty
Copy link
Contributor Author

cmonty commented Mar 18, 2020

Have you tried importing an existing 2019 replicated table?

If the table is replicated using version 2017, I'm not sure you can use the API to convert the table to a 2019 version. Let me check more on this but I think you may have to do it in the AWS console. Is this what you mean?

Can you drop a sample bit of TF code as an example of the configuration?

Configuration for what I've implemented so far (replica has more attributes that can be configured)

resource "aws_dynamodb_table" "test" {
  name         = "MyTestTable"
  hash_key     = "TestTableHashKey"
  billing_mode = "PAY_PER_REQUEST"
  stream_enabled = true
  stream_view_type = "NEW_AND_OLD_IMAGES"
  attribute {
    name = "TestTableHashKey"
    type = "S"
  }
  replica {
    region_name = "us-west-1"
  }
  replica {
    region_name = "au-southeast-2"
  }
}

@stolfi
Copy link
Contributor

stolfi commented Mar 18, 2020

Have you tried importing an existing 2019 replicated table?

If the table is replicated using version 2017, I'm not sure you can use the API to convert the table to a 2019 version. Let me check more on this but I think you may have to do it in the AWS console. Is this what you mean?

No, I was more thinking about a table I had in 2017, that I moved to 2019 via the console, and now want to import that change in to my existing statefiles.

Currently I have:

  • Table definition in region 1
  • Table definition in region 2
  • Global table definition

I'll need to remove the table definition in region 2 and global table definition, in favor of an update table definition in region 1

Now that I'm thinking about it, I may not have to import it at all....since the end state will already match the described state.

I'll give it a test and see what happens.

@cmonty
Copy link
Contributor Author

cmonty commented Mar 20, 2020

I'm running into issues being able to test additional attributes for ProvisionedThroughputOverride on both the table and global secondary indexes. The problem stems from the fact that AWS wants the table to either be "Pay-Per-Request or AutoScaled".

Pay-Per-Request is "easy" in that I can set the billing_mode to PAY_PER_REQUEST however when applying the UpdateTable AWS doesn't apply the override until the next day (the AWS Console states "Next available change to on-demand mode" is the next day and keeps the table in "Updating" status).

When adding the app auto-scaling targets and policies I still get a validation error from AWS, Table write capacity should either be Pay-Per-Request or AutoScaled. So, I'm not sure this would be a way out for the purposes of testing.

What are the typical strategies for applying changes that take 24 hours to apply?

Would it be reasonable to accept the PR as-is with only adding / deleting regions and not configuring read capacity overrides?

@cmonty cmonty marked this pull request as ready for review March 25, 2020 13:49
@cmonty cmonty requested a review from a team March 25, 2020 13:49
@stolfi
Copy link
Contributor

stolfi commented Mar 27, 2020

Typically things that I expect to take a long time to run, like a table convert to PPR, I will do in the console, or via the CLI, so that TF doesn't have to sit and wait for it.

@stolfi
Copy link
Contributor

stolfi commented Mar 27, 2020

After converting my tables to PROVISIONED, and back to PPR, my plan's are clean....not sure I can explain that.

On one, I did have to do an attribute reorder on a GSI, but that was it.

I've only been operating on my existing tables, I can do a couple table create/update/deletes to ensure those run smoothly.

@cmonty
Copy link
Contributor Author

cmonty commented Mar 30, 2020

Typically things that I expect to take a long time to run, like a table convert to PPR, I will do in the console, or via the CLI, so that TF doesn't have to sit and wait for it.

This is what I expected and makes me think updating read_capacity is not needed in the PR in favor of using the console or CLI.

@cmonty
Copy link
Contributor Author

cmonty commented Apr 1, 2020

Any update on what's needed for merge?

@bflad bflad removed needs-triage Waiting for first response or review from a maintainer. provider Pertains to the provider itself, rather than any interaction with AWS. labels Apr 3, 2020
@bflad bflad added the enhancement Requests to existing resources that expand the functionality or scope. label Apr 3, 2020
@bflad bflad self-assigned this Apr 3, 2020
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @cmonty 👋 Thank you for submitting this -- overall its off to a good start code-wise. Please see the below for some initial feedback and let us know if you have any questions or do not have time to implement the items.

Some other things to consider:

  • Documentation updates will need to be made within website/docs/r/dynamodb_table.html.markdown to add the new replica configuration block and its nested argument. The documentation for the configuration block should link to the AWS documentation for V2 global tables and explicitly note that it is for version 2019.11.21 while version 2017.11.29 must be managed via the aws_dynamodb_global_table resource. We should also consider adding something like the replica test configuration as an example to the documentation page.
  • Documentation updates to website/docs/r/dynamodb_global_table.html.markdown to denote that it is only for version 2017.11.29 global tables, linking to their AWS documentation
  • Potentially refactoring the suggested logic within the loops of updateDynamoDbReplica into the deleteDynamoDbReplicas/createDynamoDbReplicas, then just calling those two functions in the updateDynamoDbReplica instead of duplicating the logic both places.

Thanks again.

aws/resource_aws_dynamodb_table.go Outdated Show resolved Hide resolved
aws/resource_aws_dynamodb_table.go Outdated Show resolved Hide resolved
aws/resource_aws_dynamodb_table.go Outdated Show resolved Hide resolved
aws/resource_aws_dynamodb_table.go Outdated Show resolved Hide resolved
aws/resource_aws_dynamodb_table.go Outdated Show resolved Hide resolved
aws/resource_aws_dynamodb_table.go Outdated Show resolved Hide resolved
aws/resource_aws_dynamodb_table_test.go Outdated Show resolved Hide resolved
@bflad bflad added the waiting-response Maintainers are waiting on response from community or contributor. label Apr 3, 2020
@stolfi
Copy link
Contributor

stolfi commented Apr 3, 2020

Any update on what's needed for merge?

Not yet, but I should be doing a bit more testing today. I don't think we can not apply the change assuming that people will do via CLI or console....the state should match the code, unless there is a specific ignore_changes for it.

@ghost ghost removed the waiting-response Maintainers are waiting on response from community or contributor. label Apr 3, 2020
@stolfi
Copy link
Contributor

stolfi commented Apr 13, 2020

Any chance we can get the suggested changes merged in so we can do some additional testing?

Co-Authored-By: Brian Flad <bflad417@gmail.com>
@cmonty
Copy link
Contributor Author

cmonty commented Apr 14, 2020

Hi @cmonty 👋 Thank you for submitting this -- overall its off to a good start code-wise. Please see the below for some initial feedback and let us know if you have any questions or do not have time to implement the items.

Some other things to consider:

  • Documentation updates will need to be made within website/docs/r/dynamodb_table.html.markdown to add the new replica configuration block and its nested argument. The documentation for the configuration block should link to the AWS documentation for V2 global tables and explicitly note that it is for version 2019.11.21 while version 2017.11.29 must be managed via the aws_dynamodb_global_table resource. We should also consider adding something like the replica test configuration as an example to the documentation page.
  • Documentation updates to website/docs/r/dynamodb_global_table.html.markdown to denote that it is only for version 2017.11.29 global tables, linking to their AWS documentation
  • Potentially refactoring the suggested logic within the loops of updateDynamoDbReplica into the deleteDynamoDbReplicas/createDynamoDbReplicas, then just calling those two functions in the updateDynamoDbReplica instead of duplicating the logic both places.

Thanks again.

Thanks for you feedback! I've incorporated the suggestions and will work on the remaining updates to documentation and refactoring. Sorry for the delay!

@cmonty
Copy link
Contributor Author

cmonty commented Apr 14, 2020

Any chance we can get the suggested changes merged in so we can do some additional testing?

I've just accepted the suggestions. If travis fails I can fix it up tomorrow.

@bflad bflad added this to the v2.58.0 milestone Apr 14, 2020
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi again @cmonty 👋 Thanks so much for those updates. Since this was pretty close (e.g. just a lingering CI failure and a few other small things), we went ahead and took care of this. Appreciate all your hard work here!

Output from acceptance testing:

--- PASS: TestAccAWSDynamoDbTable_attributeUpdate (588.92s)
--- PASS: TestAccAWSDynamoDbTable_attributeUpdateValidation (7.18s)
--- PASS: TestAccAWSDynamoDbTable_basic (30.14s)
--- PASS: TestAccAWSDynamoDbTable_BillingMode_GSI_PayPerRequestToProvisioned (72.32s)
--- PASS: TestAccAWSDynamoDbTable_BillingMode_GSI_ProvisionedToPayPerRequest (882.42s)
--- PASS: TestAccAWSDynamoDbTable_BillingMode_PayPerRequestToProvisioned (49.30s)
--- PASS: TestAccAWSDynamoDbTable_BillingMode_ProvisionedToPayPerRequest (1025.15s)
--- PASS: TestAccAWSDynamoDbTable_disappears (17.38s)
--- PASS: TestAccAWSDynamoDbTable_disappears_PayPerRequestWithGSI (69.76s)
--- PASS: TestAccAWSDynamoDbTable_enablePitr (95.50s)
--- PASS: TestAccAWSDynamoDbTable_encryption (169.91s)
--- PASS: TestAccAWSDynamoDbTable_extended (275.87s)
--- PASS: TestAccAWSDynamoDbTable_gsiUpdateCapacity (66.23s)
--- PASS: TestAccAWSDynamoDbTable_gsiUpdateNonKeyAttributes (282.44s)
--- PASS: TestAccAWSDynamoDbTable_gsiUpdateOtherAttributes (656.30s)
--- PASS: TestAccAWSDynamoDbTable_Replica (538.95s)
--- PASS: TestAccAWSDynamoDbTable_streamSpecification (45.82s)
--- PASS: TestAccAWSDynamoDbTable_streamSpecificationValidation (5.76s)
--- PASS: TestAccAWSDynamoDbTable_tags (40.50s)
--- PASS: TestAccAWSDynamoDbTable_Ttl_Disabled (41.32s)
--- PASS: TestAccAWSDynamoDbTable_Ttl_Enabled (28.51s)

--- PASS: TestAccDataSourceAwsDynamoDbTable_basic (44.33s)

@bflad bflad merged commit 3c61296 into hashicorp:master Apr 14, 2020
bflad added a commit that referenced this pull request Apr 14, 2020
@cmonty
Copy link
Contributor Author

cmonty commented Apr 15, 2020

@bflad thanks for your help and merging!

@ghost
Copy link

ghost commented Apr 17, 2020

This has been released in version 2.58.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@gyoga99
Copy link

gyoga99 commented Apr 22, 2020

Trying to create a global dynamodb table, with GSI, provisioned billing mode and using autoscaling policy
required_providers {
aws = "~> 2.58"
}
I am seeing issue with creating a table with replica field set.
error updating DynamoDB Table (sample_table) replica: error creating DynamoDB Table (sample_table) replicas: ValidationException: Table write capacity should either be Pay-Per-Request or AutoScaled.

@giacomomagini
Copy link

I'm having some troubles with Global tables V2 and multiple replicas.
Error: error updating DynamoDB Table (my_table) replica: error creating DynamoDB Table (my_table) replicas: ValidationException: Update table operation with more than one create or delete replica actions not allowed

@ghost
Copy link

ghost commented May 15, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators May 15, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/dynamodb Issues and PRs that pertain to the dynamodb service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Version 2 of DynamoDB Global Tables
5 participants