DynamoDB On Demand Scaling

[kyeotic]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Dynamo DB released a new on-demand throughput mode today for all regions. It does not require read/write throughput to be specified on the table or any indexes.

New or Affected Resource(s)

• aws_dynamodb_table
• aws_dynamodb_global_table

Potential Terraform Configuration

resource "aws_dynamodb_table" "data" {
  name           = "${local.dynamo_table_prefix}_data"
  hash_key       = "id"

  attribute {
    name = "id"
    type = "S"
  }

  attribute {
    name = "type"
    type = "S"
  }

  global_secondary_index {
    name            = "type-index"
    hash_key        = "type"
    range_key       = "id"
    projection_type = "ALL"
  }

  ttl {
    attribute_name = "ttl"
    enabled        = true
  }

  lifecycle {
    prevent_destroy = true
  }

  // the good stuff
  billing_mode = "PAY_PER_REQUEST" | "PROVISIONED"

  server_side_encryption {
    enabled = true
  }

  point_in_time_recovery {
    enabled = true
  }
}

References

Relasee Notes

[sunilkumarmohanty]

I would suggest we have a new field called billing_mode which can be either "PROVISIONED" or "PAY_PER_REQUEST". I recommend this one, since the aws sdk has added a new field called BillingMode.

[balassit]

@sunilkumarmohanty agreed. Documentation for reference of BillingMode can be found here.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html

[kyeotic]

@sunilkumarmohanty I've updated my comment to match

[bflad]

Support for the billing_mode argument has been merged and will release with version 1.51.0 of the AWS provider, likely middle of next week. 👍

[mschroering]

I left a comment on the PR, but I believe this will not work for tables with global secondary indexes as the read_capacity and write_capacity need to be made optional for on demand tables.

[kyeotic]

@sunilkumarmohanty ☝️

[bflad]

The initial implementation has been released in version 1.51.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

I would suggest creating a separate issue for making GSI capacities optional.

[philipmclifton]

I think that read and write are still required on GSI.

 global_secondary_index {
    name            = "author"
    hash_key        = "authorId"
    read_capacity   = 1
    write_capacity  = 1
    range_key       = "dueAt"
    projection_type = "ALL"
  }

Error: aws_dynamodb_table.assignments: "global_secondary_index.0.read_capacity": required field is not set
Error: aws_dynamodb_table.assignments: "global_secondary_index.0.write_capacity": required field is not set

aws_dynamodb_table.assignments: ValidationException: One or more parameter values were invalid: ProvisionedThroughput should not be specified for index: author when BillingMode is PAY_PER_REQUEST

[willtrking]

@philipmclifton I ran into the same issue as you and was able to work around it by setting those values to "0"

Here's a sample of what I'm using across a few different tables

resource "aws_dynamodb_table" "blocks" {
  name           = "${var.blocks_dynamodb_table_name}"
  billing_mode   = "${var.blocks_billing_mode}"
  read_capacity  = "${var.blocks_billing_mode == "PROVISIONED" ? var.blocks_base_read_capacity : 0}"
  write_capacity = "${var.blocks_billing_mode == "PROVISIONED" ? var.blocks_base_write_capacity : 0}"
  hash_key       = "userId"
  range_key      = "blockedUser"

  lifecycle {
    ignore_changes = ["read_capacity", "write_capacity", "global_secondary_index"]
  }

  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"

  attribute {
    name = "userId"
    type = "S"
  }

  attribute {
    name = "blockedUser"
    type = "S"
  }

  global_secondary_index {
    name      = "Users__BlockedBy"
    hash_key  = "blockedUser"
    range_key = "userId"

    read_capacity   = "${var.blocks_billing_mode == "PROVISIONED" ? var.blocks_base_read_capacity : 0}"
    write_capacity  = "${var.blocks_billing_mode == "PROVISIONED" ? var.blocks_base_write_capacity : 0}"
    projection_type = "ALL"
  }

  point_in_time_recovery {
    enabled = true
  }
}

[LiborVilimekMassive]

@willtrking - it does not work for me, I get this kind of error:

• minimum field value of 1, UpdateTableInput.GlobalSecondaryIndexUpdates[0].Create.ProvisionedThroughput.ReadCapacityUnits.

[Hinidu]

@LiborVilimekMassive I think that this error can be safely ignored. There will be no errors on next apply and billing change is actually applied.

[dshefchik]

@Hinidu it does eventually work, but this error will fail any dependent resources or subsequent pipeline steps and leave our infrastructure in a weird state. This should be addressed.

[danieladams456]

Version 1.52 has the fix #6737.

[rosscdh]

Something similar seems to have crept back in 1.58 for aws_dynamodb_table

billing_mode is set to PAY_PER_REQUEST, and we are trying to apply a new range_key = "typeAndId"

* provider.aws: version = "~> 1.58"

Terraform has been successfully initialized!

2019-02-14T18:11:22.680+0100 [DEBUG] plugin.terraform-provider-aws_v1.58.0_x4: 2019/02/14 

* aws_dynamodb_table.MY_TABLE_NAME-table: ValidationException: At least one of ProvisionedThroughput, BillingMode, UpdateStreamEnabled, GlobalSecondaryIndexUpdates or SSESpecification is required
	status code: 400, request id: VN5Q468GITTHMM2T124245LSBJVV4KQNSO5AEMVJF66Q9ASUAAJG

[ahajimir]

Also seeing issues in 1.58

* provider.aws: version = "~> 1.58"
* provider.template: version = "~> 2.0"

Terraform has been successfully initialized!

* aws_dynamodb_table.table-name: InvalidParameter: 1 validation error(s) found.

- missing required field, UpdateTableInput.GlobalSecondaryIndexUpdates[0].Update.ProvisionedThroughput.

I have the following defined

billing_mode     = "PAY_PER_REQUEST"

And looking at the table, everything is set to ON-DEMAND

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!