[Bug]: Getting operation error ElastiCache while modifying storage for aws_elasticache_serverless_cache

[Chandan-Nayak]

Terraform Core Version

1.4.6

AWS Provider Version

5.71.0

Affected Resource(s)

A aws_elasticache_serverless_cache resource is created with storage 10 GB.
Creation is done and I want to modify the storage to 20 GB.

Terraform apply produces below mentioned error

│ Error: updating ElastiCache Serverless Cache (test-elastic-cache-cluster)
│ 
│   with module.elasticache.aws_elasticache_serverless_cache.this,
│   on ../../modules/elasticache/main.tf line 23, in resource "aws_elasticache_serverless_cache" "this":
│   23: resource "aws_elasticache_serverless_cache" "this" {
│ 
│ operation error ElastiCache: ModifyServerlessCache, https response error StatusCode: 400, RequestID: 72c2dda3-2fca-4a0e-91ba-b57a8d685729, InvalidParameterValue: This API
│ supports only cross-engine upgrades to Valkey engine currently

Steps to reproduce

1. Create aws_elasticache_serverless_cache resource (major_engine_version - 7, engine - redis)
2. Try to change the storage size from 10 to 20

Expected Behavior

Terraform should be able to apply the change successfully

Actual Behavior

Getting below error

│ Error: updating ElastiCache Serverless Cache (test-elastic-cache-cluster)
│ 
│   with module.elasticache.aws_elasticache_serverless_cache.this,
│   on ../../modules/elasticache/main.tf line 23, in resource "aws_elasticache_serverless_cache" "this":
│   23: resource "aws_elasticache_serverless_cache" "this" {
│ 
│ operation error ElastiCache: ModifyServerlessCache, https response error StatusCode: 400, RequestID: 72c2dda3-2fca-4a0e-91ba-b57a8d685729, InvalidParameterValue: This API
│ supports only cross-engine upgrades to Valkey engine currently

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_elasticache_serverless_cache" "this" {
  name                 = local.cluster_name
  engine               = var.engine
  major_engine_version = var.major_engine_version

  cache_usage_limits {
    data_storage {
      maximum = var.data_storage_maximum
      unit    = "GB"
    }
    ecpu_per_second {
      maximum = var.ecpu_per_second_maximum
    }
  }

  description        = var.description
  kms_key_id         = var.kms_key_id
  security_group_ids = [aws_security_group.elastic-cache-sg.id]
  subnet_ids         = [data.aws_subnets.selected.ids[0], data.aws_subnets.selected.ids[1]]
  tags               = local.cluster_tags

  #redis only stuff
  snapshot_retention_limit = var.snapshot_retention_limit
}

variable "engine" {
  default = "redis"
}

variable "vpc_tag_name" {
  type    = list(string)
  default = []
}

variable "description" {
  type    = string
  default = "Managed by Terraform"
}

variable "data_storage_maximum" {
  type    = number
  default = 10
}

variable "ecpu_per_second_maximum" {
  type    = number
  default = 5000
}

variable "kms_key_id" {
  default     = null
  description = "kms_key_id used for encryption on rest"
  type        = string
}

variable "snapshot_retention_limit" {
  default = 0
}

variable "ingress_cidr_blocks" {
  type = list(string)
}

variable "additional_security_group_ids" {
  type        = list(string)
  description = "The ids of the security groups from which traffic needs to allowed."
  default     = []
}

variable "major_engine_version" {
  type    = number
  default = 7
}

variable "cluster_name" {
  type = string
}

Steps to Reproduce

Steps to reproduce

1. Create aws_elasticache_serverless_cache resource (major_engine_version - 7, engine - redis)
2. Try to change the storage size from 10 to 20

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[stefanfreitag]

Good evening @Chandan-Nayak!
Thank you for raising the issue.

I can confirm that it can be reproduced with version 5.71.0 of the Terraform provider. I used below code in combination with the provider and Terraform version 1.9.3.

resource "aws_elasticache_serverless_cache" "example" {
  engine = "redis"
  name   = "example"
  cache_usage_limits {
    data_storage {
      maximum = 10
      unit    = "GB"
    }
    ecpu_per_second {
      maximum = 5000
    }
  }
  daily_snapshot_time      = "09:00"
  description              = "Test Server"
  major_engine_version     = "7"
  snapshot_retention_limit = 1
  security_group_ids       = ["sg-<redacted>"]
  subnet_ids               = ["subnet-<redacted>","subnet-<redacted>", "subnet-<redacted>"]
}

Out of curiosity I tried the older provider version 5.70.0 and did not run into the issue. If you can spare a few minutes, could you please test and confirm from your end?

Valkey support for Elasticache was added by AWS last week and there another open issue related to Elasticache and Valkey support: #39641

[Chandan-Nayak]

@stefanfreitag Thank you for a quick response.
I have tried it on aws provider 5.70.0 and it worked. So the issue is only with 5.71.0.

[stefanfreitag]

There is one difference I spotted between the two provider versions

• v5.70.0 uses api/elasticache#1.41.3

  2024-10-15T20:54:07.841+0200 [DEBUG] provider.terraform-provider-aws_v5.70.0_x5: HTTP Request Sent: aws.region=eu-central-1 http.request_content_length=327 http.url=https://elasticache.eu-central-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.3 (+https://www.terraform.io) terraform-provider-aws/5.70.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.31.0 os/linux lang/go#1.23.1 md/GOOS#linux md/GOARCH#amd64 api/elasticache#1.41.3"  ...
  

• v5.71.0 uses api/elasticache#1.43.0

  http.request.header.amz_sdk_request="attempt=1; max=25" rpc.method=ModifyServerlessCache http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.9.3 (+https://www.terraform.io) terraform-provider-aws/5.71.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.2 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/elasticache#1.43.0" ...
  

In the AWS SDK GO v2 I found for service/elasticache/api_op_ModifyServerlessCache.go a change that introduces

// Modifies the engine listed in a serverless cache request. The options are
// redis, memcached or valkey.
Engine *string
// Modifies the engine vesion listed in a serverless cache request.
MajorEngineVersion *string

as input to the ModifyServerlessCache operation. It came in with this commit and matches the changes listed here.

[ewbankkit]

We are getting the same error in CI:

=== RUN   TestAccElastiCacheServerlessCache_update
=== PAUSE TestAccElastiCacheServerlessCache_update
=== CONT  TestAccElastiCacheServerlessCache_update
    serverless_cache_test.go:214: Step 3/3 error: Error running apply: exit status 1
        Error: updating ElastiCache Serverless Cache (tf-acc-test-6481413601024454896)
          with aws_elasticache_serverless_cache.test,
          on terraform_plugin_test.tf line 12, in resource "aws_elasticache_serverless_cache" "test":
          12: resource "aws_elasticache_serverless_cache" "test" {
        operation error ElastiCache: ModifyServerlessCache, https response error
        StatusCode: 400, RequestID: adb31b87-cf1a-470a-9cc7-34150588c90c,
        InvalidParameterValue: This API supports only cross-engine upgrades to Valkey
        engine currently.
--- FAIL: TestAccElastiCacheServerlessCache_update (458.52s)

It looks like #39745 fixes this, so I think we'll prioritize that PR.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.73.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.