unexpected state 'NOT_AVAILABLE', wanted target 'AVAILABLE' in aws_api_gateway_stage

[meichhorn-conet]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

> terraform -version
Terraform v1.1.4
on windows_amd64
+ provider registry.terraform.io/hashicorp/aws v3.74.0
+ provider registry.terraform.io/hashicorp/external v2.2.0

Affected Resource(s)

• aws_api_gateway_stage

Terraform Configuration Files

resource "aws_api_gateway_stage" "service_api_stage" {
  stage_name           = var.api["stage_name"]
  rest_api_id             = aws_api_gateway_rest_api.service_api.id
  deployment_id      = aws_api_gateway_deployment.service_api_deployment.id
  xray_tracing_enabled = true
  
  # added this value to avoid constant updates of the resource
  cache_cluster_size = "0.5"

  # tags ...
}

Debug Output

-----------------------------------------------------: timestamp=2022-02-01T09:02:19.840+0100
2022-02-01T09:02:19.840+0100 [INFO]  provider.terraform-provider-aws_v3.74.0_x5.exe: 2022/02/01 09:02:19 [DEBUG] [aws-sdk-go] {"cacheClusterEnabled":false,"cacheClusterSize":"0.5","cacheClusterStatus":"NOT_AVAILABLE","createdDate":1643702539,"deploymentId":"h7wlh1","lastUpdatedDate":1643702539,"methodSettings":{},"stageName":"test","tags":{"Name":"test","XXXXX":"XXXXX","XXXXX":"XXXXX"},"tracingEnabled":true}: timestamp=2022-02-01T09:02:19.840+0100
2022-02-01T09:02:19.844+0100 [ERROR] vertex "module.xxxx-service-name.aws_api_gateway_stage.service_api_stage" error: error waiting for API Gateway Stage (ags-97zgip22qh-test) to be available: unexpected state 'NOT_AVAILABLE', wanted target 'AVAILABLE'. last error: %!s(<nil>)
2022-02-01T09:02:20.233+0100 [INFO]  provider.terraform-provider-aws_v3.74.0_x5.exe: 2022/02/01 09:02:20 [DEBUG] [aws-sdk-go] DEBUG: Response s3/UploadPart Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 0
Date: Tue, 01 Feb 2022 08:02:20 GMT
Etag: "350f670a3ce95719216d1777da245bc6"
Server: AmazonS3
X-Amz-Id-2: WxUIFvS2CRQlUtGGIjDS3kDoesPAg8TvWTRmRoXkeecZmQfV2oghFWCcDkmJgVLqhZejh+DfoP0=
X-Amz-Request-Id: 1FBEYBT1Q2DX3HY2
X-Amz-Server-Side-Encryption: AES256

Aws Cli output

> aws apigateway get-stage --rest-api-id 97zgip22qh --stage-name test
{
    "deploymentId": "h7wlh1",
    "stageName": "test",
    "cacheClusterEnabled": false,
    "cacheClusterSize": "0.5",
    "cacheClusterStatus": "NOT_AVAILABLE",
    "methodSettings": {
        "*/*": {
            "metricsEnabled": true,
            "loggingLevel": "ERROR",
            "dataTraceEnabled": false,
            "throttlingBurstLimit": 5000,
            "throttlingRateLimit": 10000.0,
            "cachingEnabled": false,
            "cacheTtlInSeconds": 300,
            "cacheDataEncrypted": false,
            "requireAuthorizationForCacheControl": true,
            "unauthorizedCacheControlHeaderStrategy": "SUCCEED_WITH_RESPONSE_HEADER"
        }
    },
    "tracingEnabled": true,
    "webAclArn": "arn:aws:wafv2:eu-central-1:472802826312:regional/webacl/xxxx-service-WAF/5474b8ef-a925-4a43-bf2a-499162cb4c8d",
    "tags": {
        "Name": "test",
        "XXXXX": "XXXXX",
        "XXXXX": "XXXXX"
    },
    "createdDate": "2022-02-01T09:24:23+01:00",
    "lastUpdatedDate": "2022-02-01T09:24:26+01:00"
}

Expected Behavior

Deployment should not fail with latest version

Actual Behavior

When deployt with 3.74.0, the deployment fails:

module.xxxx-service.aws_lambda_alias.main: Modifications complete after 0s [id=arn:aws:lambda:eu-central-1:472802826312:function:xxxxx-lambda-xxxxx-service:main]
╷
│ Error: error waiting for API Gateway Stage (ags-97zgip22qh-test) to be available: unexpected state 'NOT_AVAILABLE', wanted target 'AVAILABLE'. last error: %!s(<nil>)
│
│ with module.xxxx-service.aws_api_gateway_stage.service_api_stage,
│ on ..\..\modules\xxxxx-service\_.api-gateway.tf line 84, in resource "aws_api_gateway_stage" "service_api_stage":
│ 84: resource "aws_api_gateway_stage" "service_api_stage" {
│
╵

Version 3.73.0 still works.

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[meichhorn-conet]

The cache_cluster_size = 0.5 is mentioned in #17661 (comment)

[stephen-harris]

+1 Was just about to report this. The workaround in #17661 does avoid this error, but I think perhaps waitForCache should only be true if it the desired state of the cache it that it will be enabled.

[meichhorn-conet]

The commit which introduces this bug

[chandusb]

Any update on this issue? we are facing with the same error.

Error: error waiting for API Gateway Stage (xxx-xxx-dev) to be updated: unexpected state 'NOT_AVAILABLE', wanted target 'AVAILABLE, DELETE_IN_PROGRESS'. last error: %!s()

Terrform version: Terraform v0.13.7

[fatbasstard]

Now hitting the same error. What worked for us was manually enabling caching again and then run TF. After this caching was disabled again and TF runs ago.

Nasty workaround but for now this works

[ryancausey]

Another possible workaround is to specify that version 3.74.0 should not be used in the configuration:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      # There is an issue in 3.74.0 that prevents us from deploying AWS API Gateway
      # stages. https://github.com/hashicorp/terraform-provider-aws/issues/22866
      version = "~> 3.0, != 3.74.0"
    }
  }
}

[indrabasak]

@ryancausey It worked. Thank you

[alpacamybags118]

Been looking into this and found the issue. the logic for create will set waitForCache to true if you provide a cache_cluster_size, regardless of whether cache_cluster_enabled is true or not, so it is possible to be put into a scenario where you set a cluster size without enabling clustering, thus the cluster is never available.

Since we only need to wait for the cluster on creation if its actually enabled, going to modify the creation logic to remove the cache_cluster_size check. I think in addition, we only want to wait for the cluster on modify if a) you are enabling cache_cluster_create or b) cache_cluster_create is already enabled and you are providing or changing cache_cluster_size. if you just adjust cache_cluster_size without cache_cluster_create, it should not wait.

Will have a PR out soon with changes

[isaacngym]

@alpacamybags118 I was just looking at this a couple of days ago and was testing some fixes and came to a similar fix. However I didn't remove lines 147-150 of internal/service/apigateway/stage.go - could I ask why that's necessary there? Checking if cache is enabled before setting waitForCache = true was sufficient to fix the issue from what I could see.

[alpacamybags118]

@isaacngym good question. here's my thought process, but i can change it if it makes sense

when we are initially creating, we only care (at the moment) whether a cluster is being enabled or not to determine if we need to wait. if clustering isnt enabled on creation and a cluster size value is, it doesnt matter since we dont have the cluster enabled so the check doesn't really serve a purpose. that could always change if the underlying AWS apis change.

We do need to to check those cases on update, however, since you could either enable the clustering, or change the size of an already enabled cluster.

[wfgamal]

that solve it for me for now:

required_providers {
aws = {
source = "hashicorp/aws"
# There is an issue in 3.74.0 that prevents us from deploying AWS API Gateway
# stages. #22866
version = "~> 3.0, != 3.74.0, != 3.74.1, != 3.74.2"
}

[ricardo-f]

Folks let's try to keep simple

• Lock the version at 3.73.0 instead of add a lot of exclusions
• Follow-up this issue so when it gets resolved you will be notified 🙂

  required_providers {
    aws = {
      source = "hashicorp/aws"
      # There is an issue in 3.74.0 that prevents us from deploying AWS API Gateway
      # stages. https://github.com/hashicorp/terraform-provider-aws/issues/22866
      version = "~> 3.73.0"
    }
  }

[fatbasstard]

This issue is sort of blocking us to move to the V4 provider, any sights on when this will be fixed (and if the fix also applies to V4?)

[ruizcamilo]

Is this issue really closed? I've been trying to apply the solution all day long but it doesn't seem to be working.

Error: error waiting for API Gateway Stage (ags-yko01d3u3a-dev) to be updated: unexpected state 'NOT_AVAILABLE', wanted target 'AVAILABLE, DELETE_IN_PROGRESS'. last error: %!s()

[mauro-justgiving]

Is this issue really closed? I've been trying to apply the solution all day long but it doesn't seem to be working.

Error: error waiting for API Gateway Stage (ags-yko01d3u3a-dev) to be updated: unexpected state 'NOT_AVAILABLE', wanted target 'AVAILABLE, DELETE_IN_PROGRESS'. last error: %!s()

We need to wait for version 4.6.0 to go out.

[github-actions]

This functionality has been released in v4.6.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.