Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_storagegateway_gateway with gateway_type FILE_S3 400 error #19111

Closed
varunrsystems opened this issue Apr 26, 2021 · 15 comments · Fixed by #19116
Closed

aws_storagegateway_gateway with gateway_type FILE_S3 400 error #19111

varunrsystems opened this issue Apr 26, 2021 · 15 comments · Fixed by #19116
Labels
bug Addresses a defect in current functionality. service/ec2 Issues and PRs that pertain to the ec2 service. service/storagegateway Issues and PRs that pertain to the storagegateway service.
Milestone

Comments

@varunrsystems
Copy link

varunrsystems commented Apr 26, 2021

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.37.0

Affected Resource(s)

  • aws_storagegateway_gateway

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

terraform {
  required_version = "= 0.13.5"
}

provider "aws" {
  region  = "us-east-2"
  version = "~> 3.0"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.63.0"

  name = "test-vpc"
  cidr = "10.0.0.0/16"

  azs              = ["us-east-2a", "us-east-2b"]
  private_subnets  = ["10.0.0.0/24"]
  public_subnets   = ["10.0.10.0/24"]

  enable_nat_gateway = true
  enable_vpn_gateway = false
  single_nat_gateway = true

  enable_dns_hostnames               = true
  enable_dns_support                 = true
  enable_dhcp_options = false
}

resource "aws_storagegateway_gateway" "nfs_file_gateway" {
  gateway_ip_address = aws_instance.gateway.public_ip
  gateway_name       = "test"
  gateway_timezone   = "GMT"
  gateway_type       = "FILE_S3"
}

resource "aws_instance" "gateway" {
  ami           = "ami-08fff025e75894630"
  instance_type = "m5.xlarge"

  # Refer to AWS File Gateway documentation for minimum system requirements.
  ebs_optimized = true
  subnet_id     = module.vpc.public_subnets.0

  ebs_block_device {
    device_name           = "/dev/xvdf"
    volume_size           = "150"
    volume_type           = "gp3"
    delete_on_termination = true
  }

  vpc_security_group_ids = [
    aws_security_group.storage_gateway.id,
  ]
}

resource "aws_security_group" "storage_gateway" {
  name        = "test-security-group"
  description = "Allow inbound NFS traffic"
  vpc_id      = module.vpc.vpc_id
}

resource "aws_security_group_rule" "egress_all" {
  description       = "egress"
  from_port         = 0
  protocol          = "ALL"
  security_group_id = aws_security_group.storage_gateway.id
  to_port           = 0
  type              = "egress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "ingress_139_tcp_product" {
  description              = "For SMB"
  from_port                = 0
  protocol                 = "all"
  security_group_id        = aws_security_group.storage_gateway.id
  to_port                  = 0
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

Debug Output

https://gist.github.com/varunrsystems/07076438b639b723c5c70f363d83991a

Panic Output

Expected Behavior

Terraform apply/plan successfully executed

Actual Behavior

Break with error

Error: error reading Storage Gateway Bandwidth rate limit: InvalidGatewayRequestException: This operation is not valid for the specified gateway.
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "ef88f6ad-a94a-469b-9e81-3e8bfa93830c"
  },
  Error_: {
    ErrorCode: "UnsupportedOperationForGatewayType"
  },
  Message_: "This operation is not valid for the specified gateway."
}

Steps to Reproduce

  1. terraform apply

Important Factoids

References

  • #0000
@ghost ghost added service/ec2 Issues and PRs that pertain to the ec2 service. service/storagegateway Issues and PRs that pertain to the storagegateway service. labels Apr 26, 2021
@github-actions github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Apr 26, 2021
@gdavison
Copy link
Contributor

Thanks for reporting this, @varunrsystems. I've done a bit of research on this, and from what I can tell, you're deploying the storage gateway in us-east-2.

We check for this error at

if err != nil && !isAWSErr(err, storagegateway.ErrCodeInvalidGatewayRequestException, "The specified operation is not supported") {
return fmt.Errorf("error reading Storage Gateway Bandwidth rate limit: %s", err)
}
. For some reason, in us-east-2, the AWS API returns a slightly different message than in other regions. Everywhere else, the API returns a message matching The specified operation is not supported, while in us-east-2 the API returns This operation is not valid for the specified gateway

@gdavison gdavison removed the needs-triage Waiting for first response or review from a maintainer. label Apr 26, 2021
@adamjohnson01
Copy link

adamjohnson01 commented Apr 27, 2021

We are also seeing this in eu-west-1 using 3.37.0

@stekern
Copy link

stekern commented Apr 27, 2021

I'm experiencing the same issue in eu-west-1 using AWS provider 3.36.0.

Error: error reading Storage Gateway Bandwidth rate limit: InvalidGatewayRequestException: This operation is not valid for the specified gateway.

@ewbankkit ewbankkit added the bug Addresses a defect in current functionality. label Apr 27, 2021
@varunrsystems
Copy link
Author

I contacted the AWS support for this change in API response, they forwarded to SGW Service Team to investigate the issue and return with their findings, will update further reply here.

@github-actions github-actions bot added this to the v3.38.0 milestone Apr 27, 2021
@nerryc
Copy link

nerryc commented Apr 29, 2021

@gdavison @varunrsystems We are also seeing this in EU-WEST-1 but not in AF-SOUTH-1. Started around 26/27 April.

@sa1white
Copy link

sa1white commented Apr 29, 2021

@gdavison @varunrsystems - we saw this error in US-east-1 and eu-central-1 on April 28.

I would guess AWS is rolling out the same change to all regions, one by one.

I'm using provider v3.37.0

@sa1white
Copy link

When will v3.38.0 be released? This error is blocking some production updates we need to deploy via terraform.

@dayglojesus
Copy link

This is a me too. Stack is in us-west-2 and it was fine last week, but this week it's been balking anytime we plan it ...

Error: error reading Storage Gateway Bandwidth rate limit: InvalidGatewayRequestException: This operation is not valid for the specified gateway.
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "dd514340-7555-4039-b046-e597a4bbaa41"
  },
  Error_: {
    ErrorCode: "UnsupportedOperationForGatewayType"
  },
  Message_: "This operation is not valid for the specified gateway."
}

@nerryc
Copy link

nerryc commented Apr 29, 2021

We have also tried many different providers, ended up having to remove the storage gateway from state and add the arn values called statically just to continue our runs.. Logged a call with HashiCorp about this as well.

@rnebular
Copy link

We are experiencing this as well in the us-west-2 region. Plan/Apply worked two weeks ago on existing infrastructure. Now existing plan/apply failing as well as new undeployed gateways also showing this error during plan/apply. I would copy/paste our error but it's identical to others that have already posted. This is 100% blocking, and we've tried to hard-code to an older provider but have found no relief.

Waiting for a fix or workaround...

@varunrsystems
Copy link
Author

If anyone having AWS enterprise support then they can also raise this issue as production block on AWS, I already raised but not get response from last 2 days.

@nerryc
Copy link

nerryc commented Apr 29, 2021

We raised it and was basically told to fix the api call being made... so in this scenario terraform needs to adjust.

Pretty poor show from AWS not advertising these changes.

This is also not currently happening in all regions, adding to the confusion and frustration.

@ghost
Copy link

ghost commented Apr 30, 2021

This has been released in version 3.38.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@varunrsystems
Copy link
Author

Update from AWS and I guess they are right about how to handle the exception.

Thank you for bringing this issue to our attention. Please note that the DescribeBandwidthRateLimit API is not supported on the Amazon S3 File Gateway. Therefore, calling the API with this gateway type is expected to throw an InvalidGatewayRequestException error.

We are aware that some Terraform configuration scripts to create Storage Gateway resources appear to call the DescribeBandwidthRateLimit API for all gateway types. It appears that the exception handling that is intended to handle errors from DescribeBandwidthRateLimit with unsupported gateway types, utilizes both the error and message fields to parse the exception. Since we have updated the associated error message for this API, the script appears to fail as a result.

In response to your message, we have taken steps to update the error message in order to address the issue as quickly as possible. However, please note that we do not recommend utilizing the message field from API errors for exception handling but instead recommend only using the error field in order to avoid similar issues in the future.

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 31, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/ec2 Issues and PRs that pertain to the ec2 service. service/storagegateway Issues and PRs that pertain to the storagegateway service.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants