aws_ec2_instance_type_offerings does not return the list of Locations matching the Filters

[jnerin]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.14.8

• provider registry.terraform.io/hashicorp/aws v3.34.0

Affected Resource(s)

Data Source: aws_ec2_instance_type_offerings

Terraform Configuration Files

On us-east-1 one of the AZs (use1-az3 - us-east-1e in this particular account) does not support some EC2 instance types, so parts of our code that used to work on other regions with more homogeneous AZs fail in us-east-1 because by sheer luck our 1st subnet id in this list subnet_id = local.private_subnet_ids_list[0] happens to be the one in the AZ that does not support the instance types that we want to use for some single instances. So deployment fails.

data "aws_ec2_instance_type_offerings" "mq-type" {
  filter {
    name   = "instance-type"
    values = ["m5.xlarge"]
  }
  location_type = "availability-zone"
}

data "aws_ec2_instance_type_offerings" "mq-azs" {
  filter {
    name   = "instance-type"
    values = ["m5.xlarge"]
  }
  filter {
    name   = "location"
    values = [
  "us-east-1a",
  "us-east-1b",
  "us-east-1c",
  "us-east-1d",
  "us-east-1e",
  "us-east-1f",
]
  }
  location_type = "availability-zone"
}

Expected Behavior

Either data.aws_ec2_instance_type_offerings.mq-type or data.aws_ec2_instance_type_offerings.mq-azs, hopefully both, should contain a result being the union of both Filters, the List of AZs, and the list of instance types.

The problem I'm trying to solve is to correctly handle an ancient AZ in us-east-1 that doesn't have the "modern" instance types, so, for example for Amazon MQ the error is:

Error: BadRequestException: Availability Zone [us-east-1e] of requested subnet [subnet-XXXX] does not support broker instance type [mq.m5.xlarge]. Specify a subnet in a different Availability Zone.

And I would like to filter all the subnets (ideally) or azs (if nothing else) to fill the parameter subnet_ids with 2 subnets that do support the host_instance_type I selected for MQ. Something like:

data "aws_subnet_ids" "private_subnets" {
  vpc_id = data.aws_vpc.vpc.id
  filter {
    name   = "tag:Name"
    values = ["private-*"]
  }
  filter {
    name   = "availability_zone"
    values = slice(data.aws_ec2_instance_type_offerings.mq-type.locations, 0, 2)
  }
}

resource "aws_mq_broker" "mq-active-standby" {
  broker_name = "mq"

  host_instance_type  = "mq.m5.xlarge"
  deployment_mode     = "ACTIVE_STANDBY_MULTI_AZ"

  subnet_ids          = data.aws_subnet_ids.private_subnets
}

Other place I have trouble with that is directly launching EC2 instances:

Error: Error launching source instance: Unsupported: Your requested instance type (t3a.xlarge) is not supported in your requested Availability Zone (us-east-1e). Please retry your request by not specifying an Availability Zone or choosing us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1f.

But that could be solved with the same approach I guess.

Please see this full TF file with two attempts and the terraform console output after each data source, as well as the AWS CLI output too, note the differing outputs between terraform and AWS CLI (this breaks the Principle of least astonishment):

# data.aws_ec2_instance_type_offerings.mq-azs
data "aws_ec2_instance_type_offerings" "mq-azs" {
  filter {
    name   = "instance-type"
    values = ["m5.xlarge"]
  }

  filter {
    name   = "location"
    values = local.workspace_lists["azs"]
  }

  location_type = "availability-zone"
}

# > data.aws_ec2_instance_type_offerings.mq-azs
# {
#   "filter" = toset([
#     {
#       "name" = "instance-type"
#       "values" = tolist([
#         "mq.m5.xlarge",
#       ])
#     },
#     {
#       "name" = "location"
#       "values" = tolist([
#         "us-east-1a",
#         "us-east-1b",
#         "us-east-1c",
#         "us-east-1d",
#         "us-east-1e",
#         "us-east-1f",
#       ])
#     },
#   ])
#   "id" = "us-east-1"
#   "instance_types" = toset([])
#   "location_type" = "availability-zone"
# }

# $ aws --region us-east-1 ec2 describe-instance-type-offerings --filters "Name=instance-type,Values=m5.xlarge" "Name=location,Values=us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1e,us-east-1f" --location-type availability-zone
# {
#     "InstanceTypeOfferings": [
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1a"
#         }, 
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1c"
#         }, 
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1f"
#         }, 
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1d"
#         }, 
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1b"
#         }
#     ]
# }


# data.aws_ec2_instance_type_offerings.mq-type
data "aws_ec2_instance_type_offerings" "mq-type" {
  filter {
    name   = "instance-type"
    values = ["m5.xlarge"]
  }

  location_type = "availability-zone"
}

# > data.aws_ec2_instance_type_offerings.mq-type
# {
#   "filter" = toset([
#     {
#       "name" = "instance-type"
#       "values" = tolist([
#         "mq.m5.xlarge",
#       ])
#     },
#   ])
#   "id" = "us-east-1"
#   "instance_types" = toset([])
#   "location_type" = "availability-zone"
# }

# $ aws --region us-east-1 ec2 describe-instance-type-offerings --filters "Name=instance-type,Values=m5.xlarge" --location-type availability-zone
# {
#     "InstanceTypeOfferings": [
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1b"
#         }, 
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1a"
#         }, 
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1f"
#         }, 
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1d"
#         }, 
#         {
#             "LocationType": "availability-zone", 
#             "InstanceType": "m5.xlarge", 
#             "Location": "us-east-1c"
#         }
#     ]
# }

Note that I'm still not sure how to "handle" or expose the output of a filter like:
aws --region us-east-1 ec2 describe-instance-type-offerings --filters "Name=instance-type,Values=m5.xlarge,m4.xlarge" "Name=location,Values=us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1e,us-east-1f" --location-type availability-zone

Where the output of AWS CLI is the cartesian product of all filter conditions (2 different instance types in this case) but that match the conditions (still excluding m5.xlarge us-east-1e).

References

• EC2 instance type discovery #10989
• Data source for AWS EC2 Reserved Instances offerings. #9943
• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTypeOfferings.html

[ewbankkit]

@jnerin Thanks for raising this issue.
It has already been noticed in #15272. I'm going to close this one as a duplicate so that we can concentrate discussion in the linked issue.
Please add any additional comments there.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.