Elasticsearch/OpenSearch service has renamed instance types

[marek-obuchowicz]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

AWS provider 3.75.1

Affected Resource(s)

• aws_elasticsearch_domain

Terraform Configuration Files

resource "aws_elasticsearch_domain" "es" {
  domain_name           = "sample-elasticsearch"
  elasticsearch_version = "5.5"

  advanced_options = {
    "rest.action.multi.allow_explicit_index" = "true"
    "indices.fielddata.cache.size"           = "30"
  }

  cluster_config {
    instance_type          = var.elasticsearch_instance_type
    instance_count         = 2
    zone_awareness_enabled = true
  }

  vpc_options {
    security_group_ids = [aws_security_group.es.id]
    subnet_ids = slice(
      var.vpc_subnets_internal,
      0,
      var.elasticsearch_instance_count == 1 ? 1 : 2,
    )
  }

  ebs_options {
    ebs_enabled = var.elasticsearch_ebs_enabled
    volume_type = var.elasticsearch_ebs_volume_type
    volume_size = var.elasticsearch_ebs_volume_size
  }

  snapshot_options {
    automated_snapshot_start_hour = 2
  }
}

Expected Behavior

Terraform should allow current instance types for elasticsearch (opensearch) service. They are called now like t3.small.search - instead of t3.small.elasticsearch

Actual Behavior

Terraform allows to specify currently supported instance types

Steps to Reproduce

While providing old instance types (ie. t3.small.elasticsearch) - AWS API rejects that:

Error: InvalidTypeException: Invalid instance type: t3.small.elasticsearch

  on .terraform/modules/xxx/elasticsearch.tf line 6, in resource "aws_elasticsearch_domain" "es":
   6: resource "aws_elasticsearch_domain" "es" {

While trying to use correct, new names, t3.small.search - terraform stops with validation error:

module.xxx.aws_elasticsearch_domain.es: Modifying... [id=arn:aws:es:eu-central-1:xxxx:domain/stag-elasticsearch]

Error: ValidationException: 1 validation error detected: Value 't3.small.search' at 'elasticsearchClusterConfig.instanceType' failed to satisfy constraint: Member must satisfy enum value set: [r6gd.12xlarge.elasticsearch, i3.2xlarge.elasticsearch, ultrawarm1.xlarge.elasticsearch, m5.4xlarge.elasticsearch, t3.xlarge.elasticsearch, m6g.xlarge.elasticsearch, i3.4xlarge.elasticsearch, m3.large.elasticsearch, m6g.12xlarge.elasticsearch, r4.16xlarge.elasticsearch, t2.micro.elasticsearch, m4.large.elasticsearch, r6gd.16xlarge.elasticsearch, d2.2xlarge.elasticsearch, t3.micro.elasticsearch, m5.large.elasticsearch, i3.8xlarge.elasticsearch, i3.large.elasticsearch, d2.4xlarge.elasticsearch, t2.small.elasticsearch, c4.2xlarge.elasticsearch, t3.small.elasticsearch, c5.2xlarge.elasticsearch, c4.4xlarge.elasticsearch, c6g.2xlarge.elasticsearch, d2.8xlarge.elasticsearch, c5.4xlarge.elasticsearch, t4g.medium.elasticsearch, c6g.4xlarge.elasticsearch, c6g.xlarge.elasticsearch, m3.medium.elasticsearch, c6g.12xlarge.elasticsearch, c4.8xlarge.elasticsearch, c4.large.elasticsearch, c5.xlarge.elasticsearch, c5.large.elasticsearch, t4g.small.elasticsearch, c4.xlarge.elasticsearch, c5.9xlarge.elasticsearch, c6g.8xlarge.elasticsearch, c6g.large.elasticsearch, d2.xlarge.elasticsearch, ultrawarm1.medium.elasticsearch, t3.nano.elasticsearch, t3.medium.elasticsearch, m6g.2xlarge.elasticsearch, t2.medium.elasticsearch, t3.2xlarge.elasticsearch, c5.18xlarge.elasticsearch, m6g.4xlarge.elasticsearch, i3.xlarge.elasticsearch, r6gd.2xlarge.elasticsearch, i2.xlarge.elasticsearch, r3.2xlarge.elasticsearch, r4.2xlarge.elasticsearch, m5.xlarge.elasticsearch, m4.10xlarge.elasticsearch, r6gd.4xlarge.elasticsearch, r6g.2xlarge.elasticsearch, r3.4xlarge.elasticsearch, r5.2xlarge.elasticsearch, m5.12xlarge.elasticsearch, m4.xlarge.elasticsearch, m6g.8xlarge.elasticsearch, m6g.large.elasticsearch, r4.4xlarge.elasticsearch, m5.24xlarge.elasticsearch, m3.xlarge.elasticsearch, r6g.4xlarge.elasticsearch, i3.16xlarge.elasticsearch, t3.large.elasticsearch, r5.4xlarge.elasticsearch, ultrawarm1.large.elasticsearch, m3.2xlarge.elasticsearch, r6gd.8xlarge.elasticsearch, r6gd.large.elasticsearch, r6g.xlarge.elasticsearch, r3.8xlarge.elasticsearch, r3.large.elasticsearch, r5.xlarge.elasticsearch, m4.2xlarge.elasticsearch, r6g.12xlarge.elasticsearch, r4.8xlarge.elasticsearch, r4.xlarge.elasticsearch, r4.large.elasticsearch, r5.12xlarge.elasticsearch, m5.2xlarge.elasticsearch, r6gd.xlarge.elasticsearch, r6g.8xlarge.elasticsearch, r6g.large.elasticsearch, i2.2xlarge.elasticsearch, r3.xlarge.elasticsearch, r5.24xlarge.elasticsearch, r5.large.elasticsearch, m4.4xlarge.elasticsearch]

References

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-resources.html

[mattburgess]

Hi @marek-obuchowicz . The behaviour you're seeing is because you've specified a non-OpenSearch version in elasticsearch_version but an OpenSearch-specific instance_type. You'll want to specify either 1.1 or 1.0 in order to get an OpenSearch domain otherwise you'll end up with a legacy ElasticSearch domain (see https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html#choosing-version). Unfortunately, you'll probably want to wait until #23902 gets merged and released to get first-class OpenSearch support. Until then, you may be better off specifying the latest available ElasticSearch version (with a supported instance type).

I'll take a look to see if the API allows for an easy check to see if there is a way of detecting mismatched version -> instance types but in the meantime using a supported elasticsearch_version+instance_type combination should get you on your way.

[mattburgess]

Just noting this here for posterity; it looks like plan-time validation isn't possible here (I guessed as much given that it's not been implemented yet).

There's ListElasticsearchInstanceTypes but that requires passing in the value of elasticsearch_version. But ValidateFunc is only designed to do relatively simple (self-contained) validation and as such isn't able to see any other values in the resource data, at least as far as I can tell.

[marek-obuchowicz]

Ok, perfect, let's wait for "first class" support in issue you mentioned and close this one.

FYI, Using latest release is sadly not an option for us. Elasticsearch / OpenSearch major versions 5.x, 6.x, 7.x etc. are not compatible and sadly some of our projects rely on legacy majors. An upgrade from 5 to 6 to 7 requires a lot of development work as internal schema logic changed a lot in ES, so i'm not expecting to have it within "days" or "weeks", rather "months" or even "years" time.
I'm not sure if there wasn't a AWS API migration done, as our 5.x / 6.x clusters in AWS console show instance type: t3.small.search (among others) - they were provisioned with Terraform using t3.small.elasticsearch before.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.