aws_elasticsearch_domain failed with ValidationException: Authentication error status code: 400

[mildred]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

• terraform version: 0.11.10
• aws provider version: 1.43.2

Affected Resource(s)

• aws_elasticsearch_domain

Terraform Configuration Files

resource "aws_elasticsearch_domain" "logs" {
  domain_name           = "logs"
  elasticsearch_version = "5.6"

  cluster_config {
    instance_type = "t2.medium.elasticsearch"
  }

  ebs_options {
    ebs_enabled = true
    volume_type = "gp2"
    volume_size = "10"
  }

  advanced_options {
    "rest.action.multi.allow_explicit_index" = "true"
  }

  vpc_options {
    subnet_ids         = [...]
    security_group_ids = [...]
  }

  access_policies = "${data.aws_iam_policy_document.es-logs-policy.json}"

  tags {
    Domain = "logs"
  }
}

Debug Output

This is a one time issue and we don't have debug outputs

Error Output

aws_elasticsearch_domain.logs: Creating...
  access_policies:                                         "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"es:ESHttpPut\",\n        \"es:ESHttpPost\",\n        \"es:ESHttpHead\",\n        \"es:ESHttpGet\",\n        \"es:ESHttpDelete\"\n      ],\n      \"Resource\": \"arn:aws:es:eu-west-1:609892909616:domain/logs/*\",\n      \"Principal\": {\n        \"AWS\": \"*\"\n      }\n    },\n    {\n      \"Sid\": \"\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"es:*\",\n      \"Resource\": \"arn:aws:es:eu-west-1:609892909616:domain/logs/*\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::609892909616:user/terraform\"\n      }\n    }\n  ]\n}"
  advanced_options.%:                                      "" => "1"
  advanced_options.rest.action.multi.allow_explicit_index: "" => "true"
  arn:                                                     "" => "<computed>"
  cluster_config.#:                                        "" => "1"
  cluster_config.0.dedicated_master_enabled:               "" => "false"
  cluster_config.0.instance_count:                         "" => "1"
  cluster_config.0.instance_type:                          "" => "t2.medium.elasticsearch"
  domain_id:                                               "" => "<computed>"
  domain_name:                                             "" => "logs"
  ebs_options.#:                                           "" => "1"
  ebs_options.0.ebs_enabled:                               "" => "true"
  ebs_options.0.volume_size:                               "" => "10"
  ebs_options.0.volume_type:                               "" => "gp2"
  elasticsearch_version:                                   "" => "5.6"
  encrypt_at_rest.#:                                       "" => "<computed>"
  endpoint:                                                "" => "<computed>"
  kibana_endpoint:                                         "" => "<computed>"
  node_to_node_encryption.#:                               "" => "<computed>"
  tags.%:                                                  "" => "1"
  tags.Domain:                                             "" => "logs"
  vpc_options.#:                                           "" => "1"
  vpc_options.0.availability_zones.#:                      "" => "<computed>"
  vpc_options.0.security_group_ids.#:                      "" => "1"
  vpc_options.0.security_group_ids.4135710086:             "" => "sg-0d0b5d18902b695de"
  vpc_options.0.subnet_ids.#:                              "" => "1"
  vpc_options.0.subnet_ids.870623161:                      "" => "subnet-0f5978a9a8e37b888"
  vpc_options.0.vpc_id:                                    "" => "<computed>"
...
Releasing state lock. This may take a few moments...

Error: Error applying plan:

1 error(s) occurred:

* aws_elasticsearch_domain.logs: 1 error(s) occurred:

* aws_elasticsearch_domain.logs: ValidationException: Authentication error
	status code: 400, request id: 423f50f6-3960-11e9-85cb-8f4ed15bd4e2

Expected Behavior

No error

Actual Behavior

* aws_elasticsearch_domain.logs: ValidationException: Authentication error
	status code: 400, request id: 423f50f6-3960-11e9-85cb-8f4ed15bd4e2

Steps to Reproduce

random

Important Factoids

Usually works well

References

[obourdon]

Note that on July 6th and July 17th our CI had 2 instances of a similar error but with a different error:

	* aws_elasticsearch_domain.logs: ValidationException: Unauthorized Operation: Elasticsearch must be authorised to describeVpcs

[RohanKurane]

Has this error been resolved ? I hit the same error today. Anything I can do to get more data ?

[obourdon]

@RohanKurane we are hitting this issue on a regular but not predictable manner. I am currently trying a test in our environment and will submit it later this week if tests are successful.

[obourdon]

One new case occurred today:

* aws_elasticsearch_domain.logs: Error creating ElasticSearch domain: ValidationException: Before you can proceed, you must enable a service-linked role to give Amazon ES permissions to access your VPC.

[obourdon]

New case occurred overnight:

* aws_elasticsearch_domain.logs: Error creating ElasticSearch domain: ValidationException: Unauthorized Operation: Elasticsearch must be authorised to describeSubnets

[UrosCvijan]

I also got this one today severeal times, but i didnt get any reason why. I only got aws_elasticsearch_domain.es: Error creating ElasticSearch domain: ValidationException:
So basically the problem was that I had set 2 instances, 2 availability zones, but only 1 subnet. But there was no reason why this ValidationException was happening, lost more than an 1 hour to find the error. I think it has something with this new version 0.12.19, but could not confirm as it did not let me revert to 0.12.18 even though i deleted all states and everything.

[panilo]

I experienced the same issue today, after creating an ES cluster from the console I was able to run my TF script with no issue...

Here is the code I use to create the cluster

resource "aws_elasticsearch_domain" "es" {
  domain_name           = var.domain_name
  elasticsearch_version = "7.1"

  node_to_node_encryption {
    enabled = true
  }

  # encrypt_at_rest {
  #   enabled = true
  # }

  cluster_config {
    instance_type = var.cluster_instance_type
    # dedicated_master_count   = 3
    # dedicated_master_enabled = true
    # dedicated_master_type    = var.cluster_instance_type
    # instance_count           = "4"
    instance_count         = "2"
    zone_awareness_enabled = true
  }
  ebs_options {
    ebs_enabled = true

    # volume_type = "io1"
    volume_type = "gp2"
    volume_size = 10

    # iops        = 300
  }
  vpc_options {
    subnet_ids         = list(data.aws_subnet.private_a.id, data.aws_subnet.private_b.id)
    security_group_ids = list(aws_security_group.default.id)
  }
  access_policies = <<CONFIG
  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": [
            "*"
          ]
        },
        "Action": [
          "es:*"
        ],
        "Resource": "arn:aws:es:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:domain/${var.domain_name}/*"
      }
    ]
  }
  CONFIG
  snapshot_options {
    automated_snapshot_start_hour = var.cluster_automated_snapshot_start_hour
  }
  tags = {
    Domain = var.domain_name
  }
}

[obourdon]

@UrosCvijan on my side as I am using Terraform 0.11.14 I do not think it is related to terraform version but the AWS provider

@panilo yesterday, I migrated to 2.45.0 and it seems like the error is becoming more frequent and less "specific" as I also only get the same "truncated" message @UrosCvijan is describing above

Please also note that some weeks ago, I had written a reduced test scenario looping over creation and deletion of my ES log domain but strangely enough it never failed

[obourdon]

@UrosCvijan @panilo after looking more closely at the AWS provider code history and doing more debug on this, seems like the new empty message returned with error code Validation Exception is indeed coming from AWS API and not from a change in AWS provider code

[obourdon]

I have been successfully testing a patch. You can find the corresponding code here

[obourdon]

I've just submitted PR #11663 for that matter. Acceptance tests successfully passed in my working zone

[bflad]

In Terraform AWS Provider version 2.45.0, an upstream change in the AWS Go SDK introduced a regression where the error messaging of certain error types is no longer returned by the SDK. Created the following provider-wide tracking issue (#11682) and AWS Go SDK issue (aws/aws-sdk-go#3088) for those missing error messages.

[bflad]

Additional error messages for retry on aws_elasticsearch_domain resource creation has been merged and will release with version 2.47.0 of the Terraform AWS Provider, Thursday this week. Thanks to @obourdon for the implementation. 👍

If there are still issues on creation after the version 2.47.0 release, e.g. where retrying logic is appropriate but not working as expected, please file a new GitHub issue and we'll take a fresh look.

[obourdon]

@bflad many thanks for integrating this

[ghost]

This has been released in version 2.47.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!