Error 500 (Internal Server Error) when trying to create more than one user

[clifford-sanders]

When I try create multiple users like this

resource "elasticsearch_opendistro_user" "testuser1" {
  username = "testuser1"
  password = "testuser1"
  description   = "testuser1"

}

resource "elasticsearch_opendistro_user" "testuser2" {
  username = "testuser2"
  password = "testuser2"
  description   = "testuser2"

}

resource "elasticsearch_opendistro_user" "testuser3" {
  username = "testuser3"
  password = "testuser3"
  description   = "testuser3"

}

terraform plan correctly determines the change

Terraform will perform the following actions:

  # module.config.elasticsearch_opendistro_user.testuser1 will be created
  + resource "elasticsearch_opendistro_user" "testuser1" {
      + description = "testuser1"
      + id          = (known after apply)
      + password    = (sensitive value)
      + username    = "testuser1"
    }

  # module.config.elasticsearch_opendistro_user.testuser2 will be created
  + resource "elasticsearch_opendistro_user" "testuser2" {
      + description = "testuser2"
      + id          = (known after apply)
      + password    = (sensitive value)
      + username    = "testuser2"
    }

  # module.config.elasticsearch_opendistro_user.testuser3 will be created
  + resource "elasticsearch_opendistro_user" "testuser3" {
      + description = "testuser3"
      + id          = (known after apply)
      + password    = (sensitive value)
      + username    = "testuser3"
    }

But when I execute terraform apply only on user gets created and I get this error

Error: Error creating user mapping: elastic: Error 500 (Internal Server Error): []: {"backend_roles":[],"attributes":{},"description":"testuser1","password":"testuser1","hash":""}

  on ../modules/config/users.tf line 1, in resource "elasticsearch_opendistro_user" "testuser1":
   1: resource "elasticsearch_opendistro_user" "testuser1" {

Error: Error creating user mapping: elastic: Error 500 (Internal Server Error): []: {"backend_roles":[],"attributes":{},"description":"testuser3","password":"testuser3","hash":""}

  on ../modules/config/users.tf line 15, in resource "elasticsearch_opendistro_user" "testuser3":
  15: resource "elasticsearch_opendistro_user" "testuser3" {

I get the same error when I try to create the users like this

resource "elasticsearch_opendistro_user" "users" {
  for_each = {
    testuser1 = "testuser1"
    testuser2 = "testuser2"
    testuser3 = "testuser3"
  }
  username = each.key
  password = each.value
}

I'm using version 1.5.3 with Terraform v0.12.29 on OFDE 1.12

[phillbaker]

Hi @clifford-sanders thanks for opening an issue. This seems similar to the report here: #129.

I was able to reproduce this in a test here: https://github.com/phillbaker/terraform-provider-elasticsearch/runs/2108205592?check_suite_focus=true#step:12:2457

With the error response being:

{"status":"INTERNAL_SERVER_ERROR","message":"Error [internalusers]: version conflict, required seqNo [31], primary term [1]. current document has seqNo [32] and primary term [1]"}

This seems to be a race condition, adding a function to slow down requests on the provider side works around the error response. I think this is an issue upstream on opendistro. (My guess is that they need to synchronize access to updates using a lock to make the updates sequential.)

[clifford-sanders]

Contrary to #129 adding TF_CLI_ARGS_apply=-parallelism=1 helps in my case.

[ajmac3242]

For what's it's worth I added a depends_on to my second role and it waits for the first to be created. It seems to be a pretty decent work around.

// https://registry.terraform.io/providers/phillbaker/elasticsearch/latest/docs
resource "elasticsearch_opendistro_role" "read_only_role" {
  role_name   = format("%s_read_only", lower(var.environment))
  description = format("Role that will give access to read from %s indicies", lower(var.environment))

  index_permissions {
    index_patterns  = [format("%s_*", lower(var.environment))]
    allowed_actions = ["read", "ultrawarm_index_read"]
  }
}

resource "elasticsearch_opendistro_role" "write_role" {
  depends_on = [
    elasticsearch_opendistro_role.read_only_role
  ]
  role_name   = format("%s_write", lower(var.environment))
  description = format("Role that will give access to write to %s indicies", lower(var.environment))

  index_permissions {
    index_patterns  = [format("%s_*", lower(var.environment))]
    allowed_actions = ["read", "ultrawarm_index_read", "write", "ultrawarm_index_write"]
  }
}

[r0ck40k]

Hey,
sorry to hook up in this issue.

I am currently trying these things too and having multiple errors.
Making dependencies explicitly for keeping an order of how and when resources are created, works pretty well.
But this error happens on so many resources (i.e. users, roles, kibana_objects - in case Elasticsearch is blocked for whatever reason).

Especially when you are trying to use for_each or count for mass-creating users, importing kibana dashboard and roles this starts to be unpredictable because within the for_each we could not make the modules depend on each other and putting them, so sometimes this error occurs - sadly.

Currently we stuck on v7.9 but in my lab we are using 7.10. Currently both of the versions have the same behaviour and i can confirm that it is not directly provider related and a super annoying bug from Elasticsearch.

Maybe it would be an idea to add a default retry mechanism on the provider side which can be configured as this would currently circumvent most of the issues. Since usually the second call will be successfull 90% of the time.

@phillbaker do you have thoughts about that?

[phillbaker]

should be fixed in c81621a

[sidda-dandu]

@phillbaker Looks like Error 500 turned out to be a Error 404 now..

[phillbaker]

Hi @sidda-dandu , can you please include the following:

• terraform version
• provider version
• elasticsearch version (and opendistro version if relevant)
• redacted version of the terraform resource configuration
• terraform logs by setting TF_LOG=info

[sidda-dandu]

@phillbaker here is the info which you have asked for i.e.

• terraform version - 0.13
• provider version - 3.0
• elasticsearch version - 7.10
• Error i.e.
  2021-06-29T13:41:20.686+0530 [DEBUG] plugin.terraform-provider-elasticsearch_v1.5.8.exe: 2021/06/29 13:41:20 [INFO] Pinging url to determine version elastic_search_domain_endpoint i.e. xxx
  2021-06-29T13:41:20.686+0530 [DEBUG] plugin.terraform-provider-elasticsearch_v1.5.8.exe: 2021/06/29 13:41:20 [INFO] Pinging url to determine version elastic_search_domain_endpoint i.e. xxx
  2021-06-29T13:41:21.385+0530 [DEBUG] plugin.terraform-provider-elasticsearch_v1.5.8.exe: 2021/06/29 13:41:21 [INFO] Failed to put role mapping: error creating role mapping: elastic: Error 404 (Not Found): []
  2021-06-29T13:41:21.385+0530 [DEBUG] plugin.terraform-provider-elasticsearch_v1.5.8.exe: 2021/06/29 13:41:21 [INFO] Failed to put role mapping: error creating role mapping: elastic: Error 404 (Not Found): []
  2021-06-29T13:41:21.385+0530 [DEBUG] plugin.terraform-provider-elasticsearch_v1.5.8.exe: 2021/06/29 13:41:21 [INFO] Failed to put role mapping: error creating role mapping: elastic: Error 404 (Not Found): []

[phillbaker]

Hi @sidda-dandu please also include:

• opendistro version
• redacted version of the terraform resource configuration

[sidda-dandu]

@phillbaker please see below

• opendistro version - 1.5.8
• redacted version of the terraform resource configuration
  resource "elasticsearch_opendistro_roles_mapping" "write_opendistro_roles_mapping" {
  count = xxx-variable == true ? 1 : 0
  role_name = "xenv-write"
  description = xxxx
  backend_roles = [ xxx i.e. lambda_role_arn]
  }