[BUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF

[estebane-frb]

What is the bug?

A clear and concise description of the bug.
Getting this after upgrading to 2.1.0 and running terraform apply:

provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF

How can one reproduce the bug?

Steps to reproduce the behavior.
Here is my provider config:

provider "opensearch" {
  url         = var.opensearch_url
  username    = var.os_master_user_name
  password    = var.os_master_user_password
  healthcheck = false
}

Opensearch URL is an AWS Managed Opensearch cluster with https enabled.

What is the expected behavior?

A clear and concise description of what you expected to happen.
Apply runs without EOF errors.

What is your host/environment?

Operating system, version.
Amazon Linux, latest version

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

My module here is trying to add an ILM policy. Saw some things:

2023-11-13T19:13:54.075Z [WARN]  Provider "registry.terraform.io/opensearch-project/opensearch" produced an invalid plan for opensearch_ism_policy.default, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .body: planned value cty.StringVal("{\"policy\":{\"default_state\":\"hot\",\"description\":\"Move indexes between storage tiers\",\"ism_template\":[{\"index_patterns\":[\"*\"],\"priority\":100}],\"states\":[{\"actions\":[],\"name\":\"hot\",\"transitions\":[{\"conditions\":{\"min_index_age\":\"24h\"},\"state_name\":\"snapshot\"}]},{\"actions\":[{\"retry\":{\"backoff\":\"exponential\",\"count\":5,\"delay\":\"30m\"},\"snapshot\":{\"repository\":\"snapshot-repo\",\"snapshot\":\"ism-snapshot\"}}],\"name\":\"snapshot\",\"transitions\":[{\"conditions\":{\"min_index_age\":\"2d\"},\"state_name\":\"warm\"}]},{\"actions\":[{\"retry\":{\"backoff\":\"exponential\",\"count\":5,\"delay\":\"1h\"},\"warm_migration\":{}}],\"name\":\"warm\",\"transitions\":[{\"conditions\":{\"min_index_age\":\"30d\"},\"state_name\":\"cold\"}]},{\"actions\":[{\"cold_migration\":{\"end_time\":null,\"ignore\":\"none\",\"start_time\":null,\"timestamp_field\":\"@timestamp\"},\"retry\":{\"backoff\":\"exponential\",\"count\":5,\"delay\":\"1h\"}}],\"name\":\"cold\",\"transitions\":[{\"conditions\":{\"min_index_age\":\"60d\"},\"state_name\":\"delete\"}]},{\"actions\":[{\"cold_delete\":{}}],\"name\":\"delete\",\"transitions\":[]}]}}") does not match config value cty.StringVal("{\n  \"policy\": {\n    \"description\": \"Move indexes between storage tiers\",\n    \"default_state\": \"hot\",\n    \"states\": [\n      {\n        \"name\": \"hot\",\n        \"actions\": [],\n        \"transitions\": [\n          {\n            \"state_name\": \"snapshot\",\n            \"conditions\": {\n              \"min_index_age\": \"24h\"\n            }\n          }\n        ]\n      },\n      {\n        \"name\": \"snapshot\",\n        \"actions\": [\n          {\n            \"retry\": {\n              \"count\": 5,\n              \"backoff\": \"exponential\",\n              \"delay\": \"30m\"\n            },\n            \"snapshot\": {\n              \"repository\": \"snapshot-repo\",\n              \"snapshot\": \"ism-snapshot\"\n            }\n          }\n        ],\n        \"transitions\": [\n          {\n            \"state_name\": \"warm\",\n            \"conditions\": {\n              \"min_index_age\": \"2d\"\n            }\n          }\n        ]\n      },\n      {\n        \"name\": \"warm\",\n        \"actions\": [\n          {\n            \"retry\": {\n              \"count\": 5,\n              \"backoff\": \"exponential\",\n              \"delay\": \"1h\"\n            },\n            \"warm_migration\": {}\n          }\n        ],\n        \"transitions\": [\n          {\n            \"state_name\": \"cold\",\n            \"conditions\": {\n              \"min_index_age\": \"30d\"\n            }\n          }\n        ]\n      },\n      {\n        \"name\": \"cold\",\n        \"actions\": [\n          {\n            \"retry\": {\n              \"count\": 5,\n              \"backoff\": \"exponential\",\n              \"delay\": \"1h\"\n            },\n            \"cold_migration\": {\n              \"start_time\": null,\n              \"end_time\": null,\n              \"timestamp_field\": \"@timestamp\",\n              \"ignore\": \"none\"\n            }\n          }\n        ],\n        \"transitions\": [\n          {\n            \"state_name\": \"delete\",\n            \"conditions\": {\n              \"min_index_age\": \"60d\"\n            }\n          }\n        ]\n      },\n      {\n        \"name\": \"delete\",\n        \"actions\": [\n          {\n            \"cold_delete\": {}\n          }\n        ],\n        \"transitions\": []\n      }\n    ],\n    \"ism_template\": [\n      {\n        \"index_patterns\": [\"*\"],\n        \"priority\": 100\n      }\n    ]\n  }\n}\n")
2023-11-13T19:13:54.079Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"

Do you have any additional context?

Add any other context about the problem.
Will add more if I find any

[estebane-frb]

Something else to add here is it is only happening on Terraform apply. When I run plan everything seems okay.

[estebane-frb]

Another update: turns out for backend_roles parameter in opensearch_roles_mapping resource I was passing a list within a list. For example:

resource "opensearch_roles_mapping" "readall_and_monitor" {
  role_name     = "readall_and_monitor"
  description   = "Maps SAML read-only group to readall_and_monitor "
  backend_roles = [[my_backend_role]] # -> incorrect here
}

So I think the bug here is better error messaging validating the input to this variable "backend_roles" be a list of strings and not objects (list of lists in this case)

[estebane-frb]

Hi there, wanted to check in to see if this was reviewed. Unfortunately EOF seems to be commonplace not just for this but for a handful of issues that are hard to debug. I updated one role_mapping to a different value this time for users and EOF has popped back up.

[estebane-frb]

Hi there - another update. Ran today into another EOF file but turns out it was due to an incorrect password in the provider config. So again - EOF seems to be commonplace for multiple types of errors. It would be really helpful if the error handling printed the actual error instaed of hard-to-debug EOF.

[jordarlu]

Hello, @prudhvigodithi ,would you have a look on this issue, and any comments perhaps ? thanks