plugin crash: panic: interface conversion: interface {} is nil, not map[string]interface {}

[joaocc]

After upgrading to provider v2.0.4, we started getting this error.
This was code that didn't change recently, so not 100% sure if this is transient, was there in 2.0.3 (with something else), or is indeed from 2.0.4 (which works well elsewhere in the same code.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform planned the following actions, but then encountered a problem:

  # module.this_ekshost[0].kubectl_manifest.this["sub"] will be updated in-place
  ~ resource "kubectl_manifest" "this" {
        id                      = "/api/v1/namespaces/flux-system/secrets/subset1"
        name                    = "subset1"
      ~ yaml_incluster          = (sensitive value)
        # (16 unchanged attributes hidden)
    }

  # module.this_ekshost[0].kubectl_manifest.this["full"] will be updated in-place
  ~ resource "kubectl_manifest" "this" {
        id                      = "/api/v1/namespaces/flux-system/secrets/full"
        name                    = "full"
      ~ yaml_incluster          = (sensitive value)
        # (16 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.
╷
│ Error: Request cancelled
│ 
│   with module.probe__configs[0].module.configs["p1"].kubectl_manifest.this["_"],
│   on ../../../secrets.as-kubectl/main.tf line 2, in resource "kubectl_manifest" "this":
│    2: resource "kubectl_manifest" "this" {
│ 
│ The plugin.(*GRPCProvider).ReadResource request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.probe__configs[0].module.configs["p2"].kubectl_manifest.this["_"],
│   on ../../../secrets.as-kubectl/main.tf line 2, in resource "kubectl_manifest" "this":
│    2: resource "kubectl_manifest" "this" {
│ 
│ The plugin.(*GRPCProvider).ReadResource request was cancelled.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.probe__configs[0].module.configs["p9"].kubectl_manifest.this["_"],
│   on ../../../secrets.as-kubectl/main.tf line 2, in resource "kubectl_manifest" "this":
│    2: resource "kubectl_manifest" "this" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
│ details.
╵
╷
│ Error: Request cancelled
│ 
│   with module.cert_web["p"].kubectl_manifest.this_cert[0],
│   on ../../../cert-web/main.tf line 122, in resource "kubectl_manifest" "this_cert":
│  122: resource "kubectl_manifest" "this_cert" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.cert_web["t"].kubectl_manifest.this_cert[0],
│   on ../../../cert-web/main.tf line 122, in resource "kubectl_manifest" "this_cert":
│  122: resource "kubectl_manifest" "this_cert" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secr_akv["ak"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secr__cf["x'"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secr__cf["x1"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secr__cf["x2"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secr_g.module.this["g1"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secr_g.module.this["g2"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secr_g.module.this["g3"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secr_g.module.this["g4"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secrets_aws[0].module.tester_ssm_ps[0].module.ssm["tst-0"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Request cancelled
│ 
│   with module.ext_secrets_aws[0].module.tester_ssm_ps[0].module.ssm["tst-1"].kubectl_manifest.ext_secret[0],
│   on ../../../ext-secret/main.tf line 72, in resource "kubectl_manifest" "ext_secret":
│   72: resource "kubectl_manifest" "ext_secret" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵

Stack trace from the terraform-provider-kubectl_v2.0.4 plugin:

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 232 [running]:
github.com/alekc/terraform-provider-kubectl/kubernetes.getLiveManifestFields_WithIgnoredFields({0x0, 0x0, 0x10?}, 0x40005944f8, 0x400045d0f0)
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:1032 +0x9f8
github.com/alekc/terraform-provider-kubectl/kubernetes.getLiveManifestFingerprint(0x40009c6500?, 0x1a45dc3?, 0x8?)
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:1014 +0xac
github.com/alekc/terraform-provider-kubectl/kubernetes.resourceKubectlManifestReadUsingClient({0x1ef3270, 0x40002bea10}, 0x40009c6500, {0x4000a70bc0?, 0x2eafc20?}, {0x1efd1f8, 0x40000b07d0}, 0x40005944f8)
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:688 +0x3f0
github.com/alekc/terraform-provider-kubectl/kubernetes.resourceKubectlManifestRead({0x1ef3270, 0x40002bea10}, 0x40009c6500, {0x180b9a0?, 0x40000b8000?})
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:659 +0x2ac
github.com/alekc/terraform-provider-kubectl/kubernetes.resourceKubectlManifest.func2({0x1ef3270?, 0x40002bea10?}, 0x0?, {0x180b9a0?, 0x40000b8000?})
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:85 +0x2c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x4000185180, {0x1ef31c8, 0x4000acb3b0}, 0xd?, {0x180b9a0, 0x40000b8000})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:795 +0xe8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0x4000185180, {0x1ef31c8, 0x4000acb3b0}, 0x400011a750, {0x180b9a0, 0x40000b8000})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:1089 +0x430
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0x400081a120, {0x1ef31c8?, 0x4000acb2c0?}, 0x4000454780)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/grpc_provider.go:649 +0x3e4
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0x4000616000, {0x1ef31c8?, 0x4000acaae0?}, 0x4000bf40c0)
        github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/tf5server/server.go:789 +0x390
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x196ab20?, 0x4000616000}, {0x1ef31c8, 0x4000acaae0}, 0x40002be0e0, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:431 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x40004e2000, {0x1efcc80, 0x4000102680}, 0x40003799e0, 0x40004342d0, 0x2e91c88, 0x0)
        google.golang.org/grpc@v1.57.1/server.go:1358 +0xc04
google.golang.org/grpc.(*Server).handleStream(0x40004e2000, {0x1efcc80, 0x4000102680}, 0x40003799e0, 0x0)
        google.golang.org/grpc@v1.57.1/server.go:1735 +0x7c4
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        google.golang.org/grpc@v1.57.1/server.go:970 +0xb4
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 43
        google.golang.org/grpc@v1.57.1/server.go:981 +0x160

Error: The terraform-provider-kubectl_v2.0.4 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[joaocc]

I can confirm this is an issues with v2.0.4 (reverting to v2.0.3 fixes the issue).
Looking at other similar errors, it seems some resources have an empty/null field instead of map, which causes the provider to fail.

[joaocc]

And a bit more digging in stack trace vs code for v2.0.4 and comparing with the context of the state file, we find that there is indeed a "stringData": null in the state file.
This matches with code introduced in v2.0.4 (ca6ec65).

[joaocc]

A workaround for this is to set stringData to coalesce(something.myStringData, {}) in v2.0.3 and then upgrade to v2.0.4.

[alekc]

@joaocc thanks for reporting. Do you have a reproduction code by any chance?

[joaocc]

I can try to find it in the history of state files. However, when diagnosing this, I noticed i was doing the following:

• with provider v2.0.3, creating a manifest for a Secret, and setting stringData = null
• with provider v2.0.4, trying to do a simple plan for the same state file
  I remember seeing that the state file created by v2.0.3 did indeed have a field stringData set to null, which is what led me to v2.0.4 processing of Secrets not compliant with API and breaks expected behaviour #121 (after some back and forth, as I am not go-literate).
  Please let me know if you need me to dig the state file.

[alekc]

By default the stringData is not being set in terraform state file on 2.0.3, so this might be a bit of an edge case (I guess you were constructing yaml manifest via hcl objects?)

I even tried with

terraform {
  required_providers {
    kubectl = {
      source = "alekc/kubectl"
      version = "2.0.3"
    }
  }
}
resource "kubectl_manifest" "example" {
  yaml_body = <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: secret-basic-auth
  namespace: default
type: Opaque
stringData: null
EOF
}

and it gives following state file

{
  "version": 4,
  "terraform_version": "1.7.0",
  "serial": 9,
  "lineage": "2ad0d7a7-fb83-2a7f-c59f-32c050a136d4",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "kubectl_manifest",
      "name": "example",
      "provider": "provider[\"registry.terraform.io/alekc/kubectl\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "api_version": "v1",
            "apply_only": false,
            "field_manager": "kubectl",
            "force_conflicts": false,
            "force_new": false,
            "id": "/api/v1/namespaces/default/secrets/secret-basic-auth",
            "ignore_fields": null,
            "kind": "Secret",
            "live_manifest_incluster": "ea865e929d18452a6405374dec9f72a4e4c78e4c5564c43af32d91248c83bd77",
            "live_uid": "ca87fef6-7c65-444c-8197-4db901e3e664",
            "name": "secret-basic-auth",
            "namespace": "default",
            "override_namespace": null,
            "sensitive_fields": null,
            "server_side_apply": false,
            "timeouts": null,
            "uid": "ca87fef6-7c65-444c-8197-4db901e3e664",
            "validate_schema": true,
            "wait": null,
            "wait_for": [],
            "wait_for_rollout": true,
            "yaml_body": "apiVersion: v1\nkind: Secret\nmetadata:\n  name: secret-basic-auth\n  namespace: default\ntype: Opaque\nstringData: null\n",
            "yaml_body_parsed": "apiVersion: v1\nkind: Secret\nmetadata:\n  name: secret-basic-auth\n  namespace: default\nstringData: (sensitive value)\ntype: Opaque\n",
            "yaml_incluster": "ea865e929d18452a6405374dec9f72a4e4c78e4c5564c43af32d91248c83bd77"
          },
          "sensitive_attributes": [],
          "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
        }
      ]
    }
  ],
  "check_results": null
}

@joaocc struggling to get a test case triggering the issue.

Having said that, I found a potential error but would love to test it with the panic before releasing it (since its a bit of an edge case as I mentioned)

[joaocc]

Yes, I am building the YAML objects via HCL + yamlencode.
Unfortunately I don't think I will be able to get one of the original outputs :S

[alekc]

Ok, I will try to make some further tests later on (maybe jumping from the gavin original provider towards 2.0.4).

[joaocc]

Hi.
I have new repro of this, with 2.0.4 loading some older state file

    {
      "module": "module.mmmmm1[0].module.configs[\"pa-0\"]",
      "mode": "managed",
      "type": "kubectl_manifest",
      "name": "this",
      "provider": "provider[\"registry.terraform.io/alekc/kubectl\"]",
      "instances": [
        {
          "index_key": "_",
          "schema_version": 1,
          "attributes": {
            "api_version": "v1",
            "apply_only": false,
            "field_manager": "kubectl",
            "force_conflicts": false,
            "force_new": false,
            "id": "/api/v1/namespaces/pa/secrets/m1---pa-0",
            "ignore_fields": null,
            "kind": "Secret",
            "live_manifest_incluster": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
            "live_uid": "8286b76e-7d58-55ff-a59b-ea65a1a907cd",
            "name": "m1---pa-0",
            "namespace": "pa",
            "override_namespace": null,
            "sensitive_fields": null,
            "server_side_apply": true,
            "timeouts": null,
            "uid": "8286b76e-7d58-55ff-a59b-ea65a1a907cd",
            "validate_schema": true,
            "wait": true,
            "wait_for": [],
            "wait_for_rollout": true,
            "yaml_body": "\"apiVersion\": \"v1\"\n\"immutable\": false\n\"kind\": \"Secret\"\n\"metadata\":\n  \"annotations\":\n  \"labels\": {}\n  \"name\": \"m1---pa-0\"\n  \"namespace\": \"pa\"\n\"stringData\": null\n\"type\": \"Opaque\"\n",
            "yaml_body_parsed": "apiVersion: v1\nimmutable: false\nkind: Secret\nmetadata:\n  annotations:\  labels: {}\n  name: m1---pa-0\n  namespace: pa\nstringData: (sensitive value)\ntype: Opaque\n",
            "yaml_incluster": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          },
          "sensitive_attributes": [],
          "private": "eyI0NTZlOTE3OS03MGNmLTUxMDYtYjRiMy04NTc5NjdmMjU4N2QiOnsiY3JlYXRlIjo2MDAwMDAwMDAwMDB9LCJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
          "dependencies": [
            ...
          ]
        }
      ]
    },

I think this illustrates the example of null stringData

Stack trace from the terraform-provider-kubectl_v2.0.4 plugin:

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 220 [running]:
github.com/alekc/terraform-provider-kubectl/kubernetes.getLiveManifestFields_WithIgnoredFields({0x0, 0x0, 0x10?}, 0x400013a748, 0x4000f970f0)
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:1032 +0x9f8
github.com/alekc/terraform-provider-kubectl/kubernetes.getLiveManifestFingerprint(0x4000f20800?, 0x1a45dc3?, 0x8?)
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:1014 +0xac
github.com/alekc/terraform-provider-kubectl/kubernetes.resourceKubectlManifestReadUsingClient({0x1ef3270, 0x4000d30620}, 0x4000f20800, {0x4000d2aba0?, 0x2eafc20?}, {0x1efd1f8, 0x400031a370}, 0x400013a748)
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:688 +0x3f0
github.com/alekc/terraform-provider-kubectl/kubernetes.resourceKubectlManifestRead({0x1ef3270, 0x4000d30620}, 0x4000f20800, {0x180b9a0?, 0x4000954000?})
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:659 +0x2ac
github.com/alekc/terraform-provider-kubectl/kubernetes.resourceKubectlManifest.func2({0x1ef3270?, 0x4000d30620?}, 0x0?, {0x180b9a0?, 0x4000954000?})
        github.com/alekc/terraform-provider-kubectl/kubernetes/resource_kubectl_manifest.go:85 +0x2c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x4000185180, {0x1ef31c8, 0x400070cc60}, 0xd?, {0x180b9a0, 0x4000954000})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:795 +0xe8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0x4000185180, {0x1ef31c8, 0x400070cc60}, 0x4000daa680, {0x180b9a0, 0x4000954000})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:1089 +0x430
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0x40006c6138, {0x1ef31c8?, 0x400070cb70?}, 0x4001092300)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/grpc_provider.go:649 +0x3e4
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0x40000c1900, {0x1ef31c8?, 0x400070c390?}, 0x400075c840)
        github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/tf5server/server.go:789 +0x390
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x196ab20?, 0x40000c1900}, {0x1ef31c8, 0x400070c390}, 0x4000d30000, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:431 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x40004f8000, {0x1efcc80, 0x40001031e0}, 0x4000c1e120, 0x4000034ba0, 0x2e91c88, 0x0)
        google.golang.org/grpc@v1.57.1/server.go:1358 +0xc04
google.golang.org/grpc.(*Server).handleStream(0x40004f8000, {0x1efcc80, 0x40001031e0}, 0x4000c1e120, 0x0)
        google.golang.org/grpc@v1.57.1/server.go:1735 +0x7c4
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        google.golang.org/grpc@v1.57.1/server.go:970 +0xb4
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 44
        google.golang.org/grpc@v1.57.1/server.go:981 +0x160

Error: The terraform-provider-kubectl_v2.0.4 plugin crashed!

If you need more data pls let me know how to send privately.
Thx

[jeanbrunogabriel]

same here, I've just migrated from gavinbunney/kubectl. Downgraded to 2.0.3 and it worked

[q-lawrencewebber]

Hey do you have an idea of when this change will get merged? I'd like to use one of the features that came out with v2.0.4. Also good work on the fix!