Error: Value Conversion Error - crashes terraform plan

[adampeklay]

Terraform Version, Provider Version and Kubernetes Version

Terraform version:

1.6.6

I've also tried all of these versions, same error:

jenkins@jenkins:~$ tfenv list
* 1.6.6 (set by /var/lib/jenkins/.tfenv/version)
  1.6.0
  1.5.7
  1.5.6
  1.5.5
  1.5.4
  1.5.3
  1.5.2
  1.5.0
  1.4.7

AWS provider version:

5.31.0

EKS Version

1.28

Affected Resource(s)

EKS cluster

Terraform Configuration Files

module "eks_cluster" {
  source  = "cloudposse/eks-cluster/aws"
  version = "3.0.0"
  // https://github.com/cloudposse/terraform-aws-eks-cluster

  namespace = var.namespace
  stage     = var.stage
  name      = var.name
  region    = var.region

  apply_config_map_aws_auth                             = false
  cluster_encryption_config_enabled                     = true
  cluster_encryption_config_kms_key_enable_key_rotation = true

  kubernetes_version        = var.kubenetes_version
  addons                    = var.addons
  vpc_id                    = module.vpc.vpc_id
  subnet_ids                = module.subnets.public_subnet_ids
  public_access_cidrs       = var.public_access_cidrs
  endpoint_private_access   = var.endpoint_private_access
  endpoint_public_access    = var.endpoint_public_access
  enabled_cluster_log_types = var.enabled_cluster_log_types

  tags = var.tags

}

Steps to Reproduce

terraform plan

Expected Behavior

What should have happened?
terraform plan succeeds

Actual Behavior

What actually happened?

terraform plan fails

...
...
...
  + public_subnet_cidrs                             = [
      + (known after apply),
      + (known after apply),
      + (known after apply),
    ]
  + secondary_eks_node_group_cbd_pet_name           = (known after apply)
  + secondary_group_remote_access_security_group_id = (known after apply)
  + ssh_security_group_id                           = (known after apply)
  + vpc_arn                                         = (known after apply)
  + vpc_cidr_block                                  = (known after apply)
  + vpc_default_security_group_id                   = (known after apply)
  + vpc_id                                          = (known after apply)
  + vpc_main_route_table_id                         = (known after apply)

Warning: Argument is deprecated

  with module.subnets.aws_eip.default,
  on .****/modules/subnets/main.tf line 286, in resource "aws_eip" "default":
 286:   vpc = true

use domain attribute instead

(and 3 more similar warnings elsewhere)

Error: Value Conversion Error

  with module.eks_cluster.provider["registry.****.io/hashicorp/kubernetes"],
  on .****/modules/eks_cluster/auth.tf line 96, in provider "kubernetes":
  96: provider "kubernetes" {

An unexpected error was encountered trying to build a value. This is always
an error in the provider. Please report the following to the provider
developer:

Received unknown value, however the target type cannot handle unknown values.
Use the corresponding `types` package type or a custom type that handles
unknown values.

Path: exec
Target Type: []struct { APIVersion basetypes.StringValue
"tfsdk:\"api_version\""; Command basetypes.StringValue "tfsdk:\"command\"";
Env map[string]basetypes.StringValue "tfsdk:\"env\""; Args
[]basetypes.StringValue "tfsdk:\"args\"" }
Suggested Type: basetypes.ListValue
[Pipeline] }
[Pipeline] // stage
[Pipeline] slackSend
Slack Send Pipeline step running, values are - baseUrl: <empty>, teamDomain: <REDACTED>, channel: <REDACTED>, color: bad, botUser: false, tokenCredentialId: <REDACTED>, notifyCommitters: false, iconEmoji: <empty>, username: <empty>, timestamp: <empty>
[Pipeline] }
[Pipeline] // withVault
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: FAILURE

Important Factoids

This was working just fine on Tuesday, then I got busy Wednesday with unplanned work, then Thursday this started bombing out.

• I updated all provider versions on Thursday to see if that would help, it did not
• I tried removing (on jenkins server) .kubernetes.d, there was no lock to remove, I even deleted the workspace directories too. No luck
• I added add-ons on Tuesday, so figured I'd remove that from my .tfvars but that was not the issue either
• There are no resources built, I have a destroy pipeline I used Tuesday afternoon, and I ran it again just to be sure
• This is my sandbox cluster, but blocking work to get lower envs and production built via terraform & CI/CD
• I've also scoured the internet, nothing seems to help. It all points back to provider it seems, however I looked at open and closed issues too on several repos and there is nothing out there regarding this.
• I also set dummy_kubeapi_server = null per information in (from error message):

.terraform/modules/eks_cluster/auth.tf line 96, in provider "kubernetes"

That did not help

• I don't think its local to me but not 100% sure 🤷

Please let me know if you need any additional information, thank you!

[adampeklay]

CloudPosse helped me fix this, via their SweetOps Slack workspace.

I had to declare and pin kubernetes provider to 2.24.0

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.31.0"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.24.0"
    }
  }
}

Feedback was a new provider framework, that does not support computed lists.

I will close this out as I have a solution, hopefully that will help others.

Big shout out to CloudPosse for the help! I was pretty stuck on this the past day 🙃

[n6gse]

@adampeklay Thanks for documenting this. I was getting the same error for exactly the same reason.

[Nuru]

@adampeklay Please re-open this issue. You have a workaround for it, not a fix. The fix needs to be made by Hashicorp so that their provider does not crash.

[pellepelster]

We are affected by this issue as well the mentioned bug appeared in 2.25.0 for us and persists in 2.26.0