aws_launch_template with eks managed node groups does not allow adding metadata_options

[lorelei-rupp-imprivata]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform 0.14.7
AWS Provider 3.73.3

Affected Resource(s)

aws_launch_template

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

We have the resource defined like this:

resource "aws_launch_template" "worker-launch-template" {

  for_each               = local.node_group_for_each
  image_id               = var.eks_worker_instance_ami
  name                   = format("%s_%s_launch_template", module.eks_cluster.eks_cluster_name, each.key)
  tags                   = var.custom_tags_eks_cluster
  update_default_version = true
  key_name               = var.eks_worker_keypair_name # key name to access ec2 instance
  network_interfaces {
    associate_public_ip_address = var.associate_public_ip
  }
  block_device_mappings {
    device_name = "/dev/sda1"
    ebs {
      volume_size = var.node_group_default_disk_size
    }
  }
  tag_specifications {
    resource_type = "instance"
    tags          = local.combined_node_group_configurations_with_az_final_map[each.key].tags
  }
  user_data = base64encode(data.template_file.workers_user_data.rendered)
  metadata_options {
    http_endpoint = "enabled"
    http_tokens   = "required"
    http_put_response_hop_limit = 2
  }
}

Expected Behavior

We are trying to lock down our Instance Metadata according to https://docs.bridgecrew.io/docs/bc_aws_general_31 and best practices
We expected to be able to configure metadata_option on our launch template, and have worker nodes come online and connect to the cluster

Actual Behavior

When adding the metadata options like the snippet above we see these errors

error waiting for EKS Node Group (saas-checkovfi2-eks:saas-checkovfi2-eks-app-workers-us-west-2a) to create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:
* i-09df7a27f0d426e1a, i-0fd09d65187ba7f7e: NodeCreationFailure: Instances failed to join the kubernetes cluster
*

Steps to Reproduce

Create eks managed node group with a launch template that sets metadata_options

[justinretzolk]

Hey @lorelei-rupp-imprivata 👋 Thank you for taking the time to raise this! So that we have all of the necessary information in order to look into this, can you supply (redacted as necessary) debug logs as well?

[lorelei-rupp-imprivata]

Here is the TF Debug piece for the failure. Its really just reflecting what the aws console says too that it cannot connect to the cluster

---[ REQUEST POST-SIGN ]-----------------------------
GET /clusters/saas-XXX-eks/node-groups/saas-XXX-eks-app-workers-us-west-2a HTTP/1.1
Host: eks.us-west-2.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/0.14.7 (+https://www.terraform.io) terraform-provider-aws/3.73.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.42.34 (go1.16; linux; amd64)


-----------------------------------------------------: timestamp=2022-06-14T16:06:52.944-0400
2022-06-14T16:06:53.741-0400 [INFO]  plugin.terraform-provider-aws_v3.73.0_x5: 2022/06/14 16:06:53 [DEBUG] [aws-sdk-go] DEBUG: Response eks/DescribeNodegroup Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Content-Length: 2496
Access-Control-Allow-Headers: *,Authorization,Date,X-Amz-Date,X-Amz-Security-Token,X-Amz-Target,content-type,x-amz-content-sha256,x-amz-user-agent,x-amzn-platform-id,x-amzn-trace-id
Access-Control-Allow-Methods: GET,HEAD,PUT,POST,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: x-amzn-errortype,x-amzn-errormessage,x-amzn-trace-id,x-amzn-requestid,x-amz-apigw-id,date
Content-Type: application/json
Date: Tue, 14 Jun 2022 20:06:53 GMT
X-Amz-Apigw-Id: TumOoEfMvHcF1XA=
X-Amzn-Requestid: aa471e07-dcb5-4368-a146-9536da53eee8
X-Amzn-Trace-Id: Root=1-62a8ea5d-43beb6b6524b16f836a16eff


-----------------------------------------------------: timestamp=2022-06-14T16:06:53.741-0400
2022-06-14T16:06:53.742-0400 [INFO]  plugin.terraform-provider-aws_v3.73.0_x5: 2022/06/14 16:06:53 [DEBUG] [aws-sdk-go] {
  "nodegroup" : {
    "nodegroupName" : "saas-XXX-eks-app-workers-us-west-2a",
    "nodegroupArn" : "arn:aws:eks:us-west-2:xxxxxxxxxxxx:nodegroup/saas-XXX-eks/saas-XXX-eks-app-workers-us-west-2a/XXXXX",
    "clusterName" : "saas-XXX-eks",
    "version" : null,
    "releaseVersion" : "ami-XXXX",
    "createdAt" : 1.655235742923E9,
    "modifiedAt" : 1.655237205113E9,
    "status" : "CREATE_FAILED",
    "capacityType" : "ON_DEMAND",
    "scalingConfig" : {
      "minSize" : 1,
      "maxSize" : 10,
      "desiredSize" : 2
    },
    "instanceTypes" : [ "m5.large" ],
    "subnets" : [ "subnet-XXX" ],
    "remoteAccess" : null,
    "amiType" : "CUSTOM",
    "nodeRole" : "arn:aws:iam::xxxxxxxxxxxx:role/saas-XXX-eks-workers-role",
    "labels" : null,
    "taints" : null,
    "resources" : {
      "autoScalingGroups" : [ {
        "name" : "eks-saas-XXX-eks-app-workers-us-west-2a-XXX"
      } ],
      "remoteAccessSecurityGroup" : null
    },
    "diskSize" : null,
    "health" : {
      "issues" : [ {
        "code" : "NodeCreationFailure",
        "message" : "Instances failed to join the kubernetes cluster",
        "resourceIds" : [ "i-XXX", "i-XXX" ]
      } ]
    },
    "updateConfig" : {
      "maxUnavailable" : 1,
      "maxUnavailablePercentage" : null
    },
    "launchTemplate" : {
      "name" : "saas-XXX-eks_app-workers-us-west-2a_launch_template",
      "version" : "1",
      "id" : "lt-XXX"
    },
    "tags" : {}
  }
}: timestamp=2022-06-14T16:06:53.741-0400
2022/06/14 16:06:53 [DEBUG] module.eks-cluster-managed-workers.aws_eks_node_group.eks_worker["app-workers-us-west-2a"]: apply errored, but we're indicating that via the Error pointer rather than returning it: error waiting for EKS Node Group (saas-XXX-eks:saas-XXX-eks-app-workers-us-west-2a) to create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:
	* i-xxx, i-xxx: NodeCreationFailure: Instances failed to join the kubernetes cluster

[lorelei-rupp-imprivata]

Similar issue #25909

[Kyle-Hughes-Imprivata]

Still an issue, is there work slated to fix this?

[lorelei-rupp-imprivata]

This may actually be fixed now, in provider 4.46.0, I am seeing the workers join the cluster

[justinretzolk]

Hey @james64 👋 Thank you very much for taking the time to raise this! This was addressed with #30545, which was included in version 5.0.0 of the provider. With that in mind, we'll close this issue. If you experience additional issues with the provider, please do open a new issue to let us know.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.