new changes to blockDeviceMappings, deleting existing nodes replaces with old blockDeviceMappings and not the new ones

[claytonchew]

Version

Karpenter Version: v0.20.0

Kubernetes Version: v1.24.8-eks-ffeb93d

We created a new AWSNodeTemplate to increase blockDeviceMappings' volumeSize (previously they were not specified, hence it defaulted to 20gb). We are unable to roll this change to existing nodes.

Expected Behavior

Manually deleting existing nodes should replace new nodes based on the new blockDeviceMappings.

Actual Behavior

Manually deleting existing nodes replace new nodes based on the old blockDeviceMappings (not the new).

*However, new workloads requiring new nodes created new nodes based on the new blockDeviceMappings.

Steps to Reproduce the Problem

1. Apply new change to blockDeviceMappings.
2. Delete existing node.

Resource Specs and Logs

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 30Gi #increase size
        volumeType: gp2
        deleteOnTermination: true

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  consolidation:
    enabled: true
  requirements: []
  providerRef:
    name: default

karpenter-57c5f67dd6-qtc2r 2023-02-15T04:58:16.456Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-12-166.ap-southeast-
karpenter-57c5f67dd6-qtc2r 2023-02-15T04:59:15.494Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-12-89.ap-southeast-1
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:02:56.617Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-12-12.ap-southeast-1
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:04:12.640Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-11-49.ap-southeast-1
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:04:12.894Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-12-146.ap-southeast-
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:04:13.109Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-11-178.ap-southeast-
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:08:16.394Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-12-166.ap-southeast-
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:09:15.442Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-12-89.ap-southeast-1
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:12:56.612Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-12-12.ap-southeast-1
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:14:12.635Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-11-49.ap-southeast-1
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:14:12.843Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-12-146.ap-southeast-
karpenter-57c5f67dd6-qtc2r 2023-02-15T05:14:13.087Z    INFO    controller.inflightchecks    Inflight check failed for node, Expected 30Gi of resource ephemeral-storage, but found 20959212Ki (66.6% of expected)    {"commit": "f60dacd", "node": "ip-10-0-11-178.ap-southeast-

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

This PR fixed this issue. The hash that we were including didn't include the resource size. v0.24.0 fixes this issue