Volume of size 1GB is smaller than snapshot

[bwmetcalf]

Version

Karpenter Version: v0.22.1

Kubernetes Version: v1.24.0

Expected Behavior

We have the following node template (relevant parts shown):

...
  spec:
    amiFamily: Custom
    amiSelector:
      aws-ids: ami-012cc3171fae12cba
    blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        deleteOnTermination: true
        encrypted: true
        volumeSize: 2G
        volumeType: gp2
    - deviceName: /dev/xvdb
      ebs:
        deleteOnTermination: true
        encrypted: true
        volumeSize: 1T
        volumeType: gp2
...

and provisioner:

...
spec:
  consolidation:
    enabled: true
  limits:
    resources:
      memory: 240Gi
  providerRef:
    name: default-node-template
...

and expect a new node to spun up.

Actual Behavior

We are getting the following in the logs:

2023-01-27T16:04:56.923Z	INFO	controller.provisioner	launching node with 1 pods requesting {"cpu":"1030m","memory":"3999Mi","pods":"7"} from types m5.2xlarge, m5.xlarge	{"commit": "c4a4efd-dirty", "provisioner": "karpenter-provisioner"}
2023-01-27T16:04:58.120Z	ERROR	controller.provisioner	launching machine, creating cloud provider instance, creating instance, with fleet error(s), InvalidBlockDeviceMapping: Volume of size 1GB is smaller than snapshot 'snap-090fea468f03e1c8b', expect size >= 2GB	{"commit": "c4a4efd-dirty"}

Steps to Reproduce the Problem

Install karpenter 0.22.1 and create node template similar to the above.

Resource Specs and Logs

See above.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[bwmetcalf]

This was resolved by using Gi for the volume size units in the node template. However, I believe this is still a bug as G is accepted as units for the node template. I tried using 4G for volumeSize but still received the same error in the provisioner logs.

[jonathan-innis]

This is the case with any blockDeviceMapping that you would use for any EC2 instance. You need to provide a blockDeviceMapping that is big enough to hold the size of the AMI snapshot that you are specifying. Do you know the size for your snapshot?

[bwmetcalf]

This is the case with any blockDeviceMapping that you would use for any EC2 instance. You need to provide a blockDeviceMapping that is big enough to hold the size of the AMI snapshot that you are specifying. Do you know the size for your snapshot?

Yes. The snapshot is 2GB and even when specifying a volume size using the string 4G the error occurred. Using units of Gi works and allowed me to specify 2Gi as the volume size.

[jonathan-innis]

Have you tried launching an ec2 instance manually with the snapshot and the blockDeviceMapping and are you observing the same error. Trying to nail down if this is an issue with our launch or if this issue is deeper in EC2?

[bwmetcalf]

Have you tried launching an ec2 instance manually with the snapshot and the blockDeviceMapping and are you observing the same error. Trying to nail down if this is an issue with our launch or if this issue is deeper in EC2?

It appears the UI has the units fixed at GiB so perhaps trying to use G as the units is the source of the issue?

[jonathan-innis]

@bwmetcalf I think it's possible that our conversion is wrong here. Let me test it on my side and get back to you. We should be able to take both unit types into the blockDeviceMappings, but, for now, if G is giving you issues, use GiB

[jonathan-innis]

@bwmetcalf Found a couple issues as a result of this bug. Linked the PRs. As a short-term fix until the changes are out, I'd recommend just using Gi

[bwmetcalf]

@bwmetcalf Found a couple issues as a result of this bug. Linked the PRs. As a short-term fix until the changes are out, I'd recommend just using Gi

Awesome. Thanks for looking into this.