Failing to extend the partition size

[osibanda]

Platform I'm building on:
We are running EKS with Kubernetes version 1.21.6 and platform version eks.4.

Problem statement
The issue is disk pressure warnings occur intermittently on a particular node in the cluster. We extended the volumes attached to that node and wish to extend the partition sizes. Running df -f returns the attached output df-output.xlsx.

What I expected to happen:
When I run lsblk -f I should get the filesystem type and the breakdown of the partitions. Thus we are not able to see the filesystem type. However, running xfs_growfs -d returns /dev/nvme0n1 is not a mounted XFS filesystem and resize2fs returns open: No such file or directory while opening directory /dev/nvme0n1. I have tried using the following:

• /dev/xvda (Volume name in AWS console)
• /dev/nvme0n1 (result from lsblk)
• /dev/nvme0n1p1
• As well as the mount directory

All to no avail.

What actually happened:
lsblk -f returns blank filesystem column as shown below:

NAME            FSTYPE LABEL UUID MOUNTPOINT
loop0                                     /.bottlerocket/rootfs/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/licenses
loop1                                     /.bottlerocket/rootfs/var/lib/kernel-devel/.overlay/lower
nvme0n1
|-nvme0n1p1
|-nvme0n1p2
|-nvme0n1p3                         /.bottlerocket/rootfs/boot
|-nvme0n1p4
|-nvme0n1p5
|-nvme0n1p6
|-nvme0n1p7
|-nvme0n1p8
|-nvme0n1p9
|-nvme0n1p10
|-nvme0n1p11
`-nvme0n1p12                        /.bottlerocket/rootfs/var/lib/bottlerocket
nvme1n1
`-nvme1n1p1                         /.bottlerocket/rootfs/local
nvme2n1

lsblk returns:

NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0          7:0    0  280K  1 loop /.bottlerocket/rootfs/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/licenses
loop1          7:1    0 11.6M  1 loop /.bottlerocket/rootfs/var/lib/kernel-devel/.overlay/lower
nvme0n1      259:0    0  160G  0 disk
|-nvme0n1p1  259:3    0    4M  0 part
|-nvme0n1p2  259:4    0    5M  0 part
|-nvme0n1p3  259:5    0   40M  0 part /.bottlerocket/rootfs/boot
|-nvme0n1p4  259:6    0  920M  0 part
|-nvme0n1p5  259:7    0   10M  0 part
|-nvme0n1p6  259:8    0   25M  0 part
|-nvme0n1p7  259:9    0    5M  0 part
|-nvme0n1p8  259:10   0   40M  0 part
|-nvme0n1p9  259:11   0  920M  0 part
|-nvme0n1p10 259:12   0   10M  0 part
|-nvme0n1p11 259:13   0   25M  0 part
`-nvme0n1p12 259:14   0   42M  0 part /.bottlerocket/rootfs/var/lib/bottlerocket
nvme1n1      259:1    0   50G  0 disk
`-nvme1n1p1  259:2    0   20G  0 part /.bottlerocket/rootfs/local
nvme2n1      259:15   0   20G  0 disk

How to reproduce the problem:
Log in to an EC2 instance which is a cluster node running Bottlerocket and switch to admin bash. Increase the partition size of any mounted volume.

[webern]

I'm sorry to hear this, let me get a bit more information and try to reproduce this issue:

• Region
• Your AMI ID and/or Bottlerocket version? (Are you using a custom-built AMI or one of our published AMIs?)
• Instance type
• Are you using any custom CSI driver?

on a particular node in the cluster

Do you mean that the problem seems to be isolated to a particular EC2 instance ID? Or do you mean that the problem is happening randomly at some frequency rate when spinning up new instances?

[webern]

I initially misunderstood the question. I am now trying to extend the data partition of a Bottlerocket instance and will let you know what I come up with.

[webern]

Ok, I was able to grow the data partition using the following procedure. I believe you may have had trouble because you had not used sheltie to enter the host namespaces (see instructions below).

Increase the disk size in the AWS console

• In the AWS console, identify the EBS volume labeled /dev/xvdb. If you ran Bottlerocket with default configurations, this should be 20G. (Note that we do not need to grow the 2GB OS partition, /dev/xvda.)
• Increase the size of the data volume in the EBS console.
• Wait and verify that the size increase has taken place (for example in the EC2 con

Increase the Partition size.

We can find the commands that Bottlerocket uses to grow the data partition here https://github.com/bottlerocket-os/bottlerocket/blob/d8698d9abef537a3dde4a39eff8a894a52e2a73d/packages/release/repart-local.service#L16..L20

Note that this service runs on boot, so you could simply reboot the instance. Also note that it is critical that you run exactly these commands, otherwise on reboot the partition could shrink and then the filesystem would be out of bounds and not mountable.

If you want to grow the partition without rebooting:

• Connect to the Bottlerocket instance using SSM.

  • aws ssm start-session --target [instance-id]
  • run enable-admin-container and wait a few seconds. Wait a bit for the admin container to be downloaded and start.
  • run apiclient exec admin bash to "enter" the admin container.
  • IMPORTANT Once in the admin container run sudo sheltie to enter the host namespaces.

• If we lsblk -l we should see nvme1n1 shows the increased space. Meanwhile the partition nvme1n1p1 needs to have its size increased.

• Run the commands that we found in repart-local.service

  • /usr/bin/systemd-repart --dry-run=no /dev/disk/by-partlabel/BOTTLEROCKET-DATA
  • /usr/lib/systemd/systemd-growfs /local

• Check lsblk -l and you should see the partition size has increased.

Alternatively, from the control container, you can run these commands through apiclient exec and sheltie non-interactively:

• apiclient exec admin sudo sheltie /usr/bin/systemd-repart —dry-run=no /dev/disk/by-partlabel/BOTTLEROCKET-DATA
• apiclient exec admin sudo sheltie /usr/lib/systemd/systemd-growfs /local

[cmartell-at-ocp]

Might simply be that we need to update? AMI was bottlerocket-aws-k8s-1.20-x86_64-v1.4.1-1865c47c. I entered the admin container.

[webern]

Oh interesting. v1.4.1 is from November 2021 and the change to use systemd-repart is more recent than that: 067671a

The simplest way to grow the partition in any case is to reboot the node. It should happen automatically on boot. If you cannot afford to reboot this node, let me know and we can try to figure out what commands will work on 1.4.1.

[cmartell-at-ocp]

Yeah, that's what I've been doing (rebooting the node). I tried to use /usr/sbin/growpart /dev/disk/by-partlabel/BOTTLEROCKET-DATA as per https://github.com/bottlerocket-os/bottlerocket/blob/v1.4.1/packages/release/prepare-local.service but it would hang at "writing partition table". I'll see if I can spin up a test node and try it out again.

I'll report back.

[webern]

Growpart wasn't really designed to grow the partition outside of the boot process, and it had some issues anyway (which is why we replaced it with systemd-growfs). If growing the partition without rebooting is something that you need to do often, I would recommend moving to the latest Bottlerocket version and using systemd-growfs.

[shreyasGit]

thanks for the steps, after following these steps i see df -h showing increased partition, but when i do a kubectl describe nodes, it still shows ephemeral storage as old one in my case 20G

[webern]

@OscarSGit let us know if you need any additional help with this.

For the record, we're leaving this issue open because we want to add something to the documentation about this.

[osibanda]

@webern thank you for the suggestion, let me try it out and will update once I have feedback.

[ozancaglayan]

Hi.

I'm following this doc to create an EBS volume to pre-cache some large docker containers
https://aws.amazon.com/blogs/containers/reduce-container-startup-time-on-amazon-eks-with-bottlerocket-data-volume/

am I understanding correctly that this approach would simply fail if the containers that I'm pulling exceed 20GB? I don't want to believe that this is the case because it defies the purpose of that article :)

If yes, does one need to do the steps explained in this issue to really increase the size of that data volume? I'm using the cloudformation script provided by the above article. I'm seeing that the data volume is really 20gb in EC2 console ☹️

P.S: Okay apparently it is possible to increase the EBS size in this case using a LaunchTemplate.

Thanks

[bcressey]

Does one need to do the steps explained in this issue to really increase the size of that data volume? I'm using the cloudformation script provided by the above article. I'm seeing that the data volume is really 20gb in EC2 console

This discussion is about extending the data volume after the node has been running for some time. If you can launch the node with a large enough volume in the first place, then that makes things much easier as the initial resize will happen automatically.

For example, that doc shows an eksctl snippet with an additionalVolumes map. You could specify the volumeSize attribute for xvdb in order to set it to something besides 20 GiB. (See docs for additional volumes mapping.) The syntax is different for each system - eksctl, Karpenter, Terraform, launch templates - but they all have a way to specify the volume size.