Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add OS and Kubelet disk type options. #385

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: Add OS and Kubelet disk type options. #385

wants to merge 2 commits into from

Conversation

Carus11
Copy link

@Carus11 Carus11 commented May 17, 2024 •
edited
Loading

Fixes #363

This PR provides functionality to set the OS Disk Type and Kubelet Disk Type options as documented in:
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster_node_pool#kubelet_disk_type

This gives the user the ability to choose the use the local temporary storage available to some nodes for the Operating system, and or the Kubelet.
The benefit of doing this is the local temporary disk is typically a high performance type of storage medium. If the kubelet is set to use this fast storage, then local emptyDir volume can be used for workloads such as SASWORK. Previously many have used alternatives such as hostPath access methods to this storage which is not often permitted by security policies.

OS Disk type can be either Managed or Ephemeral.
Kubelet Disk type can be either OS or Temporary.
The default behaviour for the option, and when not set, is to use Managed and OS for the OS and Kubelet respectively. When set to Temporary and Ephemeral the local temporary storage will be used if available on the node.

Test 1

Original State. The example tfvars file contains an OS disk of 200GB. This means the kubelet will share this managed disk with the OS, placing both the kubernetes emptyDir volumes, and container image storage on the same disk as the OS.

Name       OsDiskType    KubeletDiskType
---------  ------------  -----------------
system     Managed       OS
compute    Managed       OS
stateless  Managed       OS
cas        Managed       OS
stateful   Managed       OS

Test 2

Functionality added but nothing added to tfvars file

Name       OsDiskType    KubeletDiskType
---------  ------------  -----------------
system     Managed       OS
stateful   Managed       OS
stateless  Managed       OS
compute    Managed       OS
cas        Managed       OS

Test 3

Compute node pool changed to VM which has temporary storage, Standard_E16ds_v5, os disk reduced to 80gb, and both OS and Kubelet disk type switched to their non-standard options

Name       OsDiskType    KubeletDiskType
---------  ------------  -----------------
system     Managed       OS
stateful   Managed       OS
stateless  Managed       OS
cas        Managed       OS
compute    Ephemeral     Temporary

Notice the OS has carved off the 80GB from the (as advertised) 600GB disk. And the kubelet has the remaining space.

root@aks-compute-37429462-vmss000000:/# lsblk -f
NAME    FSTYPE FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
|-sda1  ext4   1.0   cloudimg-rootfs 2c316610-0ba2-47f0-8c70-5162294156ed   56.2G    27% /
|-sda14
`-sda15 vfat   FAT32 UEFI            51A8-FB00                              98.3M     6% /boot/efi
sdb
`-sdb1  ext4   1.0                   e2269d85-dc0c-4cae-b7b1-4577761f17f7  483.7G     0% /var/lib/kubelet
                                                                                         /mnt
sr0

Test 4

Compute OS type set to ephemeral, and kubelet set to OS disk

Name       OsDiskType    KubeletDiskType
---------  ------------  -----------------
system     Managed       OS
compute    Ephemeral     OS
stateful   Managed       OS
stateless  Managed       OS
cas        Managed       OS

It looks like both kubelet and the OS use the same OS disk size, squeezing it all into the 80GB which is carved off from the local temporary disk.

root@aks-compute-34891476-vmss000000:/# lsblk -f
NAME    FSTYPE FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
|-sda1  ext4   1.0   cloudimg-rootfs 2c316610-0ba2-47f0-8c70-5162294156ed   57.2G    26% /var/lib/kubelet
|                                                                                        /
|-sda14
`-sda15 vfat   FAT32 UEFI            51A8-FB00                              98.3M     6% /boot/efi
sdb
`-sdb1  ext4   1.0                   89f157fc-0328-4eac-a8ff-53aaa01fd11c  484.8G     0% /mnt
sr0

Nothing in /mnt

root@aks-compute-34891476-vmss000000:/# ls /mnt
DATALOSS_WARNING_README.txt  lost+found

Test 5

Set the OS Disk to Managed, and the Kubelet disk to Temporary

Name       OsDiskType    KubeletDiskType
---------  ------------  -----------------
system     Managed       OS
stateful   Managed       OS
stateless  Managed       OS
cas        Managed       OS
compute    Managed       Temporary

We can see that kubelet is on sdb1, and it has almost all of the local temporary disk for use.

root@aks-compute-27846506-vmss000000:/# lsblk -f
NAME    FSTYPE FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
|-sda1  ext4   1.0   cloudimg-rootfs 2c316610-0ba2-47f0-8c70-5162294156ed   56.2G    27% /
|-sda14
`-sda15 vfat   FAT32 UEFI            51A8-FB00                              98.3M     6% /boot/efi
sdb
`-sdb1  ext4   1.0                   8a41e4e7-38ce-43b7-b7d9-48565e46c4b9  558.5G     0% /var/lib/kubelet
                                                                                         /mnt
sr0

Carus11 added 2 commits May 17, 2024 14:04
@Carus11
feat: Add OS and Kubelet disk type options.
5515edb 
OS Disk type can be either Managed or Temporary.
Kubelet Disk type can be either OS or Ephemeral.
The default behaviour for the option, and when not set, is to use
Managed and OS for the OS and Kubelet respectively. When set to
Temporary and Ephemeral the local temporary storage will be used if
available on the node.

Signed-off-by: Carus Kyle <caruskyle@gmail.com>
@Carus11
Merge branch 'main' into kubelet_disk_type_temp
0aefd6d
@riragh
Copy link
Member

riragh commented May 20, 2024

@Carus11, Thank you for providing the PR and your tests. However, until the feature request is accepted this PR will not be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Feature Request: add kubelet_disk_type to compute node pools
2 participants
@Carus11 @riragh