apiclient ephemeral-storage bind Failed to initialize ephemeral storage. specified bind directory not in allow list

[Diego-Mendes-Forter]

Hi there,

I am trying to run an ElasticSearch pods in an EKS cluster with Bottlerocket EC2 images, and I am having a hard time to create the Volumes in Bottlerocket to store the ES data into the ephemeral disks in the node.

I am currently using the node with an EC2 is4gen.2xlarge that has an SSD for the data, when I connect to the node using SSM, to try to mount the disk into a path where I can set the Volume, it fails with the message below.

[ssm-user@control]$ apiclient ephemeral-storage bind --dirs /var/lib/elasticsearch
Failed to initialise ephemeral storage: Failed POST request to '/actions/ephemeral-storage/bind': Status 500 when POSTing /actions/ephemeral-storage/bind: Unable to bind ephemeral storage: Invalid Parameter '/var/lib/elasticsearch', specified bind directory not in allow list

If I understood correctly, I can only setup the following dirs

"/var/lib/containerd", "/var/lib/kubelet", "/var/log/pods"

But I don't understand why?

Are there any other way to setup the volumes in bottlerocket to be used by the pods?

[Diego-Mendes-Forter]

On another try, I tried setting up the following to make a RAID disk, but the bootstrap commands end up overriding the RAID configurations.

I assume we can't have both as well?

# The container image specified here contains the configuration scripts needed to setup the RAID0 on the node
[settings.bootstrap-containers.setup-runtime-storage]
source = "myecr/image"
mode = "always"
essential = true

# This bootstrap command will setup an ephemeral folder on /mnt/.ephemeral that can be used by storageclasses
[settings.bootstrap-commands.k8s-ephemeral-storage]
commands = [
    ["apiclient", "ephemeral-storage", "init"],
    ["apiclient", "ephemeral-storage" ,"bind", "--dirs", "/var/lib/containerd", "/var/lib/kubelet", "/var/log/pods"]
]
essential = true
mode = "always"

[vigh-m]

Hi @Diego-Mendes-Forter , From what I understand, you are wanting to store ElasticSearch data on the Instance Store volumes on your node.
The ephemeral-storage option is more to allow caching container images in the ephemeral storage. The paths are restricted to "/var/lib/containerd", "/var/lib/kubelet", "/var/log/pods" since they are required to be configured before we start containerd and kubelet (or docker for ecs).

If you want to set /var/lib/elasticsearch to local storage, using AWS CSI drivers would be the better option. You can also use Storage Classes to set this up manually if you choose. Please be aware that you will need the right path to the ephemeral storage.
You can also use a bootstrap container to set up the RAID array and mount it in arbitrary paths. You will have to expose the mount to their pods using local volumes or PersistentVolume / PersistenVolumeClaims.

We will track your use case to allow ephemeral-storage option to mount arbitrary paths. We're also happy to review and accept PRs for this as well.

[bcressey]

We will track your use case to allow ephemeral-storage option to mount arbitrary paths. We're also happy to review and accept PRs for this as well.

I don't expect that mounting arbitrary paths will ever be allowed. /var/lib is "owned" by the host and the directories it interacts with (such as /var/lib/bottlerocket) often need special SELinux labels. It might be possible to block a list of known subdirectories, but that may create unexpected incompatibilities in the future if the host starts using another subdirectory.

I'd be more open to allowing binds to any subdirectory of /mnt, e.g. /mnt/elasticsearch, or to any subdirectory of /opt except for /opt/cni or /opt/csi (which also need special treatment).

[zaheer-merali]

I'd be more open to allowing binds to any subdirectory of /mnt, e.g. /mnt/elasticsearch, or to any subdirectory of /opt except for /opt/cni or /opt/csi (which also need special treatment).

@bcressey Would this be something to consider if using the local storage stuff in k8s to share the filesystem with bind mounts as per https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner/blob/master/docs/operations.md#operations or is there a more elegant solution to provide nvme storage to pods?

[bcressey]

@zaheer-merali I would treat it as an either/or - either use bootstrap commands or bootstrap containers to manage ephemeral storage, or else leave it entirely hands-off on the host and do all the configuration via Kubernetes. Trying to mix the two risks data loss if one of the two implementations decides to format a disk that the other wants to use.

I have this snippet from long-ago experiments with the static provisioner:

❯ cat helm/examples/bottlerocket.yaml
common:
  useNodeNameOnly: true
classes:
- name: local-storage
  hostDir: "/dev/disk/ephemeral"
  mountDir: "/dev/disk/ephemeral"
  storageClass: true

In terms of when to use it, I would say:

• Use the static provisioner when you don't want arbitrary containers to use the storage via emptyDir volumes (etc), and when you have a very clear idea of which pod should own the storage and which node should run that pod. There's some manual management of PVs and PVCs that will be painful unless you want or need that degree of control.
• Use bootstrap commands or bootstrap containers when you want all containers on the node to make use of the ephemeral storage automatically, and when pods aren't strongly tied to nodes. If the node can die at any time and the pod can come back up elsewhere, hydrate the data it needs, and rejoin its cluster as a peer, then you may not benefit from the more fine-grained control the static provisioner offers.

[zaheer-merali]

@bcressey What we want to do is to be able to subdivide volumes from a single nvme disk and be able to provide multiple distinct storage volumes for pods. The pods aren't tied to nodes and we're fine with nodes dying at any time and hydrating the data it. We had initially thought bootstrap commands to initialize the nvme storage and bind mount a number of volumes similar to what the static local provisioner docs say in the above link. This initialization and bind mounting is what we do for Amazon Linux 2 nodes in shell passed to user-data.

[zaheer-merali]

@bcressey This is a draft PR to allow /mnt/ prefixed mount points for ephemeral-storage bind bottlerocket-os/bottlerocket-core-kit#320