ImageVolume support incomplete?

[andraxin]

What happened:

I enabled the ImageVolume feature gate and attempted to create a pod
using a manifest (adapted from the tutorial example ) with a peculiar twist:
the pod's main container image was empty and the image volume was
(supposed to get) mapped to the container's root (i.e. /, details below).

Pod creation failed with the error:

failed to apply OCI options: failed to mkdir "": mkdir : no such file or directory

For all I know, this could be a wider issue, not specific to kind...

BTW, another surprise (for me, in any case) was that without enabling
the feature gate not only was the image volume not mounted, which I'd
totally understand, but the volume specification seemed to get ignored
silently, without any error -or even warning- to that effect.

What did you expect to happen:

I expected that a pod would get created with its root filesystem
mounted from the image volume, allowing the container to run
despite the fact that the container image was empty.

How to reproduce it (as minimally and precisely as possible):

1. Create a cluster with the ImageVolume feature gate enabled:

   $ cat << YAML | kind create cluster --config -
   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   featureGates:
     ImageVolume: true
   YAML

2. Apply the manifest to create a pod

   $ cat << YAML | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: image-volume
   spec:
     containers:
     - name: shell
       command: ["sleep", "infinity"]
       image: andraxin/empty
       volumeMounts:
       - name: volume
         mountPath: /
     volumes:
     - name: volume
       image:
         reference: debian
         pullPolicy: IfNotPresent
   YAML

3. Describe the pod (or monitor events) to find the exact error

   $ kubectl describe pod | sed 1,/^Events/d
     Type     Reason     Age              From               Message
     ----     ------     ----             ----               -------
     Normal   Scheduled  7s               default-scheduler  Successfully assigned default/image-volume to kind-control-plane
     Normal   Pulled     6s (x2 over 7s)  kubelet            Container image "debian" already present on machine
     Normal   Pulling    6s (x2 over 7s)  kubelet            Pulling image "andraxin/empty"
     Normal   Pulled     6s               kubelet            Successfully pulled image "andraxin/empty" in 876ms (876ms including waiting). Image size: 527 bytes.
     Warning  Failed     6s               kubelet            Error: failed to generate container "5df9cb15cb246fcfff7e2d31088ecb474eb1efc1951196ff1e2e6933a6fd0be2" spec: failed to apply OCI options: failed to mkdir "": mkdir : no such file or directory
     Normal   Pulled     5s               kubelet            Successfully pulled image "andraxin/empty" in 877ms (877ms including waiting). Image size: 527 bytes.
     Warning  Failed     5s               kubelet            Error: failed to generate container "840e2de3a7f0e822295e490aec89c9a3b21f0f4c3d4a1362ded91086bb42dafc" spec: failed to apply OCI options: failed to mkdir "": mkdir : no such file or directory

Anything else we need to know?:

The main container image andraxin/emtpy is, in fact, totally empty.
as it was created by the following one-liner:

echo FROM scratch | docker build -t andraxin/empty -

Environment:

• kind version: (use kind version): kind v0.24.0 go1.23.1 linux/amd64

• Runtime info: (use docker info, podman info or nerdctl info):

  docker info (click to expand)

  $ docker info
  Client:
   Version:    26.1.0
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  0.14.0
      Path:     /usr/libexec/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.20.3
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 3
    Running: 3
    Paused: 0
    Stopped: 0
   Images: 25
   Server Version: 26.1.0
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: true
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
   runc version: 58aa9203c123022138b22cf96540c284876a7910
   init version: de40ad007797e0dcd8b7126f27bb87401d224240
   Security Options:
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.11.0-gentoo
   Operating System: Gentoo Linux
   OSType: linux
   Architecture: x86_64
   CPUs: 16
   Total Memory: 62.57GiB
   Name: meta
   ID: RKOQ:EVBX:DOEO:HTPD:KPRX:JQZ3:3ZWN:BA2O:YDXV:UWBD:FTEP:VDLA
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: andraxin
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false

• OS (e.g. from /etc/os-release): Gentoo

• Kubernetes version: (use kubectl version):

  Client Version: v1.31.0
  Kustomize Version: v5.4.2
  Server Version: v1.31.0
  

• Any proxies or other special environment settings?: No

[aojea]

@saschagrunert do you mind taking a look?

[BenTheElder]

/remove-kind bug
/kind support
Image Volume isn't available yet, it's an alpha API that requires a CRI implementation that has this feature implemented.

The tutorial states:

The container runtime needs to support the image volumes feature
You need to exec commands in the host
You need to be able to exec into pods
You need to enable the ImageVolume feature gate

Last I checked, only cri-o had image volume support yet.

Note that none of this behavior is specific to kind, and any odd behaviors should be filed as bugs to github.com/kubernetes/kubernetes.

[BenTheElder]

See KEP state here: kubernetes/enhancements#4639

containerd support: containerd/containerd#10496