Smart Clone not possible with Windows disk

[nick123pig]

What happened:

When attempting a CSI snapshot clone with a windows disk, I am unable to clone a virtual machine. Host-assisted clone still works fine. The mount command is what fails.

Events:
  Type     Reason       Age                  From     Message
  ----     ------       ----                 ----     -------
  Warning  FailedMount  71s (x538 over 18h)  kubelet  MountVolume.MountDevice failed for volume "pvc-5cc66ec2-9c7d-438e-a2f1-fd753b1c8427" : rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-5cc66ec2-9c7d-438e-a2f1-fd753b1c8427 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/f98632dd5d308fa99779cd6b9f3055766314ae829e84fc801c0c9493b1d62eab/globalmount
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/f98632dd5d308fa99779cd6b9f3055766314ae829e84fc801c0c9493b1d62eab/globalmount: wrong fs type, bad option, bad superblock on /dev/longhorn/pvc-5cc66ec2-9c7d-438e-a2f1-fd753b1c8427, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

What you expected to happen:

I would expect it to be able to smart clone the disk

How to reproduce it (as minimally and precisely as possible):

1. Create a windows VM
2. Clone it using a basic volume snapshot smart clone

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: windows-vm-test-snap
  namespace: csi-snapshot-test
spec:
  volumeSnapshotClassName: longhorn-snapshot-vsc
  source:
    persistentVolumeClaimName: windows-vm-test

3. Create a new vm with the snapshot source

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: windows-vm-test-snap
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/domain: windows-vm-test-snap
    spec:
      domain:
        firmware:
          bootloader:
            efi: 
              secureBoot: false
        features:
          acpi: {}
          apic: {}
          smm: 
            enabled: true
          hyperv:
            relaxed: {}
            spinlocks:
              spinlocks: 8191
            vapic: {}
        clock:
          timer:
            hpet:
              present: false
            hyperv: {}
            pit:
              tickPolicy: delay
            rtc:
              tickPolicy: catchup
          utc: {}
        cpu:
          cores: 2
          threads: 2
          sockets: 1      
        devices:
          tpm: {}
          inputs:
            - type: tablet
              bus: virtio
          disks:
            - name: rootdisk
              disk:
                bus: scsi
          interfaces:
            - name: default
              masquerade: {}
              ports:
              - name: ssh
                port: 22
        resources:
          requests:
            memory: 4096M
      networks:
        - name: default
          pod: {} 
      volumes:
        - name: rootdisk
          dataVolume:
            name:  windows-vm-test-snap
  dataVolumeTemplates:
  - metadata:
      name:  windows-vm-test-snap
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: "30064771072"
      source:
        snapshot:
          namespace: csi-snapshot-test
          name: windows-vm-test-snap

4. Observe the prep pod and observe it fails to mount

Additional context:

Here's what the fdisk looks like on the host with the snapshot

# fdisk -l  /dev/longhorn/pvc-5cc66ec2-9c7d-438e-a2f1-fd753b1c8427
The backup GPT table is not on the end of the device.
Disk /dev/longhorn/pvc-5cc66ec2-9c7d-438e-a2f1-fd753b1c8427: 28 GiB, 30064771072 bytes, 58720256 sectors
Disk model: VIRTUAL-DISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 79A4F984-E96F-4C3A-B7D8-378CE1DB90B6

Device                                                     Start      End  Sectors  Size Type
/dev/longhorn/pvc-5cc66ec2-9c7d-438e-a2f1-fd753b1c8427p1    2048  1026047  1024000  500M Windows recovery environment
/dev/longhorn/pvc-5cc66ec2-9c7d-438e-a2f1-fd753b1c8427p2 1026048  1230847   204800  100M EFI System
/dev/longhorn/pvc-5cc66ec2-9c7d-438e-a2f1-fd753b1c8427p3 1230848  1263615    32768   16M Microsoft reserved
/dev/longhorn/pvc-5cc66ec2-9c7d-438e-a2f1-fd753b1c8427p4 1263616 52426751 51163136 24.4G Microsoft basic data

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): v1.61.0
• Kubernetes version (use kubectl version): v1.31.4+k3s1
• DV specification: attached

kind: DataVolume
metadata:
  annotations:
    cdi.kubevirt.io/storage.usePopulator: "true"
  creationTimestamp: "2025-01-28T20:50:12Z"
  generation: 1
  name: windows-vm-test
  namespace: csi-snapshot-test
  resourceVersion: "7321782"
  uid: 836eff07-80fb-40df-9f40-1cca9d2d8e73
spec:
  contentType: kubevirt
  source:
    upload: {}
  storage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: "26843545600"
    storageClassName: nvme-3
status:
  claimName: windows-vm-test
  conditions:
  - lastHeartbeatTime: "2025-01-28T21:01:37Z"
    lastTransitionTime: "2025-01-28T21:01:37Z"
    message: PVC windows-vm-test Bound
    reason: Bound
    status: "True"
    type: Bound
  - lastHeartbeatTime: "2025-01-28T21:01:37Z"
    lastTransitionTime: "2025-01-28T21:01:37Z"
    status: "True"
    type: Ready
  - lastHeartbeatTime: "2025-01-28T21:01:35Z"
    lastTransitionTime: "2025-01-28T21:01:35Z"
    message: Upload Complete
    reason: Completed
    status: "False"
    type: Running
  phase: Succeeded
  progress: N/A 

• Cloud provider or hardware configuration: Bare Metal
• OS (e.g. from /etc/os-release): Fedora CoreOS 41.20241215.3.0
• Kernel (e.g. uname -a): Linux localhost 6.11.11-300.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Dec 5 18:38:25 UTC 2024 x86_64 GNU/Linux
• Install tools: N/A
• Others: Longhorn

[akalenyu]

dataVolumeTemplates:

• metadata:
  name: windows-vm-test-snap
  spec:
  pvc:

what if you use spec.storage instead of spec.pvc here?
(let CDI guess the right volume/access mode)

I suspect you're restoring a block snapshot into a filesystem volume, and that is why it fails

[nick123pig]

Good pointer! spec.pvc also offers volumeMode: Block, so adding that allowed things to proceed. Thank you!

[akalenyu]

As an RFE, I think we can add some validation in CDI for this post 1.29, where snapshotcontents have the source volume mode field

/reopen

[kubevirt-bot]

@akalenyu: Reopened this issue.

In response to this:

As an RFE, I think we can add some validation in CDI for this post 1.29, where snapshotcontents have the source volume mode field

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.