KVM Hook script execution failed: internal error: Child process

[un-fuf-a-doo]

1. First, I probably messed up by trying to install Kubernetes the long way, then deleting/removing/resetting it.
2. Command: minikube start --driver=kvm2

Full output of failed command:

  minikube v1.16.0 on Ubuntu 20.04
✨  Using the kvm2 driver based on user configuration
  Starting control plane node minikube in cluster minikube
  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
  Deleting "minikube" in kvm2 ...
臘  StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=70, Domain=34, Message='Hook script execution failed: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin /etc/libvirt/hooks/qemu minikube prepare begin -) unexpected exit status 1')
  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
  Failed to start kvm2 VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=70, Domain=34, Message='Hook script execution failed: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin /etc/libvirt/hooks/qemu minikube prepare begin -) unexpected exit status 1')

❌  Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=70, Domain=34, Message='Hook script execution failed: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin /etc/libvirt/hooks/qemu minikube prepare begin -) unexpected exit status 1')

  If the above advice does not help, please let us know: 
  https://github.com/kubernetes/minikube/issues/new/choose

As stated in number 1 above, I tried to install Kubernetes the long way instead of keeping it simple by using minikube. Now I think I broke it and need some help getting it to start. My server us a nice and powerful bare metal with lots of storage and lots of ram and lots of cores running Ubuntu 20.04.

Is this something I did? Can it be resolved easily?

Thanks!!

[medyagh]

Thank you for taking the time to create this issue, do you mind providing the output of the following commands ?

virt-host-validate

and also
virsh -c qemu:///system list --all
and aslo

virsh net-list

I am curious have u installed minikube or kvm by snap ? I see some snap path in ur file paths.

btw I wonder do u have docker installed ? have u tried minikube with docker driver?
mind trying this ?

minikube delete 
minikube start --driver=docker

[un-fuf-a-doo]

Absolutely. Glad to help out where I can.

$ virt-host-validate

  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'freezer' controller support                     : PASS
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

$ virsh -c qemu:///system list --all

 Id   Name       State
---------------------------
 1    debian10   running
 -    minikube   shut off

$ virsh net-list

 Name           State    Autostart   Persistent
-------------------------------------------------
 default        active   yes         yes
 minikube-net   active   yes         yes

I installed minikube using the debian installer. I may have used snap to initially install and use microk8s. Which is how I ended up here. Regarding using the docker driver. I have not yet tried the docker driver, but will definitely give it a try now.

Docker driver appears to work without issue:

$ minikube start --driver=docker

  minikube v1.16.0 on Ubuntu 20.04
✨  Using the docker driver based on user configuration
  Starting control plane node minikube in cluster minikube
  Pulling base image ...
  Downloading Kubernetes v1.20.0 preload ...
    > preloaded-images-k8s-v8-v1....: 491.00 MiB / 491.00 MiB  100.00% 24.96 Mi
  Creating docker container (CPUs=2, Memory=32100MB) ...
  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
  Verifying Kubernetes components...
  Enabled addons: storage-provisioner, default-storageclass
  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

[priyawadhwa]

Hey @un-fuf-a-doo thanks for your response! Glad the docker driver is working out.

I'm not sure why kvm2 isn't, since the output you provided seems correct. Does a delete and start fix it?

minikube delete
minikube start --driver kvm2

[un-fuf-a-doo]

Hey. No it still fails, the full output with verbosity enabled is below. I do already have a VM running under KVM, nothing special, just a debian 10 OS running Graylog for log collection. Could libvirt be interfering? Or might this be as simple as a permissions issue?

  minikube v1.16.0 on Ubuntu 20.04
✨  Using the kvm2 driver based on user configuration
  Downloading driver docker-machine-driver-kvm2:
    > docker-machine-driver-kvm2....: 65 B / 65 B [----------] 100.00% ? p/s 0s
    > docker-machine-driver-kvm2: 13.52 MiB / 13.52 MiB  100.00% 164.12 KiB p/s
  Downloading VM boot image ...
    > minikube-v1.16.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
    > minikube-v1.16.0.iso: 212.62 MiB / 212.62 MiB [ 100.00% 19.87 MiB p/s 11s
  Starting control plane node minikube in cluster minikube
  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
  Deleting "minikube" in kvm2 ...
臘  StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: creating domain: error defining 
domain xml:
<domain type='kvm'>
  <name>minikube</name>
  <memory unit='MiB'>6000</memory>
  <vcpu>2</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>

  </features>   
  <cpu mode='host-passthrough'/>
  <os>
    <type>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
    <bootmenu enable='no'/>
  </os>
  <devices>
    <disk type='file' device='cdrom'>
      <source file='/home/jpop/.minikube/machines/minikube/boot2docker.iso'/>
      <target dev='hdc' bus='scsi'/>
      <readonly/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='default' io='threads' />
      <source file='/home/jpop/.minikube/machines/minikube/minikube.rawdisk'/>
      <target dev='hda' bus='virtio'/>
    </disk>
    <interface type='network'>
      <source network='default'/>
      <mac address='d4:37:e9:d7:8c:5b'/>
      <model type='virtio'/>
    </interface>
    <interface type='network'>
      <source network='minikube-net'/>
      <mac address='8c:48:aa:37:0c:c6'/>
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>   
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>  
    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
    </rng>

  </devices>
</domain>
: virError(Code=9, Domain=20, Message='operation failed: domain 'minikube' already exists with uuid cf30d554-4bae-43f9-a0ee-40a2f5013912')
  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
  Failed to start kvm2 VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=70, Domain=34, Message='Hook script execution failed: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin /etc/libvirt/hooks/qemu minikube prepare begin -) unexpected exit status 1')

❌  Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=70, Domain=34, Message='Hook script execution failed: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin /etc/libvirt/hooks/qemu minikube prepare begin -) unexpected exit status 1')

  If the above advice does not help, please let us know:
  https://github.com/kubernetes/minikube/issues/new/choose

[medyagh]

Thank you for taking the time to create this issue,

This does seem to be a failure in minikube to create KVM domain network

do you mind providing the output of the following commands ?

virt-host-validate

and also
virsh -c qemu:///system list --all
and aslo

virsh net-list

@prezha you have looked at the KVM issues before I wonder if you have any insight on this ?

[prezha]

@medyagh sure, i can try to help

@un-fuf-a-doo couple of things to check first please:

• in your reply above:
  $ virsh -c qemu:///system list --all and $ virsh net-list => $ indicates that you might have run this command as a plane user - ie, w/o using sudo/su? if that is the case, then i'd expect to see just empty lists there unless your user's permissions are changed?
• is your user part of the libvirt group (eg, run groups command as that user)?

---

after we clarify that, here are additional steps we can take:

• according to the last output, there's error: virError(Code=9, Domain=20, Message='operation failed: domain 'minikube' already exists with uuid cf30d554-4bae-43f9-a0ee-40a2f5013912') => the previous kvm minikube domain was not deleted - please try again with minikube delete as user, and then confirm with virsh list --all and virsh net-list - both as root (and paste here the output)
• can you replicate the issue with the latest minikube (currently 1.17.0)?
• as user try running minikube start --driver=kvm2 --alsologtostderr -v=99 for very verbose output and please share it here

[medyagh]

@un-fuf-a-doo are you still having this issue, do u have chance to provide more information as suggested by @prezha #10040 (comment)

and also did you try deleting minikube as suggested by priyawadhwa