Qemu VM failed to start with ignition configured

[kopwei]

Version Reports:

Distro version of host:

Ubuntu 16.04/16.10/17.04

Terraform Version Report

v0.10

Libvirt version

2.1.0

terraform-provider-libvirt plugin version (git-hash)

8b8c66e

---

Description of Issue/Question

Qemu failed to start with ignition configured.
Error log:
Error applying plan:

1 error(s) occurred:

• libvirt_domain.coreos: 1 error(s) occurred:

• libvirt_domain.coreos: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: warning: host doesn't support requested feature: CP
  UID.80000001H:ECX.svm [bit 2]
  2017-06-28T18:56:40.509521Z qemu-system-x86_64: -fw_cfg name=opt/com.coreos/config,file=/var/lib/libvirt/images/cluster.ign: can't load /var/lib/libvirt/images/cluster.ign')

Setup

tf file (in txt mode) is attached
libvirt.tf.txt

Steps to Reproduce Issue

terraform init
terraform plan
terraform apply

[monstermunchkin]

Is this still an issue? I cannot reproduce this issue on openSUSE Tumbleweed with libvirt 3.5.0. There's coreos/bugs#2083 which might be related.

[monstermunchkin]

Closing this due to no response. Feel free to reopen the issue if needed.

[wking]

I think this is a SELinux issue. For example:

$ cat main.tf 
provider "libvirt" {
  uri = "qemu:///system"
}

data "ignition_systemd_unit" "example" {
  name = "example.service"
  content = "[Service]\nType=oneshot\nExecStart=/usr/bin/echo Hello World\n\n[Install]\nWantedBy=multi-user.target"
}

data "ignition_config" "etcd" {
  systemd = [
      "${data.ignition_systemd_unit.example.id}",
  ]
}

resource "libvirt_ignition" "etcd" {
  name = "etcd.ign"
  content = "${data.ignition_config.etcd.rendered}"
}

resource "libvirt_volume" "base" {
  name   = "base"
  source = "file:///tmp/coreos_production_qemu_image.img"
}

resource "libvirt_volume" "etcd" {
  name = "etcd"
  base_volume_id = "${libvirt_volume.base.id}"
}

resource "libvirt_domain" "etcd" {
  name = "etcd"
  memory = "1024"
  coreos_ignition = "${libvirt_ignition.etcd.id}"

  disk {
    volume_id = "${libvirt_volume.etcd.id}"
  }
}
$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31
$ terraform init
$ terraform apply
...
libvirt_ignition.etcd: Creation complete after 1s (ID: /home/trking/VirtualMachines/etcd.ign;5b5f96c6-c821-3584-1a25-5c544a2b87b3)
libvirt_volume.etcd: Creation complete after 1s (ID: /home/trking/VirtualMachines/etcd)
libvirt_domain.etcd: Creating...
  arch:             "" => "<computed>"
  coreos_ignition:  "" => "/home/trking/VirtualMachines/etcd.ign;5b5f96c6-c821-3584-1a25-5c544a2b87b3"
  disk.#:           "" => "1"
  disk.0.scsi:      "" => "false"
  disk.0.volume_id: "" => "/home/trking/VirtualMachines/etcd"
  emulator:         "" => "<computed>"
  machine:          "" => "<computed>"
  memory:           "" => "1024"
  name:             "" => "etcd"
  running:          "" => "true"
  vcpu:             "" => "1"

Error: Error applying plan:

1 error(s) occurred:

* libvirt_domain.etcd: 1 error(s) occurred:

* libvirt_domain.etcd: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2018-07-30T22:52:54.865806Z qemu-kvm: -fw_cfg name=opt/com.coreos/config,file=/home/trking/VirtualMachines/etcd.ign: can't load /home/trking/VirtualMachines/etcd.ign')
...
$ ls -lZ ~/VirtualMachines
-rw-r--r--. qemu qemu system_u:object_r:virt_content_t:s0 base
-rw-r--r--. root root system_u:object_r:virt_image_t:s0 etcd
-rw-r--r--. root root system_u:object_r:virt_image_t:s0 etcd.ign
$ terraform destroy
$ sudo setenforce 0
$ terraform apply
...
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Details on the violation (from sealert -l 9e4d48aa-f066-464c-a7eb-237bd48a3fe5):

SELinux is preventing /usr/libexec/qemu-kvm from read access on the file etcd.ign.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that qemu-kvm should be allowed read access on the etcd.ign file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'qemu-kvm' --raw | audit2allow -M my-qemukvm
# semodule -i my-qemukvm.pp


Additional Information:
Source Context                system_u:system_r:svirt_t:s0:c117,c825
Target Context                system_u:object_r:virt_image_t:s0
Target Objects                etcd.ign [ file ]
Source                        qemu-kvm
Source Path                   /usr/libexec/qemu-kvm
Port                          <Unknown>
Host                          trking.remote.csb
Source RPM Packages           qemu-kvm-rhev-2.9.0-16.el7_4.13.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.13.1-192.el7_5.4.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Permissive
Host Name                     trking.remote.csb
Platform                      Linux trking.remote.csb 3.10.0-891.el7.x86_64 #1
                              SMP Mon May 21 14:10:11 EDT 2018 x86_64 x86_64
Alert Count                   1
First Seen                    2018-07-30 15:58:30 PDT
Last Seen                     2018-07-30 15:58:30 PDT
Local ID                      9e4d48aa-f066-464c-a7eb-237bd48a3fe5

Raw Audit Messages
type=AVC msg=audit(1532991510.321:8799): avc:  denied  { read } for  pid=19853 comm="qemu-kvm" name="etcd.ign" dev="dm-3" ino=3673906 scontext=system_u:system_r:svirt_t:s0:c117,c825 tcontext=system_u:object_r:virt_image_t:s0 tclass=file permissive=1


type=AVC msg=audit(1532991510.321:8799): avc:  denied  { open } for  pid=19853 comm="qemu-kvm" path="/home/trking/VirtualMachines/etcd.ign" dev="dm-3" ino=3673906 scontext=system_u:system_r:svirt_t:s0:c117,c825 tcontext=system_u:object_r:virt_image_t:s0 tclass=file permissive=1


type=SYSCALL msg=audit(1532991510.321:8799): arch=x86_64 syscall=open success=yes exit=EISDIR a0=560a07eee6c0 a1=0 a2=7fff02afe390 a3=7fa4affcb950 items=0 ppid=1 pid=19853 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm=qemu-kvm exe=/usr/libexec/qemu-kvm subj=system_u:system_r:svirt_t:s0:c117,c825 key=(null)

Hash: qemu-kvm,svirt_t,virt_image_t,file,read

This is with terraform-provider-libvirt built from d29a7cc.

[cgwalters]

I personally have /srv/libvirt/images-gold which is a set of immutable template images that can be used as backing files; they're all virt_image_t:s0 like the main libvirt images. Basically don't try to store them in your home directory.

[mmersin]

I also faced this issue, with v0.5.1 version on CentOS 7.6.

terraform apply failed with error below

* libvirt_domain.p03_domain: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2019-01-03T18:45:46.740404Z qemu-kvm: -fw_cfg name=opt/com.coreos/config,file=/mnt/vm/images/p03_ignition: can't load /mnt/vm/images/p03_ignition')

after setting SELinux mode to permissive, it worked.

For reference,

• p03_disk is volume created with libvirt_volume resource
• p03_ignition.ign is ignition file created by libvirt_domain resource, through libvirt_ignition. type shows as unlabeled_t.

$  ls -Z /mnt/vm/images/
-rw-r--r--. qemu qemu system_u:object_r:svirt_image_t:s0:c273,c374 p03_disk
-rw-r--r--. root root system_u:object_r:unlabeled_t:s0 p03_ignition.ign