Live migration fails with "qemu-kvm: htab_load() bad index in htab stream (htab_shift=0)"

[balamuruhans]

Description:
Basic Live migration is broken in HostOS 3.0 with "qemu-kvm: htab_load() bad index in htab stream (htab_shift=0)".

Used P9 Boston DD2.0 Host and Fedora 26 Guest:

# cat /proc/cpuinfo | grep revision
revision	: 2.0 (pvr 004e 1200)

Steps to recreate:

1. Have a healthy guest with qcow2 image with shared storage in NFS
2. Mount the image location in source and in destination.
3. Enabled ports 49152:49216 in iptables in destination host,
   # iptables -I INPUT -p tcp -m tcp --dport 49152:49216 -j ACCEPT
4. Permit selinux to allow libvirt for using NFS in source and destination,
   # setsebool virt_use_nfs 1
5. start the guest in source and perform postcopy migration
   # virsh start avocado-vt-vm1-migration

# virsh list
 Id    Name                           State
----------------------------------------------------
 49    avocado-vt-vm1-migration       running

6. Perform migration and issue is observed,

# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.221/system --live --verbose
root@9.40.193.221's password: 
error: internal error: qemu unexpectedly closed the monitor: 2017-09-28T05:20:14.858662Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/1 (label charserial0)
2017-09-28T05:20:15.182815Z qemu-kvm: htab_load() bad index 2113929216 (23554+0 entries) in htab stream (htab_shift=0)
2017-09-28T05:20:15.182851Z qemu-kvm: error while loading state for instance 0x0 of device 'spapr/htab'
2017-09-28T05:20:15.184847Z qemu-kvm: load of migration failed: Invalid argument

Test Environment: (In source and destination host)
Qemu:

# rpm -qa | grep qemu
qemu-img-ev-2.9.0-16.el7_4.5.1.ppc64le
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
qemu-kvm-ev-2.9.0-16.el7_4.5.1.ppc64le
libvirt-daemon-driver-qemu-3.2.0-14.el7_4.3.ppc64le
qemu-kvm-common-ev-2.9.0-16.el7_4.5.1.ppc64le

Libvirt:

# rpm -qa | grep libvirt
libvirt-daemon-driver-storage-core-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-config-nwfilter-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-storage-logical-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-interface-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-network-3.2.0-14.el7_4.3.ppc64le
libvirt-client-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-config-network-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-lxc-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-storage-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-secret-3.2.0-14.el7_4.3.ppc64le
libvirt-python-3.2.0-3.el7.ppc64le
libvirt-devel-3.2.0-14.el7_4.3.ppc64le
libvirt-libs-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-nwfilter-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-qemu-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-storage-disk-3.2.0-14.el7_4.3.ppc64le
libvirt-daemon-driver-nodedev-3.2.0-14.el7_4.3.ppc64le
libvirt-3.2.0-14.el7_4.3.ppc64le

Host kernel:

# uname -a
Linux ltc-boston120.aus.stglabs.ibm.com 4.13.0-4.rel.git49564cb.el7.centos.ppc64le #1 SMP Fri Sep 22 22:49:59 -03 2017 ppc64le ppc64le ppc64le GNU/Linux

Guest (Fedora 26) Kernel:

# uname -a
Linux localhost.localdomain 4.12.13-300.fc26.ppc64le #1 SMP Thu Sep 14 15:47:43 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

cde:info Mirrored with LTC bug #159384 </cde:info>

[balamuruhans]

Attachment: libvirtd debug logs of source and destination:
libvirtd_source.tar.gz
libvirtd_destination.tar.gz

[bssrikanth]

we will try migration with qemu 2.10 will get back here..

[bssrikanth]

We had got in wrong qemu version. With QEMU 2.10 we are not hitting any issues with migration. Closing this bug.