[Download] Pre-built GVT+Dmabuf QEMU package for Archlinux

[XaeroVincent]

Update: Please use Linux 4.16rc2 or newer kernels to have the kernel-side GVT + Dmabuf components. A patched QEMU is still needed.

I went ahead and packaged up QEMU 2.10 with GVT-g and Dmabuf support that I compiled on my system. The QEMU package is hosted on my OneDrive account. This package is only designed for use with Archlinux and derivative distributions and can be installed with 'pacman -U'. Be sure to follow the applicable parts of the official guide and add the proper qemu and kernel boot options for VFIO GPU passthrough to work.

Download (Built from 'stable-2.10.0' git as of 2/20/2018):
https://1drv.ms/u/s!Am9uPEIZtbF9g1vFKo3-yNuiU8Vb

Optional: Source code

[DocMAX]

PERFECT!!! Thank you very much my friend!!! Will try later on a 3rd gen X1 Carbon ThinkPad (Broadwell) and Report back. Can you try a Q35 machine Setup with OVMF to see if it works?

Perhaps we can add gvt kernel + gvt qemu in the AUR repo? Wondering why there isn't allready one.

[ghost]

That's great. Will try and report how it worked soon.

[DocMAX]

I get:

(123..) ioctl VFIO_DEVICE_QUERY_GFX_PLANE(primary): Invalid arument

On QEMU display i see "Guest has not initialized the display (yet)

[XaeroVincent]

DocMAX, Qemu will say that until the Windows guest is booted and the Intel guest driver is initialized.

I ended up using '-vga qxl' to install the quest and get everything up and running first. It's also a good idea to use Display Driver Uninstaller in safe mode to remove the default installed Windows Intel GFX drivers, which can cause BSOD in the guest with VFIO.

https://www.wagnardsoft.com/

After you install the official 15.45 or 15.65 Intel graphics drivers from Intels website, then you can change it back to '-vga none'. Be sure to shutdown the guest before making changes, of course.

[DocMAX]

XaeroVincent, can you please check if Q35 and OVMF works on your system?

[XaeroVincent]

Q35 machine profile works but re-installing the guest with OVMF/Tianocore UEFI seems to break the Dmabuf support. I guess we'll just have to use the default QEMU SeaBIOS with Q35/ICH9 or the default I440FX chipset emulation until that's fixed? Maybe one of the Intel devs can investigate this.

BTW, if you are having garbled /staticky sound, if using the HDA sound device in QEMU (when using the pulseaudio sound driver), setting the Windows playback device sample rate to 44100 Hz seems to clear it up. I think it's because default pulseaudio server sample rate on Linux is set to 44100 Hz.

[DocMAX]

I'm trying to run macOS with GVT-g. At the moment i get this error. maybe someone tries to accomplish the same and share findings.

[IGPU] Graphics driver failed to load:could not register with Framebuffer driver

[XaeroVincent]

Well in addition to the host Linux kernel and QEMU needing modifications for GVT, the guest graphics drivers also need modifications to support GVT. It has been mentioned by a dev on here that the Linux and Windows drivers have said support. However, there is no reason to believe that the built-in macOS graphics drivers also contain this support too, since IIRC, Apple's EULA only permits macOS virtualization guests on macOS hosts, running on Macintosh systems and this project's scope doesn't seem to encompass macOS host use cases.

[oscarbg]

@XaeroVincent some questions:

1)Phoronix says all needed kernel bits are in 4.16 (in 4.16rc1 already?).. it's true?
I installed it on ubuntu form kernel ppa (http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc1/)
in case yes.. 4.16 kernel has to have any non default kernel option enabled at build?
I say because I can see in /boot/config-4.16.0-041600rc1-generic file if Ubuntu is enabling that option..

regarding Qemu phoronix says it will probably be upstreamed in 2.12 already..
anybody seeing this thread can say if it's expected to be true, i.e. that's the plan?
if yes then can probably wait a month for first RC builds and non build custom Qemu branch..

2. any interest on doing the same for Ubuntu.. ie. provide precompiled packages via .deb files or setup a PPA?

3. have you tested some modern Android x86 ISO (ex: http://www.android-x86.org/releases/releasenote-7-1-r1) to see if all this works also in addition to Linux and Windows..
   it should work as they ship with modern Linux kernel and Mesa drivers underneath..
   7.1r1 ships with:
   LTS kernel 4.9.80
   notes from older 7.1rc1:
   *Support OpenGL ES 3.x hardware acceleration for Intel/AMD/Nvidia, VMware and QEMU(virgl) by Mesa 17.1.2.
   from 7.1rc2:
   *Improve QEMU virgl stability.
   *Update Mesa to 17.1.10

if that not recent enough you can test Android 8.1 (Oreo) experimental builds provided by @maurossi here:
https://drive.google.com/drive/folders/0B_OFHiIqgpSFTFpkQWc1eXV3ME0
which ship with latest kernel 4.16rc1+Mesa18.1dev using LLVM6.0 for amdgpu..

please someone share findings of success using Android x86 as guest..

[oscarbg]

Thanks for all info..
Sadly anbox seems to have issues on Ubuntu.. tried various times within a year and never worked for me or much other people.. anbox github issues site is full of people on recent Ubuntu versions having anbox dont’ working..
Also sad is Android x86 ISO’s have opengl es 3.2 support so very recent apps work.. and the latest Oreo iso even has vulkan Anvil support albeit don’t working for many apps only vulkancapsviwer.. that can’t be done very fast by anbox people.. even Android emulator started recently having gles 3.0 support.. and last also even beignet Intel opencl driver support Android although no Android x86 ISOs currently include it in future could be.. there are some apps using in on store.. finally even hardware accel video decding via va API (and encoding?) is included on Isos.. so Intel must support it.. now I remember someone sometime ago used gvt on Andrroid x86 so maybe what you are referring need X.org conf is for new dma_buf functionality we are talking after all, right?

[oscarbg]

I just opened an issue:
intel/Igvtg-qemu#3
asking if QEMU GVT dma_buf support will be ready for QEMU 2.12..

[newperson1746]

I have macOS High Sierra as well (real mac, actually on ubuntu 16.04.3) it is a macbookair with hd 6000 (bdw) graphics. Since the dma-buf doesn't seem to work (for me, it just says cursor plane not initialized by guest) I have -serial stdio with tianocore uefi, which allows me to navigate clover bootloader via the serial port! Then I just wait a long time until macos boots, and I get this: (via port forwarding 5900 to 5800 for screen sharing host-side) since the dma-buf doesn't work, I just use macos screen sharing. As you can see, the bdw kexts are loaded, but, just like a real headless mac, unless there is an edid detected on a port, qe/ci still gets disabled. We are really close here, all that needs to happen is: a) get the dma-buf to work with tianocore uefi, or b) we get clover to fake a display. b) is actually possible, we can inject an edid (meant for laptops with monitor issues) which will trick appleintelbdwgraphics into enabling acceleration, and we screen-share in until dma-buf works wiith uefi. Note: this is all on real macbookair7,2...

[DocMAX]

Great! Keep me posted on any updates! Can't wait to see dma-buf working stable on OSX guest!

[newperson1746]

I'm gonna try InjectEDID and CustomEDID to see if I can at least trick macOS into acceleration remotely. I know, I can't wait either. This will be the first time a virtualized macOS has qe/ci without dedicating a vfio-pci gpu for it.

[newperson1746]

Also, while I try and debug macOS, to get a serial console: add -serial telnet:127.0.0.1:4444,server,nowait to qemu, and macos boot-args="debug=0x8 console=3" will give you a console on serial port on macos.

[newperson1746]

I posted an issue about dma-buf not working (black screen) on ubuntu. If I had arch I could be testing this right now, but I am out of storage.

[ozeidan]

Hi guys, could anybody explain how to use this from virt-manager?
I added to my windows10.xml:

<qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:02.0/6f8c2453-2896-4cb3-82ce-c2bbdcb943de,rombar=0'/>
</qemu:commandline>

It fails with errormessage:

qemu-system-x86_64: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:02.0/6f8c2453-2896-4cb3-82ce-c2bbdcb943de,rombar=0: vfio error: 6f8c2453-2896-4cb3-82ce-c2bbdcb943de: failed to open /dev/vfio/0: Operation not permitted

When starting with qemu directly, it seems to start fine. But I am too dumb to setup the networking in qemu. (Or maybe it doesn't work on my machine because libvirt already set up some networking stuff.) I would really appreciate help on this. Thanks

[DocMAX]

try to add a udev rule like this: SUBSYSTEM=="vfio", OWNER="root", GROUP="kvm"

[XaeroVincent]

Newperson1746 ,

That's impressive Newperson1746! I stand corrected.
Dma-Buf doesn't appear to support QEMU UEFI yet, only SeaBIOS.

Perhaps one thing you can try is to patch your macOS guest to run on BIOS+MBR based partitions then macOS can boot with QEMU using SeaBIOS?

http://www.insanelymac.com/forum/files/file/762-high-sierra-mbr-and-firmware-check-patch/?st=50#commentsStart

[newperson1746]

I will try that. However, even with seabios, this does not work. I posted all the info on a new issue, so that it can be better organized.

[newperson1746]

Ok. I stand corrected myself. With a copy of Android Oreo, the virtual gpu works fine in both UEFI grubx64.efi with OVMF and the regular isolinux (i think.) What happens is that i915_drm (I think) is loaded way early in boot, therefore allowing me to see the linux kernel booting. In macOS however, the driver gets loaded but doesn't know what to do beyond that, it just sees an HD 6000 without any display outputs, and disables OpenGL (same issue as headless macs, google it.) In early boot, macOS utilizes the EFI framebuffer console to output text/graphics, not the gpu-specific driver, as apple can expect its hardware to have the efi framebuffer available. Apple_ would need to explicitly support dma-buf in their drivers too, which is a virtually unrealistic expectation of them. I don't know whether or not intel compiles or has access to the source code for "GPUDrivers-Intel" (internal name in binary kext, I used grep to find it). If they do, I hope they can add support, because as of right now, it does not work (unless on the qemu-side of things the developers implement the dma-buf display output as a standard HDMI/DP/VGA output with an EDID attached, as that is what the macOS drivers are expecting/checking for) If somehow we can make the VGPU available as a standard display output that is then dma-buffed to qemu, then acceleration will work. If we can trick macOS into thinking a display is there and enabling OpenGL, we can screenshare into macOS and access the GPU as well, just like the old way in Windows (before dma-buf worked.) So there's three paths:

1. Add an option for dma-buf to appear as a standard display output with a standard EDID attached to a "physical" display output port, thereby not only allowing macOS/non-specifically fixed drivers to use gvtg, but also allowing UEFI/BIOS POST screens to work, as then gvtg would then support the basic EFI framebuffer/vesa modes, or...
2. Fix the macOS intel gpu drivers themselves, which is the way it is currently done on linux and windows. This is why you have to wait for the OS drivers to initialize. Because the drivers themselves handle output to dma-buf, not qemu. (So dma-buf is a special output mode, not masquerading as an HDMI out or similar) However I do not think Apple would be willing to do this, considering their past history. lastly...
3. Trick macOS (and drivers) into enabling OpenGL for the headless virtual display that is there for screensharing purposes. By default if macOS sees no physical displays then it creates a virtual framebuffer with a resolution of 1280x1024, but disables OpenGL, thereby making the menubar lose its translucency and the dock lose its transparency effects. (I can currently get that far, macOS boots with HD 6000 graphics loaded and drivers loaded, but we have the "headless mac" issue that occurs with real macs if headless, except that we cannot use a headless fake display HDMI adapter or etc) If we can trick macOS into thinking there's a physical display attached, we can then screen-share into macOS itself and access full QE/CI. (Even if we can't directly see the "tricked" display's output, we can access its framebuffer which does have QE/CI enabled)
   Hope that clarifies things. I am not even a developer, so this is all gathered from observation, however I'm sure a quick glance at i915 code would either prove or disprove this.

[oscarbg]

@newperson1746
are you telling me that Oreo ISO works out the box? In LIVE MODE also (without installing to a virtual disk)? if yes the we have good news for today.. also what ISO? are using the experimental ISO "oreo_x86_64_k416rc1_amd_dc_llvm60_mesa-18.1.0devel_vulkan.iso" I linked in earlier post?

[newperson1746]

Yes, sir. Exactly that one, if booted live, or installed, will work OOB. I will provide a screenshot, one sec.

[oscarbg]

@newperson1746 great!!!
can I please ask to include a screenshot of VulkanCapsViewer app from Play store (https://play.google.com/store/apps/details?id=de.saschawillems.vulkancapsviewer) to see Vulkan driver from this ISO is working correctly under GVT-g and also one of either OpenGL extensions Viewer app or
https://play.google.com/store/apps/details?id=de.saschawillems.glescapsviewer to see identifies corerctly as Mesa 18.1dev with OpenGL ES 3.2 support..

[newperson1746]

sure one sec. screenshot:

(oops 1920x1200 framebuffer is larger than my little laptop screen)

[oscarbg]

@newperson1746 I'm not fluent enough in all this tech.. may I ask you to provide complete qemu arguments list to test that ISO is working on my system with Ubuntu 18.04dev+ 4.16rc1 +custom built QEMU+dma_buf by myself?

[newperson1746]

I switched out to the 1024x768 one. I'm setting up. One sec please.

[newperson1746]

opengles extensions: right here
vulkan:



(vulkan crashed when submitting report)

[oscarbg]

@newperson1746 thanks for being so fast.. note vulkan support is very early and even on native mode crashes sending submitting report.. it's says something about extensions string parsing right?

[newperson1746]

I am installing it, upgrading my 7.0 install so I can use vgpu. just a few minutes, I apologize for the delay. (But i want to be able to save apps i installed, not go away on reboot)

[memegauste]

I have similar issue here:
#48

Also I installed Intel driver inside, but was messing with monitor settings and... this happened.
Probably vfio-pci is not working at all, since it gets detected (what other device as display would be detected and gave black screen on GTK display?), so I can't do anything on my vm.

Screenshot on qxl-vga:

Screenshot on vfio-pci:

My config:

#!/bin/sh

# QEMU
qemu-system-x86_64 \
        -enable-kvm \
        -m 4G \
        -smp cores=2,threads=1,sockets=1,maxcpus=2 \
        -machine type=pc,accel=kvm,kernel_irqchip=on \
        -global PIIX4_PM.disable_s3=1 \
        -global PIIX4_PM.disable_s4=1 \
        -name windows10-gvt-g \
        -soundhw ac97 \
        -usb \
        -device usb-tablet \
        -display gtk,gl=on \
        -device vfio-pci,sysfsdev=/sys/devices/pci0000:00/0000:00:02.0/56fdd4da-ade1-11e8-98d0-529269fb1459,x-igd-opregion=on,display=on \
        -drive file=/home/hoshi/windows10/win10.qcow2,format=qcow2,l2-cache-size=8M \
        -cdrom /home/hoshi/windows10/win_install.iso \
        -vga qxl \
        -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd \
        -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/OVMF_VARS.fd

[joyqing]

Just show the VM MacOS render HW acceleration

It's not gvt-g. It's directly iGPU passthrough like RX580. Specs below:
Asrock Z370 Pro4, i7 8700k, rx580, 2080ti, Manjaro kde kernel 5.2, qemu 4.0.0-2, libvirt 5.4.0-1.
pass through rx580+UHD630+SATA. Add -disablefxframware in Clover. SMBios must be iMac pro 1.1. FCPX is 10.4.6 works as same as the host. Only one MacOS partition in HD.Either VM or Host can all boot from the same MacOS par.
Issue: VM POST needs very long time.
This is not fully solution because iGPU doesn't work just acts as "headless". GVT-g is the future. Hope someone can open a new thread to discuss iGPU passthrough for MacOS.

[jembhz]

Ey, how did you achieve that, I'm struggling with Proxmox (Based on QEMU/KVM) to get that done, to put the iGPU as headless, I have a RX580 as well and everything is working so far on MacOS Mojave but iGPU passthrouhg as headless, when I try it MacOS reboots forever, if you can give me some advice I will really appreciate!

[oscarbg]

@joyqing just noticed the "gpustat" MacOS command line tool you use which seems to report GPU clocks, utlization,etc.. I can't find anywhere.. it's custom made? can share source code or which Mac APIs you use to query such information from your apps? thanks..

[mateHD]

I have macOS High Sierra as well (real mac, actually on ubuntu 16.04.3) it is a macbookair with hd 6000 (bdw) graphics. Since the dma-buf doesn't seem to work (for me, it just says cursor plane not initialized by guest) I have -serial stdio with tianocore uefi, which allows me to navigate clover bootloader via the serial port! Then I just wait a long time until macos boots, and I get this: (via port forwarding 5900 to 5800 for screen sharing host-side) since the dma-buf doesn't work, I just use macos screen sharing. As you can see, the bdw kexts are loaded, but, just like a real headless mac, unless there is an edid detected on a port, qe/ci still gets disabled. We are really close here, all that needs to happen is: a) get the dma-buf to work with tianocore uefi, or b) we get clover to fake a display. b) is actually possible, we can inject an edid (meant for laptops with monitor issues) which will trick appleintelbdwgraphics into enabling acceleration, and we screen-share in until dma-buf works wiith uefi. Note: this is all on real macbookair7,2...

Hi,
Do you have any news about this project? Any success?
Thanks

[trevor403]

This issue is linked in Arch Wiki, the have some options for using OVMF UEFI to work for guests. This would enable Clover to use the OpRom for gvt https://wiki.archlinux.org/index.php/Intel_GVT-g#Using_DMA-BUF_with_UEFI/OVMF

Note that DMA-BUF will never work on macOS unless a (fairly complex) kext patch were to be developed. Less likely would be official GVT DMA-BUF support distributed by Apple. However I would love to see EDID faking to gain headless HW acceleration.

I hope Clover can help make this possible!

[danwdart]

Is this mainlined in 4.2 or 5.0?

[scorpion81]

Hi everyone,

With the help of a locally modified version of https://github.com/patmagauran/i915ovmfPkg i had partial success with GVT-g and macOS in qemu. The VM boots up and you can see the Guest IGPU displayed in MacOS BUT... if you attempt to load the (in my case) KabyLake or CoffeeLake apple drivers, booting takes very long and there are a few error messages in qemu, the host linux's dmesg (Manjaro with kernel 5.10.89) and the boot gets stuck when trying to access the frame buffer.

See here for more details and the mentioned error messages:

acidanthera/bugtracker#1914 (comment)

The relevant bits here are (dmesg):

[ 2466.228805] i915 0000:00:02.0: drm_WARN_ON(bytes > 2)
[ 2466.228844] WARNING: CPU: 4 PID: 22460 at drivers/gpu/drm/i915/gvt/cfg_space.c:325 intel_vgpu_emulate_cfg_write+0x3a3/0x4c0 [i915] 

--snip, see the link above for the stacktrace of that warning--

[ 2466.288841] gvt: vgpu(1) Invalid FORCE_NONPRIV write 7034 at offset 24d8
[ 2466.288846] gvt: vgpu(1) Invalid FORCE_NONPRIV write 7008 at offset 24dc
[ 2466.288856] gvt: vgpu 1: write invalid HWSP address, reg:0x2080, value:0x402a4000
[ 2466.288857] gvt: vgpu 1: fail to emulate MMIO write 00002080 len 4
[ 2466.288879] gvt: vgpu 1: write invalid HWSP address, reg:0x12080, value:0x402a5000
[ 2466.288879] gvt: vgpu 1: fail to emulate MMIO write 00012080 len 4
[ 2466.288909] gvt: vgpu 1: write invalid HWSP address, reg:0x22080, value:0x402a6000
[ 2466.288909] gvt: vgpu 1: fail to emulate MMIO write 00022080 len 4
[ 2466.288918] gvt: vgpu 1: write invalid HWSP address, reg:0x1a080, value:0x402a7000
[ 2466.288918] gvt: vgpu 1: fail to emulate MMIO write 0001a080 len 4
[ 2466.292585] gvt: vgpu 1: invalid mm type: 0 gma 402aa000
[ 2466.292586] gvt: vgpu 1: invalid guest context LRCA: 402a9
[ 2466.292587] gvt: vgpu 1: failed to submit desc 0
[ 2466.292588] gvt: vgpu 1: fail submit workload on ring rcs0
[ 2466.292588] gvt: vgpu 1: fail to emulate MMIO write 00002230 len 4

and qemu:

qemu-system-x86_64: vfio_pci_write_config(9085644c-8cf8-4df0-900e-0c1362deddea, 0x4, 0x100407, 0x4) failed: Ung?ltige Adresse
qemu-system-x86_64: vfio_pci_write_config(9085644c-8cf8-4df0-900e-0c1362deddea, 0x4, 0x100407, 0x4) failed: Ung?ltige Adresse
qemu-system-x86_64: vfio_region_write(9085644c-8cf8-4df0-900e-0c1362deddea:region0+0x2080, 0x402a4000,4) failed: Ung?ltige Adresse
qemu-system-x86_64: vfio_region_write(9085644c-8cf8-4df0-900e-0c1362deddea:region0+0x12080, 0x402a5000,4) failed: Ung?ltige Adresse
qemu-system-x86_64: vfio_region_write(9085644c-8cf8-4df0-900e-0c1362deddea:region0+0x22080, 0x402a6000,4) failed: Ung?ltige Adresse
qemu-system-x86_64: vfio_region_write(9085644c-8cf8-4df0-900e-0c1362deddea:region0+0x1a080, 0x402a7000,4) failed: Ung?ltige Adresse
qemu-system-x86_64: vfio_region_write(9085644c-8cf8-4df0-900e-0c1362deddea:region0+0x2230, 0x402a9119,4) failed: Ung?ltige Adresse 

the MMIO(24d8) and MMIO(24dc) are not "whitelisted" and there seem to be issues with the gtt/aperture size (some address range is being validated, but seems to fail), but i didnt have luck to solve it and not enough understanding of the nuts and bolts
of the Intel driver terminology.... all those crazy shortcuts... 🙁

So, what relevant bits are missing here in gvt which seem not to be passed further to the "Host" i915 ?