QEMU with VirtIO GPU Vulkan Support

qemu-vulkan-virtio.md

QEMU with VirtIO GPU Vulkan Support

With its latest reales qemu added the Venus patches so that virtio-gpu now support venus encapsulation for vulkan. This is one more piece to the puzzle towards full Vulkan support.

An outdated blog post on clollabora described in 2021 how to enable 3D acceleration of Vulkan applications in QEMU through the Venus experimental Vulkan driver for VirtIO-GPU with a local development environment. Following up on the outdated write up, this is how its done today.

Definitions

Let's start with the brief description of the projects mentioned in the post & extend them:

• QEMU is a machine emulator
• VirGL is an OpenGL driver for VirtIO-GPU, available in Mesa.
• Venus is an experimental Vulkan driver for VirtIO-GPU, also available in Mesa.
• Virglrenderer is a library that enables hardware acceleration to VM guests, effectively translating commands from the two drivers just mentioned to either OpenGL or Vulkan.
• libvirt is an API for managing platform virtualization
• virt-manager is a desktop user interface for managing virtual machines through libvirt

Merged Patches:

• 2024-08-14 venus: make cross-device optional merged in mesa 24.2
• 2024-11-25 KVM: Stop grabbing references to PFNMAP'd pages merged in linux 6.13
• 2024-11-12 Support blob memory and venus on qemu merged in qemu 9.2.0

Work in progress:

• libvirt Add support for more virtio-vga-gl arguments #638
• virt-manager Add support for Venus / Vulkan VirtIO-GPU driver #362

Prerequisites

Make sure you have the proper version installed on the host:

• linux kernel >= 6.13 built with CONFIG_UDMABUF
• working Vulkan and kvm setup
• qemu >= 9.2.0
• virglrenderer with enabled venus support
• mesa >= 24.2.0

You can verify this like so:

$ uname -r
6.13.0
$ ls /dev/udmabuf
/dev/udmabuf
$ ls /dev/kvm
/dev/kvm
$ qemu-system-x86_64 --version
QEMU emulator version 9.2.0
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

Check your distros package sources how they build virglrenderer.

For Vulkan to work you need the proper drivers to be installed for your graphics card.

To verfiy your setup, install vulkan-tools. Make sure mesa >= 24.2.0 and test vkcube:

$ vulkaninfo --summary | grep driverInfo
	driverInfo         = Mesa 24.2.3-1ubuntu1
	driverInfo         = Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0)
...
$ vkcube
Selected GPU x: ..., type: ...

Building qemu

If your distro doesn't (yet) ship and updated version of qemu, you can build it yourself from source:

wget https://download.qemu.org/qemu-9.2.0.tar.xz
tar xvJf qemu-9.2.0.tar.xz
cd qemu-9.2.0
mkdir build && cd build
../configure --target-list=x86_64-softmmu  \
  --enable-kvm                 \
  --enable-opengl              \
  --enable-virglrenderer       \
  --enable-gtk                 \
  --enable-sdl
make -j4

The configuration step will throgh errors if packages are missing. Check the qemu wiki for further info what to install: https://wiki.qemu.org/Hosts/Linux

Create and run an image for QEMU

Create an image & fetch the distro of your choice:

Host

ISO=ubuntu-24.10-desktop-amd64.iso  
wget https://releases.ubuntu.com/oracular/ubuntu-24.10-desktop-amd64.iso  

IMG=ubuntu-24-10.qcow2
qemu-img create -f qcow2 $IMG 16G

Run a live version or install the distro

qemu-system-x86_64                                               \
    -enable-kvm                                                  \
    -M q35                                                       \
    -smp 4                                                       \
    -m 4G                                                        \
    -cpu host                                                    \
    -net nic,model=virtio                                        \
    -net user,hostfwd=tcp::2222-:22                              \
    -device virtio-vga-gl,hostmem=4G,blob=true,venus=true        \
    -vga none                                                    \
    -display gtk,gl=on,show-cursor=on                            \
    -usb -device usb-tablet                                      \
    -object memory-backend-memfd,id=mem1,size=4G                 \
    -machine memory-backend=mem1                                 \
    -hda $IMG                                                    \
    -cdrom $ISO                                                  

Adjust the parameters accordingly:

• smp: number of cpu cores
• m: RAM
• hostmem,size: VRAM

Guest

Install mesa-utilites and vulkan-tools to test the setup:

$ glxinfo -B

$ vkcube
Selected GPU x: ..., type: ...

If the deive is llvmpipe somehting is wrong. The device should be virgl (...).

Troubleshooting

• (host) add -d guest_errors to show error messages from the guest
• (guest) try installing vulkan virtio drivers and mesa
• check the original blog post

Ubuntu 24.10

This is how you do it on Ubuntu

kernel

Install mainline: https://github.com/bkw777/mainline

sudo add-apt-repository ppa:cappelikan/ppa
sudo apt update
sudo apt install mainline

find the latest kernel (>= 6.13), at the time of writing 6.13 is a release candidate, so include those:

$ mainline check --include-rc

Install kernel:

$ sudo mainline install 6.13-rc1

Verfify installed kernels:

$ mainline list-installed
mainline 1.4.10
Installed Kernels:
linux-image-6.11.0-13-generic
linux-image-generic-hwe-24.04
linux-image-unsigned-6.13.0-061300rc1-generic
mainline: done

reboot into new kernel

verify running kernel

$ uname -r
6.13.0-061300rc1-generic

virglrenderer

the ubuntu package is not compiled with the proper flags.

If installed remove it: $ sudo apt-get remove libvirglrenderer-dev

download, build & install from source with venus enabled

wget    https://gitlab.freedesktop.org/virgl/virglrenderer/-/archive/1.1.0/virglrenderer-1.1.0.tar.gz
sudo apt-get install python3-full ninja-build libvulkan-dev libva-dev
python3 -m venv venv
venv/bin/pip install meson
venv/bin/meson build -Dvideo=true -Dvenus=true
ninja -C build
ninja install

qemu

install qemu >= 9.2.0, at the time of writing ubuntu has not yet packaged it

Install build depdencies: https://wiki.qemu.org/Hosts/Linux

sudo apt-get install build-essential pip libslirp-dev slirp
sudo apt-get install git libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev ninja-build
sudo apt-get install git-email
sudo apt-get install libaio-dev libbluetooth-dev libcapstone-dev libbrlapi-dev libbz2-dev
sudo apt-get install libcap-ng-dev libcurl4-gnutls-dev libgtk-3-dev
sudo apt-get install libibverbs-dev libjpeg8-dev libncurses5-dev libnuma-dev
sudo apt-get install librbd-dev librdmacm-dev
sudo apt-get install libsasl2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh-dev
sudo apt-get install libvde-dev libvdeplug-dev libvte-2.91-dev libxen-dev liblzo2-dev
sudo apt-get install valgrind xfslibs-dev 
sudo apt-get install libnfs-dev libiscsi-dev

build and run as described

virt-manager

-- work in progress --

Currently this is work in progress, so there is no option to add vulkan support in virt-manager. There are no fields to configure this. Also xml doesnt work, because libvirt doesn't know about these options either, so xml validation fails. There is however an option for QEMU command-line passthrough which bypasses the validation.

If you setup a default machine with 4G of memory, you can do this:

  <qemu:commandline>
    <qemu:arg value="-device"/>
    <qemu:arg value="virtio-vga-gl,hostmem=4G,blob=true,venus=true"/>
    <qemu:arg value="-object"/>
    <qemu:arg value="memory-backend-memfd,id=mem1,size=4G"/>
    <qemu:arg value="-machine"/>
    <qemu:arg value="memory-backend=mem1"/>
    <qemu:arg value="-vga"/>
    <qemu:arg value="none"/>
  </qemu:commandline>

Which gives this error:

qemu-system-x86_64: virgl could not be initialized: -1

Changing the number from 4G to 4194304k (same as memory) leds to this error:

qemu-system-x86_64: Spice: ../spice-0.15.2/server/red-qxl.cpp:435:spice_qxl_gl_scanout: condition `qxl_state->gl_draw_cookie == GL_DRAW_COOKIE_INVALID' failed

to be further investigated.

as expected both virglrenderer and mesa don't have the compile flags in cachyos or arch, currently compiling to test.

Please share your result here when you finish.

it works, both vainfo and vulkaninfo | grep video report video encoding and decoding capabilities. i also tested mpv with an AV1 video and decoding works. Sunshine recognizes the encoder but i couldn't test it due to networking.

basically native context replaces normal virgl and venus while also providing more functionality out of the box. Unfortunately performance is off and even worse than venus (vkmark on host about 18k fps while drm sits at 180fps and venus at 240/250fps).

To use it (host and guest archlinux-based):

• if you are on amd, virglrenderer from arch repo should be fine. if on intel, install virglrender-git from AUR
• build qemu from git using the patchset mentioned before. I took the qemu-git PKGBUILD from AUR and swapped the repo and tag. didn't install it (interferes with some system packages on my system), but you can launch the binary directly.
• inside the host, install mesa-git from AUR

to get qemu-git to build with the correct patchset i modified this in the PKGBUILD:

options=(!debug !lto)
source+=(
        "git+https://github.com/patchew-project/qemu"
        qemu-guest-agent.service
        65-kvm.rules)
sha256sums=('SKIP'
            'c39bcde4a09165e64419fd2033b3532378bba84d509d39e2d51694d44c1f8d88'
            'a66f0e791b16b03b91049aac61a25950d93e962e1b2ba64a38c6ad7f609b532c')

case $CARCH in
  i?86) _corearch=i386 ;;
  x86_64) _corearch=x86_64 ;;
esac

_patchew_tag="patchew/20260217214928.2970885-1-dmitry.osipenko@collabora.co"

pkgver=20260217

prepare() {
  cd "${srcdir}/${_gitname}"
  git fetch https://github.com/patchew-project/qemu tags/patchew/20260217214928.2970885-1-dmitry.osipenko@collabora.com
  git checkout FETCH_HEAD
  mkdir build-{full,headless}
  mkdir -p extra-arch-{full,headless}/usr/{bin,share/qemu}
}

Since i'm on AMD technically i only need to build qemu (host) and mesa (guest), virglrenderer should already have drm native context on 1.2.0

let me know if i can help further.

Did you try with the SDL QEMU backend? The AMD vDRM supposedly was only tested using that and not the GTK backend since the GTK backend had problems. Conversely, only the GTK backend works right for Venus + blob memory when I was testing.

Instead of using Vkmark, try a game or demanding benchmark like unigine heaven. I believe that QEMU has a hard limit on FPS but performance of AMD vDRM should be 99% of host if below that limit (say a 4k game at 60 FPS).

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21658#note_1819915

Dmitry Osipenko
@digetx Mar 13, 2023
Contributor

Are you running benchmarks at max settings/resolution? If you'll run something simpler, like glmark2 for example, you'll hit the Qemu limit. See https://github.com/qemu/qemu/blob/v7.2.0/hw/display/virtio-gpu-virgl.c#L582 where Qemu limits fence status polling to once per 10ms, hence 100fps max.

Pierre-Eric Pelloux-Prayer
@pepp Mar 14, 2023
Author Developer

For Qemu both SDL and GTK outputs need to work

Both work; GTK output's performance isn't as good as SDL output's (I think this is caused by the redraw logic being async in GTK, but I didn't investigate much)

Instead of using Vkmark, try a game or demanding benchmark like unigine heaven. I believe that QEMU has a hard limit on FPS but performance of AMD vDRM should be 99% of host if below that limit (say a 4k game at 60 FPS).

It's probably the 1st thing folks would try since it's pretty handy. I'd agree it's not something suitable for VM case given its on-screen nature (even with headless surface backend). In case you are interested, I've included the details for where it's throttled for venus and native context here: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/39283

Did you try with the SDL QEMU backend? The AMD vDRM supposedly was only tested using that and not the GTK backend since the GTK backend had problems. Conversely, only the GTK backend works right for Venus + blob memory when I was testing.

Instead of using Vkmark, try a game or demanding benchmark like unigine heaven. I believe that QEMU has a hard limit on FPS but performance of AMD vDRM should be 99% of host if below that limit (say a 4k game at 60 FPS).

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21658#note_1819915

Dmitry Osipenko
@digetx Mar 13, 2023
Contributor
Are you running benchmarks at max settings/resolution? If you'll run something simpler, like glmark2 for example, you'll hit the Qemu limit. See https://github.com/qemu/qemu/blob/v7.2.0/hw/display/virtio-gpu-virgl.c#L582 where Qemu limits fence status polling to once per 10ms, hence 100fps max.

Pierre-Eric Pelloux-Prayer
@pepp Mar 14, 2023
Author Developer

For Qemu both SDL and GTK outputs need to work

Both work; GTK output's performance isn't as good as SDL output's (I think this is caused by the redraw logic being async in GTK, but I didn't investigate much)

indeed i was using gtk, but i can switch to sdl to try.

@zzyiwei true, it's quick and easy to install and run, would something like supertuxkart be viable?

supertuxkart - 75fps locked (monitor refresh rate)
xonotic - also limited to 75fps managed to unlock fps, running at 450fps (cpu limited for sure, i'm running on a R53600 and 4 cores are available to the vm)

with a vkmark score this low it shouldn't have this perf?

i'll try with steam when i have more time

managed to sneak in unigine heaven runs, same settings but vm is limited to 4 cores:

• vm score 4187 avg fps 166 max 332 min 18
• host score 6243 avg fps 247 max 447 min 18

host managed to max out gpu most of the time (220-280W) guest only managed to reach 220W but was averaging close to 100W

looks like there's some bottleneck somewhere, i might retest passing 12 vcpus so that we can rule out cpu bottleneck besides qemu overhead

Not bad! Nearly 67% host performance is honestly pretty amazing. Theoretically, it should be able to do better (assuming the same number of cores and sufficient guest RAM).

Also try with virtio-gpu-gl-pci, I actually got better performance with Venus and VirGL when using that over virtio-vga-gl.

Also try with virtio-gpu-gl-pci, I actually got better performance with Venus and VirGL when using that over virtio-vga-gl.

i tried venus with it before and performance for me was equal. Right now with vDRM i'm using virtio-vga-gl because it was recommended in the updated docs. i'll test that first before increasing vcpu count to have a 1:1 reference with vga-gl

ERRATA: turns out i was already using virtio-gpu-gl in the previous test, i switched to virtio-gpu-gl-pci but the heaven result is the same (sligthly less but in line with variance). So no performance boost for me (i would have to test virtio-vga-gl to see if it performs worse).

Also try with virtio-gpu-gl-pci, I actually got better performance with Venus and VirGL when using that over virtio-vga-gl.

i just tested with 12 vcpus (max): score 4291 (about 200 higher than 4vcpus) avg fps 170 max 342 min 28.8

looks like there's some overhead from vDRM and maybe something is up with heaven. This is also opengl only, so i should test something that uses vulkan next.