-
Notifications
You must be signed in to change notification settings - Fork 95
GVTg_Setup_Guide
- 1 Introduction
- 2 System Reruirements
- 3 Host Environment Setup
- 4 Guest Setup
- 5 Basic usage
- 6 Features Supported
Intel GVT-g is a full GPU virtualization solution with mediated pass-through (VFIO mediated device framework based), starting from 5th generation Intel Core(TM) processors with Intel Graphics processors. GVT-g supports both Xen and KVM (a.k.a XenGT & a.k.a KVMGT). A virtual GPU instance is maintained for each VM, with part of performance critical resources directly assigned. The capability of running native graphics driver inside a VM, without hypervisor intervention in performance critical paths, achieves a good balance among performance, feature, and sharing capability.
Ubuntu 16.04.1 has been fully validated as host, other Linux operating system like RHEL / Fedora are also OK.
5th, 6th or 7th Generation Intel® Core Processor Graphics is required.
Use Ubuntu as example, there are some software package needed by host environment setup, as below:
apt-get install git libfdt-dev libpixman-1-dev libssl-dev vim socat libsdl1.2-dev libspice-server-dev autoconf libtool xtightvncviewer tightvncserver x11vnc libsdl1.2-dev uuid-runtime uuid uml-utilities bridge-utils python-dev liblzma-dev libc6-dev
GVT-g kernel: https://github.com/intel/gvt-linux/ (branch: gvt-stable-4.12)
QEMU: https://github.com/intel/igvtg-qemu (branch: stable-2.9.0)
Xen : https://github.com/intel/igvtg-xen (branch: xengt-stable-4.9)
NOTICE: For KVMGT, you also can use the current upstream Linux kernel and QEMU directly since all the enabling patches have been upstreamed.
For XenGT, you must use the repositories we provided.
Use Ubuntu as example, modify /etc/initramfs-tools/modules, like below:
kvmgt
xengt
vfio-iommu-type1
vfio-mdev
For Fedora or RHEL you should make the similar changes in "dracut".
cd gvt-linux
git checkout gvt-stable-4.12
echo ""|make oldconfig
Then make sure to enable CONFIG_DRM_I915_GVT, CONFIG_DRM_I915_GVT_KVMGT and CONFIG_DRM_I915_GVT_XENGT in ".config", which depends on CONFIG_VFIO_MDEV and CONFIG_VFIO_MDEV_DEVICE.
make -j8 && make modules_install && make install
If you use Ubuntu, you can generate the kernel debian package directly and install it with below commands:
make -j8 deb-pkg
cd ..
dpkg -i linux-image-4.12.0+_4.12.0+-2_amd64.deb
Pls follow chapter 3.3.1 for KVMGT setup and chapter 3.3.2 for XenGT setup
cd igvtg-qemu
git checkout stable-2.9.0
git submodule update --init roms/seabios
./configure --prefix=/usr \
--enable-kvm \
--disable-xen \
--enable-debug-info \
--enable-debug \
--enable-sdl \
--enable-vhost-net \
--enable-spice \
--disable-debug-tcg \
--target-list=x86_64-softmmu
make -j8
cd roms/seabios
make -j8
cd -
make install
cp roms/seabios/out/bios.bin /usr/bin/bios.bin
git clone https://github.com/intel/igvtg-xen
cd igvtg-xen
git checkout xengt-stable-4.9
cd igvtg-qemu
git checkout stable-2.9.0
cd ..
sed -i 's/\(QEMU_UPSTREAM_REVISION ?= \).*/\1origin\/stable-2.9.0/g' Config.mk
sed -i 's/\(QEMU_UPSTREAM_URL ?= \).*/\1file:\/\/localhost\/root\/igvtg-xen\/igvtg-qemu/g' Config.mk
./autogen.sh
./configure --prefix=/usr
make -j8 xen tools
make install-tools
cp xen/xen.gz /boot/vgt/xen-vgt.gz
You need to add the KVMGT and XenGT corrsponing menuentry in your grub.cfg, or you can add it to /etc/default/grub is better. (Since grub.cfg will be auto regenerated by linux distro version upgrade or kernel package update).
For KVMGT, Make sure the "i915.enable_gvt=1" exist in your menuentry, below is an example on ubuntu:
menuentry 'GVT-g Upstream' --class kvmgt --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
recordfail
load_video
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 106518cd-fb74-4048-81de-3ecee7ad74d4
else
search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
fi
linux /boot/vgt/host-gvt root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel log_buf_len=128M console=ttyS0,115200,8n1 i915.enable_gvt=1 kvm.ignore_msrs=1 intel_iommu=igfx_off drm.debug=0
initrd /boot/vgt/initrd-gvt.img
}
NOTICE: Since some windows guest 3rd patry application / tools (like GPU-Z / Passmark9.0) will trigger MSR read / write directly, if it access the unhandled msr register, guest will trigger BSOD soon. So we added the "kvm.ignore_msrs=1" into grub for workaround.
For XenGT, make sure the "i915.enable_gvt=1" exist in your menuentry, below is an example on ubuntu:
menuentry 'upstream vfio' --class kvmgt --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
recordfail
load_video
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 106518cd-fb74-4048-81de-3ecee7ad74d4
else
search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
fi
multiboot /boot/vgt/xen-gvt.gz dom0_max_vcpus=2 dom0_mem=2048M iommu=1 loglvl=all guest_loglvl=all msi=1 conring_size=4M console=com1 com1=115200,8n1 sync_console
module /boot/vgt/dom0-gvt root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel intel_iommu=igfx_off i915.hvm_boot_foreground=1 log_buf_len=128M console=hvc0 drm.debug=0 i915.enable_gvt=1
module /boot/vgt/initrd-gvt.img
}
You can prepare any version Ubuntu or any other Linux guest OS image manually. Then you can change the guest kernel to the same kernel as host, you also can download the stable version from https://www.kernel.org/ or use Ubuntu16.10 or Fedora25 as guest OS (with 4.8 kernel which is support GVT-g) directly.
-
We need to make the special X11 configuration change for Linux guest to use the Intel graphics card as a display card . *The emulated graphics card (linke "QXL", "Cirrus")not support OpenCL / OpenGL hardware acceleration, we must make this change. *Here we use Ubuntu as example, we save the following section at /etc/X11/xorg.conf file:
Section "Device" Identifier "intel" Driver "intel" BusID "PCI:0:4:0" # Sample: "PCI:0:2:0" EndSection
Section "Screen" Identifier "intel" Device "intel" EndSection
BTW: The PCI ID was chosen by looking at the output of the "lspci" command.
root@gvtg-host:~# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
00:04.0 VGA compatible controller: Intel Corporation Broadwell-U Integrated Graphics (rev 0a)
NOTICE: After you changed the display card to Intel graphics card, Qemu will comes to black screen, you only can see guest desktop by remote protocal.
We suggest to use the X11VNC as the remote protocol to validation which has been validated by us.
Use Ubuntu as example, to support X11VNC, you must install the xhost in guest first, the xhost program is used to add and delete host names or user names to the list allowed to make connections to the X server, you must make some configuration for lightdm, the example as below:
root@gvtg-host:~# cat /etc/profile.d/lightdm_display.sh
#!/bin/bash
export DISPLAY=:0
if [ ! -z "$DISPLAY" ]; then
xhost +local:
fi
root@gvtg-host:~# chmod +x /etc/profile.d/lightdm_display.sh
root@gvtg-host:~# cat /etc/lightdm/lightdm.conf
[SeatDefaults]
greeter-session=unity-greeter
user-session=ubuntu
greeter-setup-script=/etc/profile.d/lightdm_display.sh
session-setup-script=/etc/profile.d/lightdm_display.sh
After you completed the lightdm configuration, you can run the X11VNC service in guest by following commands or similar commands:
x11vnc -display :0 -forever -bg -repeat -nowf -o ~/.vnc/x11vnc.log -rfbport 5900
Then connect to the guest by host with guest X support by typing the following command:
vncviewer $guestIP
NOTICE: $guestIP means guest IP address, you can got the guest IP address by "nmap" / "arp-scan" or any other network related tools in host.
Except X11VNC, there are some other alternatives, which were not validated by us, you also can use it as remote protocol for a try:
- NX
- X2Go and based on nx-libs (it seems QVD uses it too)
- QVD (open source version)
- NoMachine: Closed source.
- OpenNX. Client only. Not updated since 2016-10-01.
- VNC
- TurboVNC
Win7-32 / Win7-64 / Win8.1-64 /Win10-RS1-64 are validated.
-
After you boot up Windows guest, you can see two graphics card in "Device manager"
-
The virtual GFX card "QXL" or "Cirrus"
-
The Intel GFX card
-
Then you must install the Intel GFX driver for the Intel Graphics card to support OpenCL / OpenGL / DirectX9 / DirectX10/ DirectX11 / DirectX12 hardware acceleration .
-
You can get the Windows driver from graphics drivers section of the Intel websi te. Depending on your Intel processor, the latest available driver versions as below:
Kabylake: 15.45.19.4678 Skylake: 15.45.19.4678 Broadwell: 15.40.34.4703
NOTICE: After you installed Windows GFX driver, you can see guest desktop by both QEMU and remote protocol. QEMU display the emulated driver and the remote protocol display the Intel GFX driver. BTW, There is a known issue that some 3D workload cannot run with hardware acceleration while both GFX card is available, so suggest to disable the emulated GFX card in "Device Manager" after you make sure the Intel GFX card is working.
We suggest to use Tightvnc as the remote protocol to validation which has been validated by us.
Then connect to the guest by host with guest X support by typing the following command:
vncviewer $guestIP
NOTICE: $guestIP means guest IP address, you can got the guest IP address by "nmap" / "arp-scan" or any other network related tools in host.
All Windows Remote protocol are supported, such as TightVNC, HP RGS, RDP.
Except Tightvnc, there are some other alternatives, which were not validated by us, you also can use it as remote protocol for a try:
- VNC (Virtual Network Computing)
- HP's RGS (Remote Graphics Software), which is an enterprise-level application.
- Microsoft's RDP (Remote Desktop Protocol).
- FreeRDP: The best Open Source RDP implementation.
- Remmina (GTK+ 3): Based on FreeRDP. It supports the RDP, VNC, NX, XDMCP and SSH protocols.
- WinConn: Promising but abandoned project. Just put here as reference, not useful for end users (the readers of this document).
- PAC Manager
- KDE's KRDC (Remote Display Client): It supports both VNC and RDP protocols.
- Gnome's Vinagre
- NX
After you boot up you host to GVT-g menuentry, you make sure the "mdev_supported_types" node with differenct vgpu type existed(The "V#ID" represent the corresponding platform, "V4" means it is "Broadwell" platform, "V5" means it is "Skylake" or "Kabylake" platform), the example as below
root@gvtg-host:~# ll /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/
total 0
drwxr-xr-x 5 root root 0 Apr 10 21:24 ./
drwxr-xr-x 12 root root 0 Apr 10 20:04 ../
drwxr-xr-x 3 root root 0 Apr 10 21:24 i915-GVTg_V4_1/ $ This node with low_gm_size=512MB, high_gm_size=2048MB, fense=4, resolution=1920x1200
drwxr-xr-x 3 root root 0 Apr 10 21:24 i915-GVTg_V4_2/ # This node with low_gm_size=256MB, high_gm_size=1024MB, fense=4, resolution=1920x1200
drwxr-xr-x 3 root root 0 Apr 10 21:24 i915-GVTg_V4_4/ # This node with low_gm_size=128MB, high_gm_size=512MB, fense=4, resolution=1920x1200
drwxr-xr-x 3 root root 0 Apr 10 21:24 i915-GVTg_V4_8/ # This node with low_gm_size=64MB, high_gm_size=384MB, fense=4, resolution=1024x768
And you can check the detail description for each vgpu type,
root@gvtg-host:/sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/i915-GVTg_V4_1# cat description
low_gm_size: 512MB
high_gm_size: 2048MB
fence: 4
resolution: 1920x1200
Use Ubuntu as example, you can use the third-party tool to generate uuid like "uuid", "uuidgen", if we need to create 3 VM, we must create 3 uuid for 3 VGPU, like below:
root@gvtg-host:~# uuid -n 3
a297db4a-f4c2-11e6-90f6-d3b88d6c9525
a297de6a-f4c2-11e6-90f7-cb6a86ce449f
a297deec-f4c2-11e6-90f8-c39f1ae774e5
Use the uuid you generated to create VGPU, notes: if you want to boot up multi guests, you can use echo "uuid" > "/sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/i915-GVTg_V#ID_#Type/create", like below:
echo "a297db4a-f4c2-11e6-90f6-d3b88d6c9525" > "/sys/bus/pci/devices/0000:00:02.0/ mdev_supported_types/i915-GVTg_V4_4/create"
echo "a297de6a-f4c2-11e6-90f7-cb6a86ce449f" > "/sys/bus/pci/devices/0000:00:02.0/ mdev_supported_types/i915-GVTg_V4_4/create"
echo "a297deec-f4c2-11e6-90f8-c39f1ae774e5" > "/sys/bus/pci/devices/0000:00:02.0/ mdev_supported_types/i915-GVTg_V4_4/create"
Then all the vgpu directory you created can be found at "/sys/bus/pci/devices/0000:00:02.0/"
If you want to remove one vgpu you create, you need to "echo 1 > remove" at the vgpu directory and created, the example as below:
echo 1 > /sys/bus/pci/devices/0000:00:02.0/a297db4a-f4c2-11e6-90f6-d3b88d6c9525/remove
If you want remove all vgpu you created, you can run below command.
for i in /sys/bus/pci/devices/0000:00:02.0/*/remove; do echo 1 > $i; done
For KVMGT, in order to let guest using the same network segment as host, you must create the network bridge. Besides, in order to let Qemu using the network bridge you created, you also need create a scripte as below (We saved it at /etc/qemu-ifup):
Detail you can reference the the "Configuring Guest Networking" wiki page http://www.linux-kvm.org/page/Networking
For XenGT you can just create the network bridge by yourself.
Pls refer chapter 5.6.1 to create KVMGT VM and chapter 5.6.2 to create XenGT VM
We sugguest to use the secondary gfx card mode (virtual "qxl" or "cirrus" gfx card as 1st guest gfx card, intel gfx card as 2nd guest gfx card)to create VM, the example of create VM script as below :
#! /bin/bash -x
/usr/bin/qemu-system-x86_64 \
-m 2048 -smp 2 -M pc \
-name gvt-g-guest \
-hda /home/img/ubuntu-1.qcow2 \
-bios /usr/bin/bios.bin -enable-kvm \
-net nic,macaddr=00:A1:00:00:00:1A -net tap,script=/etc/qemu-ifup \
-vga qxl \
-k en-us \
-serial stdio \
-vnc :1 \
-machine kernel_irqchip=on \
-global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 \
-cpu host -usb -usbdevice tablet \
-device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:02.0/a297db4a-f4c2-11e6-90f6-d3b88d6c9525,rombar=0
You can create hvm configuration file by modifying /etc/xen/xlexample.hvm like below:
builder='hvm'
vgt=1
vcpus=2
memory = 4096
name = "win10-1"
vif = [ 'type=ioemu,bridge=xenbr0,mac=00:25:10:03:0A:01,model=e1000' ]
disk = ['/home/img/win10-red-stone.img,raw,hda,w']
device_model_version = 'qemu-xen'
device_model_override = '/usr/lib/xen/bin/qemu-system-i386'
boot="dc"
opengl=0
serial='pty'
usb=1
usbdevice='tablet'
sdl=0
vnc=1
keymap='en-us'
vgt_low_gm_sz=128
acpi_s4=0
acpi_s3=0
viridian=["all", "!apic_assist"]
NOTICE: The "vgt_low_gm_sz" is used to set vm aperture size, the supported valid value are 64 / 128 / 256 / 512 which means vm aperture size is 64MB / 128MB / 256MB / 512MB. Other invalid value will cause VM start failure.
After that, you can create XenGT VM via the below command:
xl create xlexample.hvm
- General:
- Supported Linux distributions (64bit): Ubuntu, Fedora, RHEL, SUSE, etc
- Supported Windows guest: Windows7 (32bit, 64bit), Windows8.1 (64bit), Windows10 RedStone (64bit)
- Supported total vGPU number <= 7
- Single display with resolution up to 1920x1080p
- 3D/2D:
- OpenGL4.5 for Linux guest
- OpenGL4.4 for Windows guest
- DirectX9, 10, 11, 12 for Windows guest
- Compute:
- OpenCL2.0 for Windows/Linux guest
- Media:
- Intel Media Service Studio (2015R5)
- Remote display protocol:
- X11VNC for Linux guest
- TightVNC, HP RGS, RDP for Windows guest
- 3D / 2D:
- 3Dmark06, 3Dmark11, WebGL, Passmark, Unigine-tropics, Unigine-heaven, Phoronix 3D, Cairo 2D, etc
- Media:
- Multiple thread decode/transcode for JPEG, MPEG2, H264, HEVC, VC1, VP8, etc
- OpenCL:
- LuxMark, Beignet