KVM - User contributions [en]

Talk:How to assign devices with VT-d in KVM

2017-05-27T12:37:03Z

Jan: /* method obsolete -> use vfio-pci */ new section

Hello,

Does this work with AMD IOMMU too?

Thanks, Peter

== method obsolete -> use vfio-pci ==

Just found someone blindly referencing this. Should be updated, specifically now that legacy pci-assign has been removed.
-- Jan

Getting the kvm kernel modules

2011-10-25T11:09:07Z

Jan: Update git references to point to kiszka.org

There are a variety of sources of kvm kernel modules. Which ones to use depend on whether you are interested in using, testing, or developing kvm.

As of now, all major community and enterprise distributions contain kvm kernel modules and userspace; these are either installed by default or require installing a kvm package. For someone looking for stability, these are the best choice. No effort is needed to build or install the modules, support is provided by the distribution, and the distribution/module combination is well tested.

For those stuck on an older kernel, but wishing to run a modern, stable kvm release the kvm maintainers provide an external module kit that can run a modern kvm on an older kernel. These packages are named kvm-kmod-3.x.y (or kvm-kmod-3.x), and correspond to the kvm code contained in Linux 3.x.y. They can generally be run on any Linux kernel older than 3.x, but not older than 2.6.24.

If you wish to help testing the kvm component of the next Linux release, but without running a -rc kernel from kernel.org, you may use kvm-kmod-3.x-rcy, containing the code from Linux 3.x-rcy backported to run on older kernels. Feedback from running this release will help improve the kvm in the next kernel release.

For testing development snapshots, the [http://git.kiszka.org/?p=kvm-kmod.git kvm-kmod git repository] can be cloned. See [http://git.kiszka.org/?p=kvm-kmod.git;a=blob;f=README kvm-kmod README] for detailed build instructions. Modules generated via this source don't correspond exactly to any Linux release, but instead use code from the kvm.git (the kvm development repository) at the point of their release. These are generally the most up-to-date modules.

In summary:

{|
! Release type
! Suitable for
|-
| Distribution provided
| Mainstream use; supported by distribution
|-
| kvm-kmod-2.6.x.y
| Running modern, stable kvm releases on older kernels
|-
| kvm-kmod-2.6.x-rcy
| Testing the next kvm-kmod-2.6.x.y release
|-
| kvm-kmod git
| Testing bleeding-edge development code
|}

File:KVM-Forum-2011-RT-KVM.pdf

2011-08-13T04:13:02Z

Jan:

Getting the kvm kernel modules

2011-07-23T11:42:21Z

Jan:

There are a variety of sources of kvm kernel modules. Which ones to use depend on whether you are interested in using, testing, or developing kvm.

As of now, all major community and enterprise distributions contain kvm kernel modules and userspace; these are either installed by default or require installing a kvm package. For someone looking for stability, these are the best choice. No effort is needed to build or install the modules, support is provided by the distribution, and the distribution/module combination is well tested.

For those stuck on an older kernel, but wishing to run a modern, stable kvm release the kvm maintainers provide an external module kit that can run a modern kvm on an older kernel. These packages are named kvm-kmod-3.x.y (or kvm-kmod-3.x), and correspond to the kvm code contained in Linux 3.x.y. They can generally be run on any Linux kernel older than 3.x, but not older than 2.6.24.

If you wish to help testing the kvm component of the next Linux release, but without running a -rc kernel from kernel.org, you may use kvm-kmod-3.x-rcy, containing the code from Linux 3.x-rcy backported to run on older kernels. Feedback from running this release will help improve the kvm in the next kernel release.

For testing development snapshots, the [http://git.kernel.org/?p=virt/kvm/kvm-kmod.git kvm-kmod git repository] can be cloned. See [http://git.kernel.org/?p=virt/kvm/kvm-kmod.git;a=blob;f=README kvm-kmod README] for detailed build instructions. Modules generated via this source don't correspond exactly to any Linux release, but instead use code from the kvm.git (the kvm development repository) at the point of their release. These are generally the most up-to-date modules.

In summary:

{|
! Release type
! Suitable for
|-
| Distribution provided
| Mainstream use; supported by distribution
|-
| kvm-kmod-2.6.x.y
| Running modern, stable kvm releases on older kernels
|-
| kvm-kmod-2.6.x-rcy
| Testing the next kvm-kmod-2.6.x.y release
|-
| kvm-kmod git
| Testing bleeding-edge development code
|}

Status

2011-05-27T11:03:05Z

Jan: add VGA assigment

=Status=

KVM is included in the mainline linux kernel since 2.6.20 and is stable and fast for most workloads.

It is also available as a patch for recent Linux kernel versions and as an external module that can be used with your favorite distro- provided kernel going back up to 2.6.16, therefore including all latest versions for Enterprise Linux Distributions.

===Working:===

* Intel-based hosts (requires VT capable processors)
* AMD-based hosts (requires SVM capable processors)
* Windows/Linux/Unix guests (32-bit and 64-bit)
* SMP hosts
* SMP guests (as of kvm-61, max 16 cpu supported)
* Live [[Migration]] of guests from one host to another (32-bit and 64-bit)
* See the [[Guest Support Status]] page for a list of guest operating systems known to work
* See the [[Host Support Status]] page for information on host hardware.
* Guest swapping
* [[Paravirtualized networking]]
* [[Paravirtualized block device]]
* [[How_to_assign_devices_with_VT-d_in_KVM|PCI-Express passthrough]]

===In progress:===

* [[PowerPC|PowerPC port]]
* IA64 port
* xenner (http://kraxel.fedorapeople.org/xenner), a project to run x86 xen guest (domU) kernels
* [https://wiki.ncl.cs.columbia.edu/wiki/index.php/AndroidVirt:MainPage ARM port]
* [[VGA_device_assignment|VGA device assignment]]

VGA device assignment

2011-05-27T10:58:48Z

Jan:

= Assigning physical VGA adapters =

On first glance, assigning modern PCI/PCIe VGA adapters should not be different from [[How_to_assign_devices_with_VT-d_in_KVM|passing through normal PCI devices]]. But there are additional challenges to solve, specifically related to legacy VGA requirement and proprietary mechanisms to relocate I/O regions of the adapters. Moreover, some QEMU and SeaBIOS deficits contribute to the current situation that things rarely work out of the box.

== Open issues ==

* '''Insufficient PCI window size''' This is a fixable SeaBIOS limitations. Patches are under development, e.g. by [http://www.kraxel.org/cgit/seabios/log/?h=kraxel.q35&id=99ff4d141d5a460eaf20fe2d4d13117c888eae15 Gerd Hoffmann]
* '''Non-standard I/O region remapping''' At least NVIDIA adapters of the Quadro series a known to be affected by this, but probably only during early boot. A workaround could be identity mapping, i.e. locating the memory BARs in the guest at the same locations as on the host. Of course, this breaks if guest OS or guest driver decided to remap them for some reason.
* '''Legacy VGA support''' In order to get boot messages of the guest BIOS, boot loader, or early OS stages, access to the legacy VGA PIO and MMIO regions need to be forwarded to the passed-through adapter. Moreover, the VBIOS of the adapter must be bootable inside the guest. In details, the following aspects need to be addressed:
** '''Legacy VGA pass-through''' QEMU requires patches to forward guest access to legacy VGA to the assigned adapter.
** '''PAM/SRAM QEMU breakage''' Current QEMU contains broken code that prevents access to legacy VGA even if the adapter is emulated but differently initialized (test case: -vga none -device VGA)
** '''Legacy VGA arbitration''' As the legacy IO resources cannot be remapped, access need to be switched between guests and host on demand. The Linux kernel provides a VGA arbiter for this purpose, but it is not used consistently by all involved parties. Moreover, QEMU requires additional patches to support this.
** '''VBIOS ROM access''' To re-run the POST procedures of the assigned adapter inside the guest, the proper VBIOS ROM image has to be used. However, when passing through the primary adapter of the host, Linux provides only access to the shadowed version of the VBIOS which may differ from the pre-POST version (due to modification applied during POST). This has be been observed with NVDIA Quadro adapters. A workaround is to retrieve the VBIOS from the adapter while it is in secondary mode and use this saved image (-device pci-assign,...,romfile=...). But even that may fail, either due to problems of the host chipset or BIOS (host kernel complains about unmappable ROM BAR).

== Potential tweaks & tricks ==

* Some adapters may advertise MSI support even if that feature does not work properly. In that case, KVM will use MSI on the host side unconditionally unless instructed to follow the guest driver usage via -device pci-assign,...,prefer_msi=off.

== Success stories ==

Despite all existing problems, some users already succeeded passing through VGA adapters:

* [http://thread.gmane.org/gmane.comp.emulators.kvm.devel/72987 Radeon HD 6950 with patched SeaBIOS]
* [http://thread.gmane.org/gmane.comp.emulators.kvm.devel/71981 Radeon HD 5850]

How to assign devices with VT-d in KVM

2010-11-02T16:44:55Z

Jan: Update command line switch and monitor commands to new syntax

= How to assign devices with VT-d in KVM =

== VT-d support ==
* In order to assign devices in KVM, you'll need a system which supports VT-d. This has nothing to do with the VT-x support of your CPU, VT-d needs to be supported by your chipset on your motherboard.

* If you are in doubt whether your motherboard supports VT-d or not, the Xen VT-d wikipage has some pointers of VT-d enabled chipsets and motherboards:
http://wiki.xensource.com/xenwiki/VTdHowTo

* If your hardware doesn't have an IOMMU ("Intel VT-d" support in case of Intel - "AMD I/O Virtualization Technology" support in case of AMD), you'll not be able to assign devices in KVM. Some work towards allowing this were done, but the code never made it into KVM, due to various issues with the code. At the moment it doesn't seem like device assignment without hardware support, will ever be integrated into KVM.

* Assignment of graphics cards are not supported at the moment, but it seems like one person is currently working on writing patches for this in his spare time (February, 2010):
http://www.spinics.net/lists/kvm/msg25977.html

== Assigning device to guest ==
'''1. Modifying kernel config:'''
* make menuconfig
* set "Bus options (PCI etc.)" -> "Support for DMA Remapping Devices" to "*"
* set "Bus options (PCI etc.)" -> "Enable DMA Remapping Devices" to "*"
* set "Bus options (PCI etc.)" -> "PCI Stub driver" to "*"
* optional setting:
set "Bus options (PCI etc.)" -> "Support for Interrupt Remapping" to "*"
* exit/save

'''2. build kernel:'''
* make
* make modules_install
* make install

'''3. reboot and verify that your system has IOMMU support'''
* dmesg | grep -e DMAR -e IOMMU
...
DMAR:DRHD base: 0x000000feb03000 flags: 0x0
IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000
...
* If you get no output you'll need to fix this before moving on. Check if your hardware supports VT-d and check that it has been enabled in BIOS.

'''NOTE:''' If you still get an error "No IOMMU found." Check dmesg for errors suggesting your BIOS is broken. Another possible reason:
CONFIG_DMAR_DEFAULT_ON is not set. In that case, pass "intel_iommu=on" as kernel parameter to enable it.

'''4. unbind device from host kernel driver (example PCI device 01:00.0)
'''
* lspci -n
* locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9
...
01:00.0 0200: 8086:10b9 (rev 06)
...

* echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
* echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
* echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

'''5. load KVM modules:'''
* modprobe kvm
* modprobe kvm-intel

'''6. assign device:'''
* /usr/local/bin/qemu-system-x86_64 -m 512 -boot c -net none -hda /root/ia32e_rhel5u1.img -device pci-assign,host=01:00.0

== VT-d device hotplug ==
KVM also supports hotplug devices with VT-d to guest. In guest command interface (you can press Ctrl+Alt+2 to enter it), you can use following command to hot add/remove devices to/from guest:
* hot add:
device_add pci-assign,host=01:00.0,id=mydevice
* hot remove:
device_del mydevice

== Notes ==
* VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
* If the device doesn't support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.

How to assign devices with VT-d in KVM

2010-11-02T16:36:25Z

Jan: Explain the CONFIG_DMAR_DEFAULT_ON trap

= How to assign devices with VT-d in KVM =

== VT-d support ==
* In order to assign devices in KVM, you'll need a system which supports VT-d. This has nothing to do with the VT-x support of your CPU, VT-d needs to be supported by your chipset on your motherboard.

* If you are in doubt whether your motherboard supports VT-d or not, the Xen VT-d wikipage has some pointers of VT-d enabled chipsets and motherboards:
http://wiki.xensource.com/xenwiki/VTdHowTo

* If your hardware doesn't have an IOMMU ("Intel VT-d" support in case of Intel - "AMD I/O Virtualization Technology" support in case of AMD), you'll not be able to assign devices in KVM. Some work towards allowing this were done, but the code never made it into KVM, due to various issues with the code. At the moment it doesn't seem like device assignment without hardware support, will ever be integrated into KVM.

* Assignment of graphics cards are not supported at the moment, but it seems like one person is currently working on writing patches for this in his spare time (February, 2010):
http://www.spinics.net/lists/kvm/msg25977.html

== Assigning device to guest ==
'''1. Modifying kernel config:'''
* make menuconfig
* set "Bus options (PCI etc.)" -> "Support for DMA Remapping Devices" to "*"
* set "Bus options (PCI etc.)" -> "Enable DMA Remapping Devices" to "*"
* set "Bus options (PCI etc.)" -> "PCI Stub driver" to "*"
* optional setting:
set "Bus options (PCI etc.)" -> "Support for Interrupt Remapping" to "*"
* exit/save

'''2. build kernel:'''
* make
* make modules_install
* make install

'''3. reboot and verify that your system has IOMMU support'''
* dmesg | grep -e DMAR -e IOMMU
...
DMAR:DRHD base: 0x000000feb03000 flags: 0x0
IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000
...
* If you get no output you'll need to fix this before moving on. Check if your hardware supports VT-d and check that it has been enabled in BIOS.

'''NOTE:''' If you still get an error "No IOMMU found." Check dmesg for errors suggesting your BIOS is broken. Another possible reason:
CONFIG_DMAR_DEFAULT_ON is not set. In that case, pass "intel_iommu=on" as kernel parameter to enable it.

'''4. unbind device from host kernel driver (example PCI device 01:00.0)
'''
* lspci -n
* locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9
...
01:00.0 0200: 8086:10b9 (rev 06)
...

* echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
* echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
* echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

'''5. load KVM modules:'''
* modprobe kvm
* modprobe kvm-intel

'''6. assign device:'''
* /usr/local/bin/qemu-system-x86_64 -m 512 -boot c -net none -hda /root/ia32e_rhel5u1.img -pcidevice host=01:00.0

== VT-d device hotplug ==
KVM also supports hotplug devices with VT-d to guest. In guest command interface (you can press Ctrl+Alt+2 to enter it), you can use following command to hot add/remove devices to/from guest:
* hot add:
pci_add pci_addr=auto host host=01:00.0
* hot remove (e.g bdf is 00:06.0 in guest):
pci_del pci_addr=6

== Notes ==
* VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
* If the device doesn't support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.

KvmForum2010CFP

2010-04-21T15:59:26Z

Jan: Copy&pasted the CFP

=================================================================

CALL FOR PAPERS

KVM Forum 2010

=================================================================

DESCRIPTION

The KVM Forum is back! After a break last year we're proud to present
this year's gathering around KVM again. The idea is to have everyone
involved with KVM development come together to talk about the future
and current state of KVM, teaching everyone some pieces of the puzzle
they might be missing without.

So if you're a KVM developer, mark the dates in your calendar! If
possible, also submit a talk -- we're interested in a wide variety of
KVM topics, so don't hesitate to propose a talk on your work.

If you're not a KVM developer, please read on nevertheless (or jump to
END USER COLLABORATION).

DATES / LOCATION

Conference: August 9 - 10, 2010
Location: Renaissance Boston Waterfront in Boston, MA

Abstracts due: May 14th, 2010
Notification: May 28th, 2010

Yes, we're colocated with LinuxCon. Tickets for the KVM Forum also count
for LinuxCon.

http://events.linuxfoundation.org/component/registrationpro/?func=details&did=34

PROCESS

At first check if it's before May 14th. If you're past that date, you're
out of luck. Now try to think hard and come up with a great idea that
you could talk about. Once you have that set, we need you to write up
a short abstract (~150 words) on it. In your submission please note
how long your talk will take. Slots vary in length up to one hour.
Also include in your proposal the proposal type -- one of: technical talk,
breakout session, or end-user talk. Add that information to the abstract
and submit it at the following URL:

http://events.linuxfoundation.org/cfp/cfp-add

Now, wait until May 24th. You will receive a notification on whether
your talk was accepted or not.

SCOPE OF TALKS

We have a list of suggested presentation topics below. These suggestions
are just for guidance, please feel free to submit a proposal on any
of these or related topics. In general, the more it's about backend
infrastructure, the better.

KVM
- Scaling and performance
- Nested virtualization
- I/O improvements
- Driver domains
- Time keeping
- Memory management (page sharing, swapping, huge pages, etc)
- Fault tolerance
- VEPA, vswitch

Embedded KVM
- KVM on ARM, PPC, MIPS, ...?
- Real-time requirements host/guest
- Device pass-through w/o iommu
- Custom device/platform models

QEMU
- Device model improvements
- New devices
- Security model
- Scaling and performance
- Desktop virtualization
- Increasing robustness
- Management interfaces
- QMP protocol and implementation
- Live migration

Virtio
- Speeding up existing devices
- Vhost
- Alternatives
- Using virtio in non-kvm environments
- Virtio on non-Linux

Management infrastructure
- Libvirt
- Kvm autotest
- Easy networking
- Qemud

BREAKOUT SESSION

We will reserve some time each day to break out for working sessions.
These sessions will be less formal than a presentation and more focused
on developing a solution to some real development issue. If you are
interested in getting developers together to hack on some code, submit
your proposal and just make it clear it's a breakout session proposal.

END-USER COLLABORATION

One of the big challenges as developers is to know what, where and how
people actually use our software. To solve this issue at least a little,
there will be a few slots reserved for end users talking about their
deployments, problems and achievements.

So if you have a KVM based deployment running in production or are about
to roll out one, please also submit a talk (see PROCESS), and simply
mark it asn an end-user collaboration proposal. We would love to have
an open discussion of fields where KVM/Qemu can still improve and you
would have the unique chance to steer that process!

Keep in mind that most of the Forum will be focused on development though,
so we suggest you also come with a good portion of technical interest :-) .
And of course, no product marketing please! The purpose is to engage
with KVM developers.

LIGHTNING TALKS

In addition to submitted talks we will also have some room for lightning
talks. So if you have something you think might be done until the KVM
Forum, but you're not sure you could fill 15 minutes with it. Or if you
don't know if you'll make it until there, just keep in mind that you
will still get the chance to talk about it. Lightning talk submissions
and scheduling will be handled on-site at KVM Forum.

Thank you for your interest in KVM. We're looking forward to your
submissions and seeing you at the KVM Forum 2010 in August! Now, start
thinking about that talk you want to give.

Thanks,
your KVM Forum 2010 Program Commitee

Alexander Graf, Novell
Anthony Liguori, IBM
Avi Kivity, Red Hat
Chris Wright, Red Hat
Dor Laor, Red Hat
Jan Kiszka, Siemens

Please contact us with any questions or comments.

KVM-Forum-2010-PC@redhat.com