From KVM
m (Added IOMMU messages on AMD)
m (VT-d device hotplug)
 
(9 intermediate revisions by 6 users not shown)
Line 2: Line 2:
  
 
== VT-d support ==
 
== VT-d support ==
* In order to assign devices in KVM, you'll need a system which supports VT-d. This has nothing to do with the VT-x support of your CPU, VT-d needs to be supported by both your chipset on your motherboard and by your CPU.
+
* In order to assign devices in KVM, you'll need a system which supports VT-d, not to be confused with the VT-x support of your CPU. VT-d needs to be supported by both your motherboard chipset and your CPU.
  
* If you are in doubt whether your motherboard or CPU supports VT-d or not, the Xen VT-d wikipage has some pointers of VT-d enabled chipsets, motherboards and CPUs:
+
* If you are in doubt whether your motherboard or CPU supports VT-d, the Xen VT-d wiki has some information about VT-d enabled chipsets, motherboards, and CPUs:
         http://wiki.xensource.com/xenwiki/VTdHowTo
+
         http://wiki.xenproject.org/wiki/VTdHowTo
  
* If your hardware doesn't have an IOMMU ("Intel VT-d" support in case of Intel - "AMD I/O Virtualization Technology" support in case of AMD), you'll not be able to assign devices in KVM. Some work towards allowing this were done, but the code never made it into KVM, due to various issues with the code. At the moment it doesn't seem like device assignment without hardware support, will ever be integrated into KVM.
+
* If your hardware does not have an IOMMU (known as "Intel VT-d" on Intel-based machines and "AMD I/O Virtualization Technology" on AMD-based machines), you will not be able to assign devices in KVM.
  
* Assignment of graphics cards are not supported at the moment, but it seems like one person is currently working on writing patches for this in his spare time (February, 2010):
+
* Assigning graphics cards is not officially supported at the moment, but there has been some success passing through a secondary Radeon HD 5850 as a VM's secondary display.
        http://www.spinics.net/lists/kvm/msg25977.html
+
  
 
+
== Assigning the device ==
== Assigning device to guest ==
+
'''1. Modify the kernel config'''
'''1. Modifying kernel config:'''
+
* Navigate to the kernel source code directory (usually located in /usr/src/linux) and configure the kernel:
* make menuconfig
+
      cd /usr/src/linux
* set "Bus options (PCI etc.)" -> "Support for DMA Remapping Devices" to "*"
+
      make menuconfig
* set "Bus options (PCI etc.)" -> "Enable DMA Remapping Devices" to "*"
+
* Navigate to "Bus options (PCI etc.)" and enable the following options:
* set "Bus options (PCI etc.)" -> "PCI Stub driver" to "*"
+
      Support for DMA Remapping Devices
* optional setting:
+
      Enable DMA Remapping Devices
        set "Bus options (PCI etc.)" -> "Support for Interrupt Remapping" to "*"
+
      PCI Stub driver
* exit/save
+
* Optionally, this can also be enabled:
 +
      Support for Interrupt Remapping
 +
* Save the changes and exit
 
   
 
   
'''2. build kernel:'''
 
* make
 
* make modules_install
 
* make install
 
  
'''3. reboot and verify that your system has IOMMU support'''
+
'''2. Build the kernel'''
* AMD Machine
+
      make
** dmesg | grep IOMMU
+
      make modules_install
 +
      make install
 +
 
 +
 
 +
'''3. Reboot and verify that your system has IOMMU support'''
 +
* AMD-based machines:
 +
      dmesg | grep AMD-Vi
 
       ...
 
       ...
 
       AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
 
       AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
 
       AMD-Vi: Lazy IO/TLB flushing enabled
 
       AMD-Vi: Lazy IO/TLB flushing enabled
 +
      AMD-Vi: Initialized for Passthrough Mode
 
       ...
 
       ...
* Intel Machine
+
* Intel-based machines:
** dmesg | grep -e DMAR -e IOMMU
+
      dmesg | grep -e DMAR -e IOMMU
 
       ...
 
       ...
 
       DMAR:DRHD base: 0x000000feb03000 flags: 0x0
 
       DMAR:DRHD base: 0x000000feb03000 flags: 0x0
 
       IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000
 
       IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000
 
       ...
 
       ...
* If you get no output you'll need to fix this before moving on. Check if your hardware supports VT-d and check that it has been enabled in BIOS.
+
Note: If you get no output, you will need to fix this before moving on. Try the following:
 +
* Verify that your hardware supports VT-d and that it has been enabled in the BIOS.
 +
* Check dmesg for errors suggesting that the BIOS is broken.
 +
* CONFIG_DMAR_DEFAULT_ON is not set. Pass one of the following commands as a kernel parameter:
 +
      intel_iommu=on      # Intel only
 +
      iommu=pt iommu=1    # AMD only
 +
Note: The kernel parameter can be passed temporarily using the GRUB menu by highlighting the OS, pressing "e", and appending the parameter to the end of the line beginning with "linux".  
  
'''NOTE:''' If you still get an error "No IOMMU found."  Check dmesg for errors suggesting your BIOS is broken. Another possible reason:
+
'''4. Unbind the device from the host kernel driver (Example: PCI device 01:00.0)
CONFIG_DMAR_DEFAULT_ON is not set. In that case, pass "intel_iommu=on" as kernel parameter to enable it.
+
 
+
'''4. unbind device from host kernel driver (example PCI device 01:00.0)
+
 
'''
 
'''
 
* Load the PCI Stub Driver if it is compiled as a module
 
* Load the PCI Stub Driver if it is compiled as a module
        modprobe pci_stub
+
      modprobe pci_stub
* lspci -n
+
      lspci -n
* locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9
+
        ...
+
        01:00.0 0200: 8086:10b9 (rev 06)
+
        ...
+
  
* echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
+
* Locate the entry for device 01:00.0 and note the vendor and device ID (8086:10b9 in this example)
* echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
+
      ...
* echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
+
      01:00.0 0200: 8086:10b9 (rev 06)
 +
      ...
  
'''5. load KVM modules:'''
+
* Place this information in the following files:
* modprobe kvm
+
      echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
* modprobe kvm-intel
+
      echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
 +
      echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/bind
  
'''6. assign device:'''
 
* /usr/local/bin/qemu-system-x86_64 -m 512 -boot c -net none -hda /root/ia32e_rhel5u1.img -device pci-assign,host=01:00.0
 
  
 +
'''5. Load the KVM modules'''
 +
      modprobe kvm
 +
      modprobe kvm-intel
  
== VT-d device hotplug ==
 
KVM also supports hotplug devices with VT-d to guest. In guest command interface (you can press Ctrl+Alt+2 to enter it), you can use following command to hot add/remove devices to/from guest:
 
* hot add:
 
        device_add pci-assign,host=01:00.0,id=mydevice
 
* hot remove:
 
        device_del mydevice
 
  
 +
'''6. Assign the device'''
 +
      /usr/local/bin/qemu-system-x86_64 -m 512 -boot c -net none -hda /root/ia32e_rhel5u1.img -device pci-assign,host=01:00.0
 +
 +
== VT-d device hotplug ==
 +
With VT-d, KVM also supports hotplugging devices on the guest. In the guest command interface (accessible with Ctrl+Alt+2), you can use the following commands to add/remove devices on the guest:
 +
* Add
 +
    device_add pci-assign,host=01:00.0,id=mydevice
 +
* Remove
 +
    device_del mydevice
  
 
== Notes ==
 
== Notes ==
 
* VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
 
* VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
 
* If the device doesn't support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.
 
* If the device doesn't support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.
 +
 +
 +
[[Category:Docs]][[Category:HowTo]][[Category:Devices]]

Latest revision as of 00:34, 25 July 2016

How to assign devices with VT-d in KVM

VT-d support

  • In order to assign devices in KVM, you'll need a system which supports VT-d, not to be confused with the VT-x support of your CPU. VT-d needs to be supported by both your motherboard chipset and your CPU.
  • If you are in doubt whether your motherboard or CPU supports VT-d, the Xen VT-d wiki has some information about VT-d enabled chipsets, motherboards, and CPUs:
       http://wiki.xenproject.org/wiki/VTdHowTo
  • If your hardware does not have an IOMMU (known as "Intel VT-d" on Intel-based machines and "AMD I/O Virtualization Technology" on AMD-based machines), you will not be able to assign devices in KVM.
  • Assigning graphics cards is not officially supported at the moment, but there has been some success passing through a secondary Radeon HD 5850 as a VM's secondary display.

Assigning the device

1. Modify the kernel config

  • Navigate to the kernel source code directory (usually located in /usr/src/linux) and configure the kernel:
     cd /usr/src/linux
     make menuconfig
  • Navigate to "Bus options (PCI etc.)" and enable the following options:
     Support for DMA Remapping Devices
     Enable DMA Remapping Devices
     PCI Stub driver
  • Optionally, this can also be enabled:
     Support for Interrupt Remapping
  • Save the changes and exit


2. Build the kernel

     make
     make modules_install
     make install


3. Reboot and verify that your system has IOMMU support

  • AMD-based machines:
     dmesg | grep AMD-Vi
     ...
     AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
     AMD-Vi: Lazy IO/TLB flushing enabled
     AMD-Vi: Initialized for Passthrough Mode
     ...
  • Intel-based machines:
     dmesg | grep -e DMAR -e IOMMU
     ...
     DMAR:DRHD base: 0x000000feb03000 flags: 0x0
     IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000
     ...

Note: If you get no output, you will need to fix this before moving on. Try the following:

  • Verify that your hardware supports VT-d and that it has been enabled in the BIOS.
  • Check dmesg for errors suggesting that the BIOS is broken.
  • CONFIG_DMAR_DEFAULT_ON is not set. Pass one of the following commands as a kernel parameter:
     intel_iommu=on      # Intel only
     iommu=pt iommu=1    # AMD only

Note: The kernel parameter can be passed temporarily using the GRUB menu by highlighting the OS, pressing "e", and appending the parameter to the end of the line beginning with "linux".

4. Unbind the device from the host kernel driver (Example: PCI device 01:00.0)

  • Load the PCI Stub Driver if it is compiled as a module
     modprobe pci_stub
     lspci -n
  • Locate the entry for device 01:00.0 and note the vendor and device ID (8086:10b9 in this example)
     ...
     01:00.0 0200: 8086:10b9 (rev 06)
     ...
  • Place this information in the following files:
     echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
     echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
     echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/bind


5. Load the KVM modules

     modprobe kvm
     modprobe kvm-intel


6. Assign the device

     /usr/local/bin/qemu-system-x86_64 -m 512 -boot c -net none -hda /root/ia32e_rhel5u1.img -device pci-assign,host=01:00.0

VT-d device hotplug

With VT-d, KVM also supports hotplugging devices on the guest. In the guest command interface (accessible with Ctrl+Alt+2), you can use the following commands to add/remove devices on the guest:

  • Add
   device_add pci-assign,host=01:00.0,id=mydevice
  • Remove
   device_del mydevice

Notes

  • VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
  • If the device doesn't support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.