https://linux-kvm.org/api.php?action=feedcontributions&user=Fghaas&feedformat=atomKVM - User contributions [en]2024-03-28T10:42:54ZUser contributionsMediaWiki 1.39.5https://linux-kvm.org/index.php?title=Nested_Guests&diff=173900Nested Guests2018-02-12T14:01:37Z<p>Fghaas: Add link to kernel.org bug 53851</p>
<hr />
<div>= Nested Guests =<br />
<br />
Nested guests are KVM guests run in a KVM guest. As of Feb 2018 this feature is considered working but experimental, and some limitations apply.<br />
<br />
When describing nested guests, we will use the following terminology:<br />
<br />
* "L0" – the bare metal host, running KVM<br />
* "L1" – a VM running on L0; also called the "guest hypervisor" — as it ''itself'' is capable of running KVM<br />
* "L2" – a VM running on L1, also called the "nested guest"<br />
<br />
<br />
= Detailed =<br />
== Why use it? ==<br />
<br />
An additional layer of virtualization sometimes comes in handy. You might have access to a large virtual machine in a cloud environment that you want to compartmentalize into multiple workloads. You might be running a lab environment in a training session.<br />
<br />
== How to run ==<br />
<br />
The KVM kernel modules do not enable nesting by default (though your distribution may override this default). To enable nesting, set the <code>nested</code> module parameter to <code>Y</code> or <code>1</code>. You may set this parameter persistently in a file in <code>/etc/modprobe.d</code> in the L0 host, for example:<br />
<br />
# If you have an Intel CPU, use this:<br />
$ cat /etc/modprobe.d/kvm_intel.conf<br />
options kvm-intel nested=Y<br />
<br />
# If you have an AMD CPU, then this:<br />
$ cat /etc/modprobe.d/kvm_amd.conf<br />
options kvm-amd nested=1<br />
<br />
Once your L0 host is capable of nesting, you should be able to start an L1 guest with the <code>-cpu host</code> option (or for better live migration compatibility, use a named CPU model supported by QEMU, such as: <code>-cpu Haswell-noTSX-IBRS,vmx=on</code>) and the guest will subsequently be capable of running an L2 guest with accelerated KVM.<br />
<br />
<br />
== Limitations ==<br />
<br />
Once an L1 guest has started an L2 guest, it is no longer capable of being migrated, saved, or loaded (see [[Migration]] for details on these actions) until the L2 guest shuts down. This is currently an inherent limitation (that is being worked on, as of Feb 2018) of the KVM implementation on all architectures except s390x.<br />
<br />
Attempting to migrate or save & load an L1 guest while an L2 guest is running will result in undefined behavior. You might see a <code>kernel BUG!</code> entry in <code>dmesg</code>, a kernel oops, or an outright kernel panic. At any rate, a thus migrated or loaded L1 guest can no longer be considered stable or secure, and must be restarted.<br />
<br />
Migrating an L1 guest merely ''configured to support'' nesting, while not ''actually'' running L2 guests, is expected to function normally. Live-migrating an L2 guest from one L1 guest to another is also expected to succeed.<br />
<br />
= Links =<br />
* [https://www.redhat.com/archives/libvirt-users/2018-February/msg00014.html Discussion on limitations of nested guests] (<code>libvirt-users</code> mailing list, February 2018)<br />
* [https://bugzilla.redhat.com/show_bug.cgi?id=1076294 Red Hat Bugzilla Bug 1076294 ]<br />
* [https://bugzilla.kernel.org/show_bug.cgi?id=53851 kernel.org Bugzilla Bug 53851]<br />
* [https://bugzilla.kernel.org/show_bug.cgi?id=198621 kernel.org Bugzilla Bug 198621]<br />
* [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/virtual/kvm/nested-vmx.txt Intel nVMX upstream specification]<br />
[[Category:Docs]]</div>Fghaashttps://linux-kvm.org/index.php?title=Nested_Guests&diff=173896Nested Guests2018-02-08T17:45:38Z<p>Fghaas: Fix title</p>
<hr />
<div>= Nested Guests =<br />
<br />
Nested guests are KVM guests run in a KVM guest. As of 2018 this feature is considered working but experimental, and some limitations apply.<br />
<br />
When describing nested guests, the following conventions apply:<br />
<br />
* "L0" is an unvirtualized host system capable of running KVM.<br />
* "L1" is a virtual system running on L0, which is itself ''also'' capable of running KVM.<br />
* "L2" is a virtual system running on L1, which does no further virtualization.<br />
<br />
In this context, we refer to L1 as an unnested guest, and L2 as a nested guest.<br />
<br />
= Detailed =<br />
== Why use it? ==<br />
<br />
An additional layer of virtualization sometimes comes in handy. You might have access to a large virtual machine in a cloud environment that you want to compartmentalize into multiple workloads. You might be running a lab environment in a training session.<br />
<br />
== How to run ==<br />
<br />
The KVM kernel modules do not enable nesting by default (though your distribution may override this default). To enable nesting, set the <code>nested</code> module parameter to <code>Y</code> or <code>1</code>. You may set this parameter persistently in a file in <code>/etc/modprobe.d</code> in the L0 host, for example:<br />
<br />
# cat /etc/modprobe.d/kvm_intel.conf<br />
options kvm-intel nested=Y<br />
<br />
# cat /etc/modprobe.d/kvm_amd.conf<br />
options kvm-amd nested=1<br />
<br />
Once your L0 host is capable of nesting, you should be able to start an L1 guest with the <code>-cpu host</code> option, and the guest will subsequently be capable of running an L2 guest with accelerated KVM.<br />
<br />
<br />
== Limitations ==<br />
<br />
Once an L1 guest has started an L2 guest, it is no longer capable of being migrated, saved, or loaded (see [[Migration]] for details on these actions) until the L2 guest shuts down. This is currently an inherent limitation of the KVM implementation on all architectures except s390x.<br />
<br />
Attempting to migrate or save & load an L1 guest while an L2 guest is running will result in undefined behavior. You might see a <code>kernel BUG!</code> entry in <code>dmesg</code>, a kernel oops, or an outright kernel panic. At any rate, a thus migrated or loaded L1 guest can no longer be considered stable or secure, and must be restarted.<br />
<br />
Migrating an L1 guest merely ''configured to support'' nesting, while not ''actually'' running L2 guests, is expected to function normally. Live-migrating an L2 guest from one L1 guest to another is also expected to succeed.<br />
<br />
= Links =<br />
* [https://www.redhat.com/archives/libvirt-users/2018-February/msg00014.html Discussion on limitations of nested guests] (<code>libvirt-users</code> mailing list, February 2018)<br />
* [https://bugzilla.redhat.com/show_bug.cgi?id=1076294 Red Hat Bugzilla Bug 1076294 ]<br />
* [https://bugzilla.kernel.org/show_bug.cgi?id=198621 kernel.org Bugzilla Bug 198621]<br />
<br />
[[Category:Docs]]</div>Fghaashttps://linux-kvm.org/index.php?title=KVM_Features&diff=173895KVM Features2018-02-08T17:40:32Z<p>Fghaas: Add Nested Guests cross-reference</p>
<hr />
<div>= KVM-Features =<br />
<br />
This is a possibly incomplete list of KVM features, together with their status. Feel free to update any of them as you see fit.<br />
<br />
As a guideline, there is a feature description template in [[FeatureDescription/Template|here]]<br />
<br />
<br />
* [[MonitorProtocol|QMP]] - Qemu Monitor Protocol<br />
* [[KSM|KSM]] - Kernel Samepage Merging<br />
* [[KVMClock|Kvm Paravirtual Clock]] - A Paravirtual timesource for KVM<br />
* [[CPUHotPlug|CPU Hotplug support]] - Adding cpus on the fly<br />
* [[pcihotplug|PCI Hotplug support]] - Adding pci devices on the fly<br />
* [[VMchannel_Requirements | vmchannel ]] - Communication channel between the host and guests<br />
* [[migration]] - Migrating Virtual Machines<br />
* [[Nested Guests]] - Running virtual machines within virtual machines<br />
* [[VhostNet|vhost]] - <br />
* [[scsi|SCSI disk emulation]] - <br />
* [[virtio|Virtio Devices]] - <br />
* [[CPU clustering]] - <br />
* [[hpet]] - <br />
* [[Device assignment]] - <br />
* [[pxe boot]] - <br />
* [[iscsi boot]] - <br />
* [[x2apic]] -<br />
* [[Floppy]] -<br />
* [[cdrom|CDROM]] - <br />
* [[USB]] -<br />
* [[UsbPassthrough|USB host device passthrough]] - <br />
* [[Sound]] -<br />
* [[UserspaceIrqchip|Userspace irqchip emulation]] -<br />
* [[UserspacePit|Userspace pit emulation]] -<br />
* [[Balloon|Balloon memory driver]] - <br />
* [[LargePages|Large pages support]] -<br />
* [[StableABI|Stable Guest ABI]] -</div>Fghaashttps://linux-kvm.org/index.php?title=Migration&diff=173894Migration2018-02-08T17:40:21Z<p>Fghaas: /* Problems / Todo */ Mention limitations on nested guests, and add cross-reference to Nested Guests page</p>
<hr />
<div>=Migration=<br />
<br />
== Introduction ==<br />
KVM currently supports savevm/loadvm and offline or live migration<br />
Migration commands are given when in qemu-monitor (Alt-Ctrl-2). <br />
Upon successful completion, the migrated VM continues to run on the destination host.<br />
<br />
==== Note ====<br />
You can migrate a guest between an AMD host to an Intel host and back. Naturally, a 64-bit guest can only be migrated to a 64-bit host, but a 32-bit guest can be migrated at will.<br />
<br />
There are some older Intel processors which don't support NX (or XD), which may cause problems in a cluster which includes NX-supporting hosts. To disable NX for a given guest, start it with such a parameter: -cpu qemu64,-nx<br />
<br />
== Requirements ==<br />
* The VM image is accessible on both source and destination hosts (located on a shared storage, e.g. using nfs). <br />
* It is recommended an images-directory would be found on the same path on both hosts (for migrations of a copy-on-write image -- an image created on top of a base-image using "qemu-image create -b ...") <br />
* The src and dst hosts must be on the same subnet (keeping guest's network when tap is used). <br />
* Do not use -snapshot qemu command line option. <br />
* For tcp: migration protocol <br />
* the guest on the destination must be started the same way it was started on the source. <br />
<br />
== highlights / merits ==<br />
* Almost unnoticeable guest down time<br />
* Guest is not involved (unique to KVM Live Migration [#1 1])<br />
* Capability to tunnel VM state through an external program (unique to KVM Live Migration [#1 1])<br />
* ssh/gzip/bzip2/gpg/your own <br />
* Upon success guest continues to run on destination host, upon failure guest continues to run on source host (with one exception) <br />
* Short and Simple <br />
* Easy to enhance <br />
* Hardware independence (almost). <br />
* Support for migration of stopped (paused) VMs. <br />
* Open <br />
<br />
1 These features are unique to KVM Live Migration as far as I know. If you know of other hypervisor that support any of them please update this page or let me (Uri) know. <br />
<br />
== User Interface ==<br />
The user interface is through the qemu monitor (alt-ctrl-2 on the SDL window)<br />
<br />
=== Management ===<br />
migrate [-d] <URI><br />
migrate_cancel <br />
<br />
The '-d' will let you query the status of the migration.<br />
<br />
With no '-d' the monitor prompt returns when the migration completes.<br />
URI can be one of 'exec:<command>' or tcp:<ip:port> <br />
<br />
=== Status ===<br />
<br />
info migrate <br />
<br />
=== Migration Parameters ===<br />
migrate_set_speed <speed> set bandwidth control parameters (max transfer rate per second)<br />
<br />
== Example / HOWTO ==<br />
A is the source host, B is the destination host:<br />
===''TCP'' example:===<br />
1. Start the VM on B with the exact same parameters as the VM on A, in migration-listen mode:<br />
B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))<br />
2. Start the migration (always on the source host):<br />
A: migrate -d tcp:B:4444 (or other PORT)<br />
3. Check the status (on A only):<br />
A: (qemu) info migrate <br />
<br />
===''Offline'' example:===<br />
1. unlimit bandwidth used for migration:<br />
A: migrate_set_speed 1g <br />
2. stop the guest: <br />
A: stop <br />
3. continue with either TCP or exec migration as described above.<br />
<br />
== Problems / Todo ==<br />
* TSC offset on the new host must be set in such a way that the guest sees a monotonically increasing TSC, otherwise the guest may hang indefinitely after migration. <br />
* usbdevice tablet complains after migration. <br />
* handle migration completion better (especially when network problems occur). <br />
* More informative status. <br />
* Migration does not work while CPU real-mode/protected mode are still changing.<br />
* Migration does not work when running [[Nested Guests]]: if you are running a KVM guest that is itself running a KVM guest, migration of the unnested guest (including savevm/loadvm as described below) will result in undefined behavior.<br />
<br />
== savevm/loadvm to an external state file (using pseudo-migration) ==<br />
<br />
* To be supported directly by Migration Protocols, but until then...<br />
* Save VM state into a compressed file <br />
** Save <br />
stop <br />
migrate_set_speed 4095m <br />
migrate "exec:gzip -c > STATEFILE.gz" <br />
<br />
** Load<br />
gzip -c -d STATEFILE.gz | <qemu-command-line> -incoming "exec: cat" or<br />
<qemu-command-line> -incoming "exec: gzip -c -d STATEFILE.gz"<br />
* Save VM State into an encrypted file (assumes KEY has already been generated)<br />
** Save <br />
stop <br />
migrate_set_speed 4095m<br />
migrate "exec:gpg -q -e -r KEY -o STATFILE.gpg"<br />
<br />
* Load VM state from an encrypted file<br />
gpg -q -d -r KEY STATEFILE.gpg | <qemu-command-line> -incoming "exec:cat"<br />
<br />
== more exec: options ==<br />
<br />
* Send encrypted VM state from host A to host B (Note: ssh is probably better, this is just for show)<br />
<br />
** on host A<br />
migrate "exec:gpg -q -e -r KEY | nc B 4444"<br />
<br />
** on host B<br />
nc -l 4444 | gpg -q -d -r KEY | <qemu-command-line> -incoming "exec:cat"<br />
<br />
== Algorithm (the short version) ==<br />
1. Setup<br />
* Start guest on destination, connect, enable dirty page logging and more<br />
2. Transfer Memory<br />
* Guest continues to run<br />
* Bandwidth limitation (controlled by the user)<br />
* First transfer the whole memory<br />
* Iteratively transfer all dirty pages (pages that were written to by the guest).<br />
3. Stop the guest<br />
* And sync VM image(s) (guest's hard drives).<br />
4. Transfer State<br />
* As fast as possible (no bandwidth limitation)<br />
* All VM devices' state and dirty pages yet to be transferred<br />
5. Continue the guest<br />
* On destination upon success<br />
** Broadcast "I'm over here" Ethernet packet to announce new location of NIC(s).<br />
* On source upon failure (with one exception).<br />
<br />
<br />
Instructions for kvm-13 and below: [[MigrationQemu0.8.2]].</div>Fghaashttps://linux-kvm.org/index.php?title=Nested_Guests&diff=173893Nested Guests2018-02-08T17:39:29Z<p>Fghaas: Add page on nested guests</p>
<hr />
<div>= Feature functional description =<br />
<br />
Nested guests are KVM guests run in a KVM guest. As of 2018 this feature is considered working but experimental, and some limitations apply.<br />
<br />
When describing nested guests, the following conventions apply:<br />
<br />
* "L0" is an unvirtualized host system capable of running KVM.<br />
* "L1" is a virtual system running on L0, which is itself ''also'' capable of running KVM.<br />
* "L2" is a virtual system running on L1, which does no further virtualization.<br />
<br />
In this context, we refer to L1 as an unnested guest, and L2 as a nested guest.<br />
<br />
= Detailed =<br />
== Why use it? ==<br />
<br />
An additional layer of virtualization sometimes comes in handy. You might have access to a large virtual machine in a cloud environment that you want to compartmentalize into multiple workloads. You might be running a lab environment in a training session.<br />
<br />
== How to run ==<br />
<br />
The KVM kernel modules do not enable nesting by default (though your distribution may override this default). To enable nesting, set the <code>nested</code> module parameter to <code>Y</code> or <code>1</code>. You may set this parameter persistently in a file in <code>/etc/modprobe.d</code> in the L0 host, for example:<br />
<br />
# cat /etc/modprobe.d/kvm_intel.conf<br />
options kvm-intel nested=Y<br />
<br />
# cat /etc/modprobe.d/kvm_amd.conf<br />
options kvm-amd nested=1<br />
<br />
Once your L0 host is capable of nesting, you should be able to start an L1 guest with the <code>-cpu host</code> option, and the guest will subsequently be capable of running an L2 guest with accelerated KVM.<br />
<br />
<br />
== Limitations ==<br />
<br />
Once an L1 guest has started an L2 guest, it is no longer capable of being migrated, saved, or loaded (see [[Migration]] for details on these actions) until the L2 guest shuts down. This is currently an inherent limitation of the KVM implementation on all architectures except s390x.<br />
<br />
Attempting to migrate or save & load an L1 guest while an L2 guest is running will result in undefined behavior. You might see a <code>kernel BUG!</code> entry in <code>dmesg</code>, a kernel oops, or an outright kernel panic. At any rate, a thus migrated or loaded L1 guest can no longer be considered stable or secure, and must be restarted.<br />
<br />
Migrating an L1 guest merely ''configured to support'' nesting, while not ''actually'' running L2 guests, is expected to function normally. Live-migrating an L2 guest from one L1 guest to another is also expected to succeed.<br />
<br />
= Links =<br />
* [https://www.redhat.com/archives/libvirt-users/2018-February/msg00014.html Discussion on limitations of nested guests] (<code>libvirt-users</code> mailing list, February 2018)<br />
* [https://bugzilla.redhat.com/show_bug.cgi?id=1076294 Red Hat Bugzilla Bug 1076294 ]<br />
* [https://bugzilla.kernel.org/show_bug.cgi?id=198621 kernel.org Bugzilla Bug 198621]<br />
<br />
[[Category:Docs]]</div>Fghaas