KVM - User contributions [en]

Migration

2009-06-09T16:41:54Z

Uril: replace '-incoming stdio' with '-incoming "exec:cat" + cleanup (mainly whitespace)

=Migration=

== Introduction ==
KVM currently supports savevm/loadvm and offline or live migration
Migration commands are given when in qemu-monitor (Alt-Ctrl-2).
Upon successful completion, the migrated VM continues to run on the destination host.

==== Note ====
You can migrate a guest between an AMD host to an Intel host and back. Naturally, a 64-bit guest can only be migrated to a 64-bit host, but a 32-bit guest can be migrated at will.

There are some older Intel processors which don't support NX (or XD), which may cause problems in a cluster which includes NX-supporting hosts. To disable NX for a given guest, start it with such a parameter: -cpu qemu64,-nx

== Requirements ==
* The VM image is accessible on both source and destination hosts (located on a shared storage, e.g. using nfs).
* It is recommended an images-directory would be found on the same path on both hosts (for migrations of a copy-on-write image -- an image created on top of a base-image using "qemu-image create -b ...")
* The src and dst hosts must be on the same subnet (keeping guest's network when tap is used).
* Do not use -snapshot qemu command line option.
* For tcp: migration protocol
* the guest on the destination must be started the same way it was started on the source.

== highlights / merits ==
* Almost unnoticeable guest down time
* Guest is not involved (unique to KVM Live Migration [#1 1])
* Capability to tunnel VM state through an external program (unique to KVM Live Migration [#1 1])
* ssh/gzip/bzip2/gpg/your own
* Upon success guest continues to run on destination host, upon failure guest continues to run on source host (with one exception)
* Short and Simple
* Easy to enhance
* Hardware independence (almost).
* Support for migration of stopped (paused) VMs.
* Open

[[Anchor(1)]] 1 These features are unique to KVM Live Migration as far as I know. If you know of other hypervisor that support any of them please update this page or let me (Uri) know.

== User Interface ==
The user interface is through the qemu monitor (alt-ctrl-2 on the SDL window)

=== Management ===
migrate [-d] <URI>
migrate_cancel

The '-d' will let you query the status of the migration.

With no '-d' the monitor prompt returns when the migration completes.
URI can be one of 'exec:<command>' or tcp:<ip:port>

=== Status ===

info migrate

=== Migration Parameters ===
migrate_set_speed <speed> set bandwidth control parameters (max transfer rate per second)

== Example / HOWTO ==
A is the source host, B is the destination host:
===''TCP'' example:===
1. Start the VM on B with the exact same parameters as the VM on A, in migration-listen mode:
B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
2. Start the migration (always on the source host):
A: migrate -d tcp:B:4444 (or other PORT)
3. Check the status (on A only):
A: (qemu) info migrate

===''Offline'' example:===
1. unlimit bandwidth used for migration:
A: migrate_set_speed 1g
2. stop the guest:
A: stop
3. continue with either TCP or exec migration as described above.

== Problems / Todo ==
* TSC offset on the new host must be set in such a way that the guest sees a monotonically increasing TSC, otherwise the guest may hang indefinitely after migration.
* usbdevice tablet complains after migration.
* handle migration completion better (especially when network problems occur).
* More informative status.
* Migration does not work while CPU real-mode/protected mode are still changing.

== savevm/loadvm to an external state file (using pseudo-migration) ==

* To be supported directly by Migration Protocols, but until then...
* Save VM state into a compressed file
** Save
stop
migrate_set_speed 4095m
migrate "exec:gzip -c > STATEFILE.gz"

** Load
gzip -c -d STATEFILE.gz | <qemu-command-line> -incoming "exec: cat" or
<qemu-command-line> -incoming "exec: gzip -c -d STATEFILE.gz"
* Save VM State into an encrypted file (assumes KEY has already been generated)
** Save
stop
migrate_set_speed 4095m
migrate "exec:gpg -q -e -r KEY -o STATFILE.gpg"

* Load VM state from an encrypted file
gpg -q -d -r KEY STATEFILE.gpg | <qemu-command-line> -incoming "exec:cat"

== more exec: options ==

* Send encrypted VM state from host A to host B (Note: ssh is probably better, this is just for show)

** on host A
migrate "exec:gpg -q -e -r KEY | nc B 4444"

** on host B
nc -l 4444 | gpg -q -d -r KEY | <qemu-command-line> -incoming "exec:cat"

== Algorithm (the short version) ==
1. Setup
* Start guest on destination, connect, enable dirty page logging and more
2. Transfer Memory
* Guest continues to run
* Bandwidth limitation (controlled by the user)
* First transfer the whole memory
* Iteratively transfer all dirty pages (pages that were written to by the guest).
3. Stop the guest
* And sync VM image(s) (guest's hard drives).
4. Transfer State
* As fast as possible (no bandwidth limitation)
* All VM devices' state and dirty pages yet to be transferred
5. Continue the guest
* On destination upon success
** Broadcast "I'm over here" Ethernet packet to announce new location of NIC(s).
* On source upon failure (with one exception).

Instructions for kvm-13 and below: [[MigrationQemu0.8.2]].

KVM-Autotest/TODO

2009-03-24T14:08:16Z

Uril:

= KVM-Autotest To-Do list =

== Framework ==

* <s>Verify MD5 sum of ISO image before starting a VM</s> [DONE]
* <s>Enable multiple user controlled port redirections (with -redir)</s> [DONE]
* <s>Implement functions to translate VM ports and addresses (e.g. VM.get_port(), VM.get_address())</s> [DONE]
* Support TAP networking with a user specified MAC address for each NIC
* <s>Add <tt>display</tt> parameter to enable user control of the QEMU display method (VNC/SDL/nographic)</s> [DONE]
* Implement VM.clone() -- to be used by tests that need to add new VMs by themselves
* kickstart installation support
* kvm-autotest auto resolve qemu-kvm backtraces
* Make sure ppm_utils.image_crop() handles bad input properly
* Enable tests to modify a VM's params (while it's down)
* Make tests shorter by using small utility functions that fail a test and display a failure message
* Improve the stepmaker data naming system (e.g. name PPM files according to their MD5 sums)
* Redirect guest monitor to serial and capture guest error messages written to console. Log the errors and also scan for specific problems (e.g. oops)
* Support an "expected to fail" param (at least in our html report)
* Log a gdb-backtrace for tests (executables) which generate a core dump file.
* Enable "guest-load" for a VMs before/while tests are running (e.g. migration of a VM, while a movie is playing on it)
* Add support for kvm-trace.
* Fail upon indentation/white-space problems in the configuration file
* Add a way to shutdown the VMs when the whole job completes (compared to doing nothing or shuting-down after every test).

== Tests ==

* Find out how to use Autotest's netperf test with guests
* If needed, write a netperf/iperf test
* Write a time drift test (pic/apic/pm_timer)
* Write a test which adds VMs until one of them becomes unresponsive, and records the maximum number of VMs successfully started.
* cpu hotplug - Glauber
* pci net hot plug -Glauber
* pci block hot plug -Glauber
* s3 hibernation - Gleb
* tap networking
* pxe boot * net types
* qcow1, qcow2, vmdk, vhd
* qcow2 internal snapshots
* <s>Run autotest tests inside Linux guest</s> [ALREADY DONE]
* Further migration
* Multiple nics
* ksm
* -vga stg, nographics
* kvm unit-tests
* Add unattended guest installations (using kickstart or answer-files)
* kvm_install -- keep src and build directories under <kvm_autotest_root> instead of under kvm_runtest_2.
* Stable tsc test - http://people.redhat.com/mingo/time-warp-test/MINI-HOWTO
* Guest installation with pv-block device/driver (for both windows and Linux)

== Misc ==

* Goal -- get accepted into upstream autotest
* Add docstrings to all functions that still lack them, including the ones in stepmaker.py, stepeditor.py and kvm_tests.py
* Add module headers and docstrings
* Document the setupssh.iso and setuptelnet.iso creation procedures in the wiki
* Rename all Windows ISOs currently used to their official MSDN names
* Move Windows-setup iso images into the windows directory.
* Share all the stepmaker data somehow so other people can debug stepfiles
* Fill the sections "Working with step files" and "Step file creation tips" in the wiki
* Document the kvm_install test in the "Tests" section of the wiki

== Bugs ==

None so far?

RunningKVM

2009-03-24T12:46:15Z

Uril: Use qemu-kvm when using a kvm package on RHEL or similar

== Prerequisites ==

You will need (see download section):

* kvm-''release''.tar.gz
* A VT capable Intel processor, or an SVM capable AMD processor
* qemu prerequisites:
** `zlib` libraries and headers
** `SDL` libraries and headers
** `alsa` libraries and headers (optional alsa support: disabled by default but can be enabled with --enable-alsa)
** `gnutls` libraries and headers (optional VNC TLS support: enabled by default but can be disabled with --disable-vnc-tls)
** kernel headers (on Fedora, the kernel-devel package)

On a debian etch system you can install the prerequisites with:
apt-get install gcc libsdl1.2-dev zlib1g-dev libasound2-dev linux-kernel-headers pkg-config libgnutls-dev

'''Note:''' When building from git, you also need gawk.

Please report problems (and successes) to the [[Lists,_IRC| mailing list]].

== Unpacking and configuring kvm components ==

You may wish to take a look at the ["Kernel-optimizations"] page.
There exists a [attachment:kvm-26-alt-grab.diff.gz patch] which will change the SDL keygrab combination from ctrl-alt to ctrl-alt-shift. It was written primarily to deal with the heavy use of ctrl-alt-delete in NT-based VMs.

If you are using a patched kernel (e.g. a recent -mm kernel or the kvm git tree), configure the kernel normally, boot into it, and:

tar xzf kvm-release.tar.gz
cd kvm-release
./configure --prefix=/usr/local/kvm --with-patched-kernel
make
sudo make install
sudo /sbin/modprobe kvm-intel
# or: sudo /sbin/modprobe kvm-amd

If you're ''not'' running a patched kernel:

tar xzf kvm-release.tar.gz
cd kvm-release
./configure --prefix=/usr/local/kvm
make
sudo make install
sudo /sbin/modprobe kvm-intel
# or: sudo /sbin/modprobe kvm-amd

'''Note:''' if sound doesn't play in the guest vm you can add --audio-drv-list="alsa oss" to ./configure as explained in http://www.linux-kvm.com/content/sound-problem-solved

== Creating a disk image for the guest ==
/usr/local/kvm/bin/qemu-img create -f qcow vdisk.img 10G

== Installing a guest operating system ==

sudo /usr/local/kvm/bin/qemu-system-x86_64 -hda vdisk.img -cdrom /path/to/boot-media.iso \
-boot d -m 384

(kvm doesn't make a distinction between i386 and x86_64 so even in i386 you should use `qemu-system-x86_64`)[[BR]]

If you're installing Windows, forcing your emulated machine to not have ACPI support by using -no-acpi could result in much faster installation and performance while running. See ["Windows ACPI Workaround"] for details.

If you have less than 1GB of memory don't use the -m 384 flag (which allocates 384 MB of RAM for the guest). For computers with 512MB of RAM it's safe to use -m 192, or even -m 128 (the default)

== Running the newly-installed guest ==
sudo /usr/local/kvm/bin/qemu-system-x86_64 vdisk.img -m 384

or a slightly more complicated example, where it is assumed that bridged networking is available on tap0; see ["Kernel-optimizations"] for some setup hints:

/usr/local/kvm/bin/qemu-system-x86_64 -hda xp-curr.img -m 512 -soundhw es1370 -no-acpi -snapshot -localtime -boot c -usb -usbdevice tablet -net nic,vlan=0,macaddr=00:00:10:52:37:48 -net tap,vlan=0,ifname=tap0,script=no

(kvm doesn't make a distinction between i386 and x86_64 so even in i386 you should use `qemu-system-x86_64`)

If you're on Debian Etch, substitute `kvm` for `qemu-system-x86_64` (thanks to fromport, soren and mael_). See also the entries under the label "Ubuntu" on the [[HOWTO]] page.
qemu-system-x86_64`

If you're on Fedora/RHEL/CentOS (and installed a kvm package and not built kvm yourself from source) then substitute '''qemu-kvm''' for '''qemu-system-x86_64'''

KVM-Autotest/TODO

2009-03-24T07:46:20Z

Uril: guest-installation with pv-block test

= KVM-Autotest To-Do list =

== Framework ==

* <s>Verify MD5 sum of ISO image before starting a VM</s> [DONE]
* <s>Enable multiple user controlled port redirections (with -redir)</s> [DONE]
* <s>Implement functions to translate VM ports and addresses (e.g. VM.get_port(), VM.get_address())</s> [DONE]
* Support TAP networking with a user specified MAC address for each NIC
* <s>Add <tt>display</tt> parameter to enable user control of the QEMU display method (VNC/SDL/nographic)</s> [DONE]
* Implement VM.clone() -- to be used by tests that need to add new VMs by themselves
* kickstart installation support
* kvm-autotest auto resolve qemu-kvm backtraces
* Make sure ppm_utils.image_crop() handles bad input properly
* Enable tests to modify a VM's params (while it's down)
* Make tests shorter by using small utility functions that fail a test and display a failure message
* Improve the stepmaker data naming system (e.g. name PPM files according to their MD5 sums)
* Redirect guest monitor to serial and capture guest error messages written to console
* Support an "expected to fail" param (at least in our html report)
* Log a gdb-backtrace for tests (executables) which generate a core dump file.
* Enable "guest-load" for a VMs before/while tests are running (e.g. migration of a VM, while a movie is playing on it)
* Add support for kvm-trace.
* Ignore, Warn, or even fail, upon indentation/white-space problems in the configuration file
* Add a way to shutdown the VMs when the whole job completes (compared to doing nothing or shuting-down after every test).

== Tests ==

* Find out how to use Autotest's netperf test with guests
* If needed, write a netperf/iperf test
* Write a time drift test
* Write a test which adds VMs until one of them becomes unresponsive, and records the maximum number of VMs successfully started.
* cpu hotplug - Glauber
* pci net hot plug -Glauber
* pci block hot plug -Glauber
* s3 hibernation - Gleb
* tap networking
* pxe boot * net types
* qcow1, qcow2, vmdk, vhd
* qcow2 internal snapshots
* <s>Run autotest tests inside Linux guest</s> [ALREADY DONE]
* Further migration
* Multiple nics
* ksm
* -vga stg, nographics
* kvm unit-tests
* Add unattended guest installations (using kickstart or answer-files)
* kvm_install -- keep src and build directories under <kvm_autotest_root> instead of under kvm_runtest_2.
* time drift test (pic/apic/pm_timer)
* Stable tsc test - http://people.redhat.com/mingo/time-warp-test/MINI-HOWTO
* Guest installation with pv-block device/driver (for both windows and Linux)

== Misc ==

* Goal -- get accepted into upstream autotest
* Add docstrings to all functions that still lack them, including the ones in stepmaker.py, stepeditor.py and kvm_tests.py
* Add module headers and docstrings
* Document the setupssh.iso and setuptelnet.iso creation procedures in the wiki
* Rename all Windows ISOs currently used to their official MSDN names
* Move Windows-setup iso images into the windows directory.
* Share all the stepmaker data somehow so other people can debug stepfiles
* Fill the sections "Working with step files" and "Step file creation tips" in the wiki
* Document the kvm_install test in the "Tests" section of the wiki

== Bugs ==

None so far?

KVM-Autotest/TODO

2009-03-23T14:46:00Z

Uril: infrastructure: config-file-white-space-issue and job-postprocess-for-shutdown-VMs

= KVM-Autotest To-Do list =

== Framework ==

* <s>Verify MD5 sum of ISO image before starting a VM</s> [DONE]
* <s>Enable multiple user controlled port redirections (with -redir)</s> [DONE]
* <s>Implement functions to translate VM ports and addresses (e.g. VM.get_port(), VM.get_address())</s> [DONE]
* Support TAP networking with a user specified MAC address for each NIC
* <s>Add <tt>display</tt> parameter to enable user control of the QEMU display method (VNC/SDL/nographic)</s> [DONE]
* Implement VM.clone() -- to be used by tests that need to add new VMs by themselves
* kickstart installation support
* kvm-autotest auto resolve qemu-kvm backtraces
* Make sure ppm_utils.image_crop() handles bad input properly
* Enable tests to modify a VM's params (while it's down)
* Make tests shorter by using small utility functions that fail a test and display a failure message
* Improve the stepmaker data naming system (e.g. name PPM files according to their MD5 sums)
* Redirect guest monitor to serial and capture guest error messages written to console
* Support an "expected to fail" param (at least in our html report)
* Log a gdb-backtrace for tests (executables) which generate a core dump file.
* Enable "guest-load" for a VMs before/while tests are running (e.g. migration of a VM, while a movie is playing on it)
* Add support for kvm-trace.
* Ignore, Warn, or even fail, upon indentation/white-space problems in the configuration file
* Add a way to shutdown the VMs when the whole job completes (compared to doing nothing or shuting-down after every test).

== Tests ==

* Find out how to use Autotest's netperf test with guests
* If needed, write a netperf/iperf test
* Write a time drift test
* Write a test which adds VMs until one of them becomes unresponsive, and records the maximum number of VMs successfully started.
* cpu hotplug - Glauber
* pci net hot plug -Glauber
* pci block hot plug -Glauber
* s3 hibernation - Gleb
* tap networking
* pxe boot * net types
* qcow1, qcow2, vmdk, vhd
* qcow2 internal snapshots
* <s>Run autotest tests inside Linux guest</s> [ALREADY DONE]
* Further migration
* Multiple nics
* ksm
* -vga stg, nographics
* kvm unit-tests
* Add unattended guest installations (using kickstart or answer-files)
* kvm_install -- keep src and build directories under <kvm_autotest_root> instead of under kvm_runtest_2.

== Misc ==

* Goal -- get accepted into upstream autotest
* Add docstrings to all functions that still lack them, including the ones in stepmaker.py, stepeditor.py and kvm_tests.py
* Add module headers and docstrings
* Document the setupssh.iso and setuptelnet.iso creation procedures in the wiki
* Rename all Windows ISOs currently used to their official MSDN names
* Move Windows-setup iso images into the windows directory.
* Share all the stepmaker data somehow so other people can debug stepfiles
* Fill the sections "Working with step files" and "Step file creation tips" in the wiki
* Document the kvm_install test in the "Tests" section of the wiki

== Bugs ==

None so far?

KVM-Autotest/TODO

2009-03-22T16:48:33Z

Uril: kvm_install: keep src and build directories under <kvm_autotest_root>

= KVM-Autotest To-Do list =

== Framework ==

* <s>Verify MD5 sum of ISO image before starting a VM</s> [DONE]
* <s>Enable multiple user controlled port redirections (with -redir)</s> [DONE]
* <s>Implement functions to translate VM ports and addresses (e.g. VM.get_port(), VM.get_address())</s> [DONE]
* Support TAP networking with a user specified MAC address for each NIC
* <s>Add <tt>display</tt> parameter to enable user control of the QEMU display method (VNC/SDL/nographic)</s> [DONE]
* Implement VM.clone() -- to be used by tests that need to add new VMs by themselves
* kickstart installation support
* kvm-autotest auto resolve qemu-kvm backtraces
* Make sure ppm_utils.image_crop() handles bad input properly
* Enable tests to modify a VM's params (while it's down)
* Make tests shorter by using small utility functions that fail a test and display a failure message
* Improve the stepmaker data naming system (e.g. name PPM files according to their MD5 sums)
* Redirect guest monitor to serial and capture guest error messages written to console
* Support an "expected to fail" param (at least in our html report)
* Log a gdb-backtrace for tests (executables) which generate a core dump file.
* Enable "guest-load" for a VMs before/while tests are running (e.g. migration of a VM, while a movie is playing on it)
* Add support for kvm-trace.

== Tests ==

* Find out how to use Autotest's netperf test with guests
* If needed, write a netperf/iperf test
* Write a time drift test
* Write a test which adds VMs until one of them becomes unresponsive, and records the maximum number of VMs successfully started.
* cpu hotplug - Glauber
* pci net hot plug -Glauber
* pci block hot plug -Glauber
* s3 hibernation - Gleb
* tap networking
* pxe boot * net types
* qcow1, qcow2, vmdk, vhd
* qcow2 internal snapshots
* <s>Run autotest tests inside Linux guest</s> [ALREADY DONE]
* Further migration
* Multiple nics
* ksm
* -vga stg, nographics
* kvm unit-tests
* Add unattended guest installations (using kickstart or answer-files)
* kvm_install -- keep src and build directories under <kvm_autotest_root> instead of under kvm_runtest_2.

== Misc ==

* Goal -- get accepted into upstream autotest
* Add docstrings to all functions that still lack them, including the ones in stepmaker.py, stepeditor.py and kvm_tests.py
* Add module headers and docstrings
* Document the setupssh.iso and setuptelnet.iso creation procedures in the wiki
* Rename all Windows ISOs currently used to their official MSDN names
* Move Windows-setup iso images into the windows directory.
* Share all the stepmaker data somehow so other people can debug stepfiles
* Fill the sections "Working with step files" and "Step file creation tips" in the wiki
* Document the kvm_install test in the "Tests" section of the wiki

== Bugs ==

None so far?

KVM-Autotest/TODO

2009-03-22T16:38:33Z

Uril:

= KVM-Autotest To-Do list =

== Framework ==

* <s>Verify MD5 sum of ISO image before starting a VM</s> [DONE]
* <s>Enable multiple user controlled port redirections (with -redir)</s> [DONE]
* <s>Implement functions to translate VM ports and addresses (e.g. VM.get_port(), VM.get_address())</s> [DONE]
* Support TAP networking with a user specified MAC address for each NIC
* <s>Add <tt>display</tt> parameter to enable user control of the QEMU display method (VNC/SDL/nographic)</s> [DONE]
* Implement VM.clone() -- to be used by tests that need to add new VMs by themselves
* kickstart installation support
* kvm-autotest auto resolve qemu-kvm backtraces
* Make sure ppm_utils.image_crop() handles bad input properly
* Enable tests to modify a VM's params (while it's down)
* Make tests shorter by using small utility functions that fail a test and display a failure message
* Improve the stepmaker data naming system (e.g. name PPM files according to their MD5 sums)
* Redirect guest monitor to serial and capture guest error messages written to console
* Support an "expected to fail" param (at least in our html report)
* Log a gdb-backtrace for tests (executables) which generate a core dump file.
* Enable "guest-load" for a VMs before/while tests are running (e.g. migration of a VM, while a movie is playing on it)
* Add support for kvm-trace.

== Tests ==

* Find out how to use Autotest's netperf test with guests
* If needed, write a netperf/iperf test
* Write a time drift test
* Write a test which adds VMs until one of them becomes unresponsive, and records the maximum number of VMs successfully started.
* cpu hotplug - Glauber
* pci net hot plug -Glauber
* pci block hot plug -Glauber
* s3 hibernation - Gleb
* tap networking
* pxe boot * net types
* qcow1, qcow2, vmdk, vhd
* qcow2 internal snapshots
* <s>Run autotest tests inside Linux guest</s> [ALREADY DONE]
* Further migration
* Multiple nics
* ksm
* -vga stg, nographics
* kvm unit-tests
* Add unattended guest installations (using kickstart or answer-files)

== Misc ==

* Goal -- get accepted into upstream autotest
* Add docstrings to all functions that still lack them, including the ones in stepmaker.py, stepeditor.py and kvm_tests.py
* Add module headers and docstrings
* Document the setupssh.iso and setuptelnet.iso creation procedures in the wiki
* Rename all Windows ISOs currently used to their official MSDN names
* Move Windows-setup iso images into the windows directory.
* Share all the stepmaker data somehow so other people can debug stepfiles
* Fill the sections "Working with step files" and "Step file creation tips" in the wiki
* Document the kvm_install test in the "Tests" section of the wiki

== Bugs ==

None so far?