Migration

From KVM
Revision as of 06:35, 4 February 2009 by WikiSysop (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Introduction

KVM currently supports savevm/loadvm and offline or live migration BR Migration commands are given when in qemu-monitor (Alt-Ctrl-2).BR Upon successful completion, the migrated VM continues to run on the destination host. BRBR

Note

You can migrate a guest between an AMD host to an Intel host and back. Naturally, a 64-bit guest can only be migrated to a 64-bit host, but a 32-bit guest can be migrated at will.

There are some older Intel processors which don't support NX (or XD), which may cause problems in a cluster which includes NX-supporting hosts. We may add a feature to hide NX if this proves to be a problem in actual deployments.

Requirements

  • The VM image is accessible on both source and destination hosts (located on a shared storage, e.g. using nfs).
  • It is recommended an images-directory would be found on the same path on both hosts (for migrations of a copy-on-write image -- an image created on top of a base-image using "qemu-image create -b ...")
  • The src and dst hosts must be on the same subnet (keeping guest's network when tap is used).
  • Do not use -snapshot qemu command line option.
  • For tcp: migration protocol
  • the guest on the destination must be started the same way it was started on the source.

highlights / merits

  • Almost unnoticeable guest down time
  • Guest is not involved (unique to KVM Live Migration [#1 1])
  • Capability to tunnel VM state through an external program (unique to KVM Live Migration [#1 1])
  • ssh/gzip/bzip2/gpg/your own
  • Upon success guest continues to run on destination host, upon failure guest continues to run on source host (with one exception)
  • Short and Simple
  • Easy to enhance
  • Hardware independence (almost).
  • Support for migration of stopped (paused) VMs.
  • Open

Anchor(1) 1 These features are unique to KVM Live Migration as far as I know. If you know of other hypervisor that support any of them please update this page or let me (Uri) know.

User Interface

The user interface is through the qemu monitor (alt-ctrl-2 on the SDL window)

Management

migrate [-d] <URI>
migrate_cancel    

The '-d' will let you query the status of the migration.

With no '-d' the monitor prompt returns when the migration completes. URI can be one of 'exec:<command>' or tcp:<ip:port>

Status

info migrate 

Migration Parameters

migrate_set_speed <speed>   set bandwidth control parameters (max transfer rate per second)

Example / HOWTO

A is the source host, B is the destination host:

  • __TCP__ example:

1. Start the VM on B with the exact same parameters as the VM on A, in migration-listen mode (-incoming tcp:0:4444 (or other PORT)) 2. Start the migration (always on the source host):

A: migrate -d tcp:B:4444 (or other PORT)                                                                

3. Check the status (on A only):

info migrate                                                                                            
  • __Offline__ example:

1. unlimit bandwidth used for migration:

A: migrate_set_speed 1g                

2. stop the guest:

A: stop                                

3. continue with either TCP or exec migration as described above.

Problems / Todo

  • TSC offset on the new host must be set in such a way that the guest sees a monotonically increasing TSC, otherwise the guest may hang indefinitely after migration.
  • usbdevice tablet complains after migration.
  • handle migration completion better (especially when network problems occur).
  • More informative status.
  • Migration does not work while CPU real-mode/protected mode are still changing.

savevm/loadvm to an external state file (using pseudo-migration)

  • let the state file be /saved_images/VM.STATE
  • savevm (qemu monitor):
stop                                                                 
migrate file:///saved_images/VM.STATE                                
  • loadvm (qemu command line and qemu monitor):
<qemu-command-line> -incoming file:///saved_images/VM.STATE          
cont                                                                 

more exec: options

  • To be supported directly by Migration Protocols, but until then...
  • Save VM state into a compressed file
   * Save                                                          
stop                                                               
migrate_set_speed 4095m                                            
migrate "exec:gzip -c > STATEFILE.gz"                              
   * Load
gzip -c -d STATEFILE.gz | <qemu-command-line> -incoming stdio
  • Save VM State into an encrypted file (assumes KEY has already been generated)
   * Save                                                                     
stop                                                                          
migrate_set_speed 4095m                                                       
migrate "exec:gpg -q -e -r KEY -o STATFILE.gpg"
  • Load VM state from an encrypted file
gpg -q -d -r KEY STATEFILE.gpg | <qemu-command-line> -incoming stdio
  • Send encrypted VM state from host A to host B (Note: ssh is probably better, this is just for show)
   * on host A
migrate "exec:gpg -q -e -r KEY | nc B 4444"
   * on host B
nc -l 4444 | gpg -q -d -r KEY | <qemu-command-line> -incoming stdio

Algorithm (the short version)

1. Setup
  * Start guest on destination, connect, enable dirty page logging and more
2. Transfer Memory
  * Guest continues to run
  * Bandwidth limitation (controlled by the user)
  * First transfer the whole memory
  * Iteratively transfer all dirty pages (pages that were written to by the guest).
3. Stop the guest
  * And sync VM image(s) (guest's hard drives).
4. Transfer State
  * As fast as possible (no bandwidth limitation)
  * All VM devices' state and dirty pages yet to be transferred
5. Continue the guest
  * On destination upon success
     * Broadcast "I'm over here" Ethernet packet to announce new location of NIC(s).
  * On source upon failure (with one exception).


Instructions for kvm-13 and below: MigrationQemu0.8.2.