From KVM
 
m (anchor is not informative)
 
(13 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 +
=Migration=
 +
 
== Introduction ==
 
== Introduction ==
KVM currently supports savevm/loadvm and offline or live migration [[BR]]
+
KVM currently supports savevm/loadvm and offline or live migration
Migration commands are given when in qemu-monitor (Alt-Ctrl-2).[[BR]]    
+
Migration commands are given when in qemu-monitor (Alt-Ctrl-2).     
Upon successful completion, the migrated VM continues to run on the destination host. [[BR]][[BR]]
+
Upon successful completion, the migrated VM continues to run on the destination host.
  
 
==== Note ====
 
==== Note ====
You can migrate a guest between an AMD host to an Intel host and back. Naturally, a
+
You can migrate a guest between an AMD host to an Intel host and back. Naturally, a 64-bit guest can only be migrated to a 64-bit host, but a 32-bit guest can be migrated at will.
64-bit guest can only be migrated to a 64-bit host, but a 32-bit guest can be migrated at will.
+
  
There are some older Intel processors which don't support NX (or XD), which may cause problems in a
+
There are some older Intel processors which don't support NX (or XD), which may cause problems in a cluster which includes NX-supporting hosts. To disable NX for a given guest, start it with such a parameter: -cpu qemu64,-nx
cluster which includes NX-supporting hosts. We may add a feature to hide NX if this proves to be a
+
problem in actual deployments.                                                                   
+
  
 
== Requirements ==
 
== Requirements ==
Line 32: Line 31:
 
* Open                                                                                                   
 
* Open                                                                                                   
  
[[Anchor(1)]] 1  These features are unique to KVM Live Migration as far as I know. If you know of other hypervisor that support any of them please update this page or let me (Uri) know.                         
+
1  These features are unique to KVM Live Migration as far as I know. If you know of other hypervisor that support any of them please update this page or let me (Uri) know.                         
  
 
== User Interface ==
 
== User Interface ==
Line 55: Line 54:
 
== Example / HOWTO ==
 
== Example / HOWTO ==
 
A is the source host, B is the destination host:
 
A is the source host, B is the destination host:
*__TCP__ example:                              
+
===''TCP'' example:===
1. Start the VM on B with the exact same parameters as the VM on A, in migration-listen mode (-incoming tcp:0:4444 (or other PORT))                                                                              
+
1. Start the VM on B with the exact same parameters as the VM on A, in migration-listen mode:
2. Start the migration (always on the source host):                                                    
+
B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
  A: migrate -d tcp:B:4444 (or other PORT)                                                              
+
2. Start the migration (always on the source host):
3. Check the status (on A only):                                                                        
+
  A: migrate -d tcp:B:4444 (or other PORT)
  info migrate                                                                                          
+
3. Check the status (on A only):
 +
  A: (qemu) info migrate                  
  
* __Offline__ example:
+
===''Offline'' example:===
 
1. unlimit bandwidth used for migration:
 
1. unlimit bandwidth used for migration:
 
  A: migrate_set_speed 1g                 
 
  A: migrate_set_speed 1g                 
Line 77: Line 77:
  
 
== savevm/loadvm to an external state file (using pseudo-migration) ==
 
== savevm/loadvm to an external state file (using pseudo-migration) ==
* let the state file be /saved_images/VM.STATE                       
 
* savevm (qemu monitor):                                             
 
stop                                                               
 
migrate file:///saved_images/VM.STATE                               
 
* loadvm (qemu command line and qemu monitor):                       
 
<qemu-command-line> -incoming file:///saved_images/VM.STATE         
 
cont                                                               
 
  
== more exec: options ==
 
 
* To be supported directly by Migration Protocols, but until then...
 
* To be supported directly by Migration Protocols, but until then...
 
* Save VM state into a compressed file                               
 
* Save VM state into a compressed file                               
    * Save                                                           
+
** Save                                                           
 
  stop                                                               
 
  stop                                                               
 
  migrate_set_speed 4095m                                             
 
  migrate_set_speed 4095m                                             
 
  migrate "exec:gzip -c > STATEFILE.gz"                               
 
  migrate "exec:gzip -c > STATEFILE.gz"                               
  
    * Load
+
** Load
  gzip -c -d STATEFILE.gz | <qemu-command-line> -incoming stdio
+
  gzip -c -d STATEFILE.gz | <qemu-command-line> -incoming "exec: cat"  or
 +
<qemu-command-line> -incoming "exec: gzip -c -d STATEFILE.gz"
 
* Save VM State into an encrypted file (assumes KEY has already been generated)
 
* Save VM State into an encrypted file (assumes KEY has already been generated)
    * Save                                                                     
+
** Save                                                                     
 
  stop                                                                           
 
  stop                                                                           
  migrate_set_speed 4095m                                                      
+
  migrate_set_speed 4095m
 
  migrate "exec:gpg -q -e -r KEY -o STATFILE.gpg"
 
  migrate "exec:gpg -q -e -r KEY -o STATFILE.gpg"
  
 
* Load VM state from an encrypted file
 
* Load VM state from an encrypted file
  gpg -q -d -r KEY STATEFILE.gpg | <qemu-command-line> -incoming stdio
+
  gpg -q -d -r KEY STATEFILE.gpg | <qemu-command-line> -incoming "exec:cat"
 +
 
 +
== more exec: options ==
 +
 
 
* Send encrypted VM state from host A to host B (Note: ssh is probably better, this is just for show)
 
* Send encrypted VM state from host A to host B (Note: ssh is probably better, this is just for show)
    * on host A
+
 
 +
** on host A
 
  migrate "exec:gpg -q -e -r KEY | nc B 4444"
 
  migrate "exec:gpg -q -e -r KEY | nc B 4444"
    * on host B
+
 
  nc -l 4444 | gpg -q -d -r KEY | <qemu-command-line> -incoming stdio
+
** on host B
 +
  nc -l 4444 | gpg -q -d -r KEY | <qemu-command-line> -incoming "exec:cat"
  
 
== Algorithm (the short version) ==
 
== Algorithm (the short version) ==
1. Setup
+
1. Setup
  * Start guest on destination, connect, enable dirty page logging and more
+
* Start guest on destination, connect, enable dirty page logging and more
2. Transfer Memory
+
2. Transfer Memory
  * Guest continues to run
+
* Guest continues to run
  * Bandwidth limitation (controlled by the user)
+
* Bandwidth limitation (controlled by the user)
  * First transfer the whole memory
+
* First transfer the whole memory
  * Iteratively transfer all dirty pages (pages that were written to by the guest).
+
* Iteratively transfer all dirty pages (pages that were written to by the guest).
3. Stop the guest
+
3. Stop the guest
  * And sync VM image(s) (guest's hard drives).
+
* And sync VM image(s) (guest's hard drives).
4. Transfer State
+
4. Transfer State
  * As fast as possible (no bandwidth limitation)
+
* As fast as possible (no bandwidth limitation)
  * All VM devices' state and dirty pages yet to be transferred
+
* All VM devices' state and dirty pages yet to be transferred
5. Continue the guest
+
5. Continue the guest
  * On destination upon success
+
* On destination upon success
      * Broadcast "I'm over here" Ethernet packet to announce new location of NIC(s).
+
** Broadcast "I'm over here" Ethernet packet to announce new location of NIC(s).
  * On source upon failure (with one exception).
+
* On source upon failure (with one exception).
  
  
 
Instructions for kvm-13 and below: [[MigrationQemu0.8.2]].
 
Instructions for kvm-13 and below: [[MigrationQemu0.8.2]].

Latest revision as of 20:31, 16 May 2015

Migration

Introduction

KVM currently supports savevm/loadvm and offline or live migration Migration commands are given when in qemu-monitor (Alt-Ctrl-2). Upon successful completion, the migrated VM continues to run on the destination host.

Note

You can migrate a guest between an AMD host to an Intel host and back. Naturally, a 64-bit guest can only be migrated to a 64-bit host, but a 32-bit guest can be migrated at will.

There are some older Intel processors which don't support NX (or XD), which may cause problems in a cluster which includes NX-supporting hosts. To disable NX for a given guest, start it with such a parameter: -cpu qemu64,-nx

Requirements

  • The VM image is accessible on both source and destination hosts (located on a shared storage, e.g. using nfs).
  • It is recommended an images-directory would be found on the same path on both hosts (for migrations of a copy-on-write image -- an image created on top of a base-image using "qemu-image create -b ...")
  • The src and dst hosts must be on the same subnet (keeping guest's network when tap is used).
  • Do not use -snapshot qemu command line option.
  • For tcp: migration protocol
  • the guest on the destination must be started the same way it was started on the source.

highlights / merits

  • Almost unnoticeable guest down time
  • Guest is not involved (unique to KVM Live Migration [#1 1])
  • Capability to tunnel VM state through an external program (unique to KVM Live Migration [#1 1])
  • ssh/gzip/bzip2/gpg/your own
  • Upon success guest continues to run on destination host, upon failure guest continues to run on source host (with one exception)
  • Short and Simple
  • Easy to enhance
  • Hardware independence (almost).
  • Support for migration of stopped (paused) VMs.
  • Open

1 These features are unique to KVM Live Migration as far as I know. If you know of other hypervisor that support any of them please update this page or let me (Uri) know.

User Interface

The user interface is through the qemu monitor (alt-ctrl-2 on the SDL window)

Management

migrate [-d] <URI>
migrate_cancel    

The '-d' will let you query the status of the migration.

With no '-d' the monitor prompt returns when the migration completes. URI can be one of 'exec:<command>' or tcp:<ip:port>

Status

info migrate 

Migration Parameters

migrate_set_speed <speed>   set bandwidth control parameters (max transfer rate per second)

Example / HOWTO

A is the source host, B is the destination host:

TCP example:

1. Start the VM on B with the exact same parameters as the VM on A, in migration-listen mode:

B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))

2. Start the migration (always on the source host):

A: migrate -d tcp:B:4444 (or other PORT)

3. Check the status (on A only):

A: (qemu) info migrate                   

Offline example:

1. unlimit bandwidth used for migration:

A: migrate_set_speed 1g                

2. stop the guest:

A: stop                                

3. continue with either TCP or exec migration as described above.

Problems / Todo

  • TSC offset on the new host must be set in such a way that the guest sees a monotonically increasing TSC, otherwise the guest may hang indefinitely after migration.
  • usbdevice tablet complains after migration.
  • handle migration completion better (especially when network problems occur).
  • More informative status.
  • Migration does not work while CPU real-mode/protected mode are still changing.

savevm/loadvm to an external state file (using pseudo-migration)

  • To be supported directly by Migration Protocols, but until then...
  • Save VM state into a compressed file
    • Save
stop                                                               
migrate_set_speed 4095m                                            
migrate "exec:gzip -c > STATEFILE.gz"                              
    • Load
gzip -c -d STATEFILE.gz | <qemu-command-line> -incoming "exec: cat"   or
<qemu-command-line> -incoming "exec: gzip -c -d STATEFILE.gz"
  • Save VM State into an encrypted file (assumes KEY has already been generated)
    • Save
stop                                                                          
migrate_set_speed 4095m
migrate "exec:gpg -q -e -r KEY -o STATFILE.gpg"
  • Load VM state from an encrypted file
gpg -q -d -r KEY STATEFILE.gpg | <qemu-command-line> -incoming "exec:cat"

more exec: options

  • Send encrypted VM state from host A to host B (Note: ssh is probably better, this is just for show)
    • on host A
migrate "exec:gpg -q -e -r KEY | nc B 4444"
    • on host B
nc -l 4444 | gpg -q -d -r KEY | <qemu-command-line> -incoming "exec:cat"

Algorithm (the short version)

1. Setup

  • Start guest on destination, connect, enable dirty page logging and more

2. Transfer Memory

  • Guest continues to run
  • Bandwidth limitation (controlled by the user)
  • First transfer the whole memory
  • Iteratively transfer all dirty pages (pages that were written to by the guest).

3. Stop the guest

  • And sync VM image(s) (guest's hard drives).

4. Transfer State

  • As fast as possible (no bandwidth limitation)
  • All VM devices' state and dirty pages yet to be transferred

5. Continue the guest

  • On destination upon success
    • Broadcast "I'm over here" Ethernet packet to announce new location of NIC(s).
  • On source upon failure (with one exception).


Instructions for kvm-13 and below: MigrationQemu0.8.2.