Add LCOW live migration support across the controller stacks by rawahars · Pull Request #2790 · microsoft/hcsshim

rawahars · 2026-06-24T12:02:20Z

Implement end-to-end live migration support for LCOW pods so a running guest and its workloads can be moved between hosts. Every controller in the stack — VM, pod, Linux container, process, network, and SCSI/Plan9/VPCI devices — gains a save/import lifecycle plus the destination-side patch and resume plumbing needed to rebind local resources and bring the workload back online.

Migration lifecycle:

Source: Save serializes a controller's state into a self-describing protobuf envelope and freezes the controller (StateSourceMigrating) so no mutating ops race the transfer; Resume rolls the freeze back, and a finalize Stop or VM teardown terminates it.
Destination: Import rehydrates a controller into a migrating state, Patch repoints saved resources (layer VHDs, process IO/bundle, network namespace) at the destination host, and Resume binds the live VM, guest, and devices and republishes events so containerd treats the task as locally running. AbortMigrated discards an imported-but-never-resumed controller and emits synthetic exits so Delete can proceed.

VM controller:

Add the source/destination migrating states and the full HCS migration lifecycle: InitializeLiveMigrationOnSource, StartLiveMigrationOnSource, StartLiveMigrationTransfer, FinalizeLiveMigration, plus StartWithMigrationOptions on the destination.
Exchange the opaque compatibility blob, retain the final HCS document so the destination can recreate an identical VM, and recover the GCS port/bridge-id allocator floors so reissued ids cannot collide.
Make SCSI initialization lazy (built on first use from the HCS document) and handle never-started/destination teardown paths, including the already-stopped HCS error.

Controller-specific changes:

SCSI controller switches to an RWMutex and rejects all ops while migrating; ReserveForRootfs now carries the full disk config.
Process, network, container, and VM state machines document and enforce the new migrating states and transitions.
Pod gains a migrating guard, AbortMigrated fan-out, and routes new containers through lazy SCSI init.

Includes accompanying unit tests for the new save/import/patch/resume paths across all controllers.

jterry75 · 2026-06-30T18:58:38Z

 type Controller struct {
 	// mu serializes all public operations on the Controller.
-	mu sync.Mutex
+	mu sync.RWMutex


You changed no usage of this why did it become rw?

jterry75 · 2026-06-30T18:59:58Z


 type Controller struct {
-	mu sync.Mutex
+	mu sync.RWMutex


jterry75 · 2026-06-30T19:00:57Z

+
+	// nextGuestPort is the GCS IO port-allocator floor restored from a
+	// migration snapshot; consumed by [Controller.Resume].
+	nextGuestPort uint32


I thought you didnt want to store a policy at this layer?

Unfortunately for LM, we need to make this exception. These fields are part of the saved state payload and can only be decoded by VM controller. Now we already decode the saved state during Import but these need to be set on the new bridge which happens in Resume. Alternate would be for migration controller to store the opaque state and then send the state as Resume param where it gets decoded again.

That is just redundancy and hence I stored it in the controller layer during import.

jterry75 · 2026-06-30T19:01:38Z

+	// hcsDocument is the final HCS document used to create this VM,
+	// retained for lazy SCSI controller construction and for shipping to
+	// the destination during live migration.
+	hcsDocument *hcsschema.ComputeSystem


You would have to maintain it over time right? How can it be lazy?

So storing the hcsDocument serves 2 purposes-

Lazy initialization of SCSI controller which needs to reserve the rootfs slots

It is shipped to the destination where it is used to recreate the VM

For the LM capable workloads, when we create the VM on the destination, we override the SCSI map with that from the imported controller and therefore it is consistent.

Apart from the SCSI controller, nothing else needs maintenance over time as those features are not supported. When they are, we would override them in a similar manner.

Implement end-to-end live migration support for LCOW pods so a running guest and its workloads can be moved between hosts. Every controller in the stack — VM, pod, Linux container, process, network, and SCSI/Plan9/VPCI devices — gains a save/import lifecycle plus the destination-side patch and resume plumbing needed to rebind local resources and bring the workload back online. Migration lifecycle: - Source: Save serializes a controller's state into a self-describing protobuf envelope and freezes the controller (StateSourceMigrating) so no mutating ops race the transfer; Resume rolls the freeze back, and a finalize Stop or VM teardown terminates it. - Destination: Import rehydrates a controller into a migrating state, Patch repoints saved resources (layer VHDs, process IO/bundle, network namespace) at the destination host, and Resume binds the live VM, guest, and devices and republishes events so containerd treats the task as locally running. AbortMigrated discards an imported-but-never-resumed controller and emits synthetic exits so Delete can proceed. VM controller: - Add the source/destination migrating states and the full HCS migration lifecycle: InitializeLiveMigrationOnSource, StartLiveMigrationOnSource, StartLiveMigrationTransfer, FinalizeLiveMigration, plus StartWithMigrationOptions on the destination. - Exchange the opaque compatibility blob, retain the final HCS document so the destination can recreate an identical VM, and recover the GCS port/bridge-id allocator floors so reissued ids cannot collide. - Make SCSI initialization lazy (built on first use from the HCS document) and handle never-started/destination teardown paths, including the already-stopped HCS error. Controller-specific changes: - SCSI controller switches to an RWMutex and rejects all ops while migrating; ReserveForRootfs now carries the full disk config. - Process, network, container, and VM state machines document and enforce the new migrating states and transitions. - Pod gains a migrating guard, AbortMigrated fan-out, and routes new containers through lazy SCSI init. Includes accompanying unit tests for the new save/import/patch/resume paths across all controllers. Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

marma-dev

Overall: solid, cohesive design with strong tests. I'd want findings (nil-deref) and (mutex) resolved before merge, and (hcsDoc mutation) / (Resume Guards) / (Disks() vs ReserveForRootFs) clarified.

marma-dev · 2026-07-08T13:34:31Z

+			return fmt.Errorf("cannot create VM in state %s: imported HCS document has no VirtualMachine", c.vmState)
+		}
+		hcsDocument = c.hcsDocument
+		hcsDocument.VirtualMachine.MigrationOptions = opts.MigrationOptions


Can opts.MigrationOptions ever be nil? If so we might dereference a nil MigrationOptions two lines below. Maybe add a precondition check (return an error if opts.MigrationOptions == nil in StateMigratingImported)

marma-dev · 2026-07-08T13:37:14Z

+		// SCSI controller is the source of truth for the destination
+		// topology (rootfs + hot-added, path-patched); use it verbatim.
+		if c.scsiController != nil {
+			hcsDocument.VirtualMachine.Devices.Scsi = c.scsiController.HCSAttachments()


This code block mutates c.hcsDocument in place (hcsDocument is the same pointer). Since this document is the "source of truth" retained for lazy SCSI init and re-shipped on save, mutating MigrationOptions/Devices.Scsi on the shared instance could bite on any retry/re-entry. Consider deep-copying before stamping migration-specific fields

marma-dev · 2026-07-08T13:39:44Z

+func (c *Controller) Resume(ctx context.Context, vm VMSCSIOps, guest GuestSCSIOps) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	c.vm = vm
+	c.guest = guest
+	c.isMigrating = false
+
+	log.G(ctx).Debug("resumed scsi controller")


Resume unconditionally sets c.vm, c.guest, and clears isMigrating. Every other controller's Resume guards on the migrating state and errors otherwise. A stray Resume here would silently overwrite the live vm/guest. Worth a guard for consistency/safety (even if callers are currently correct).

marma-dev · 2026-07-08T13:44:39Z

+	defer c.mu.RUnlock()
+
+	// Collect the config of each indexed disk.
+	configs := make([]disk.Config, 0, len(c.disksByPath))


It seems ReserveForRootfs populates controllerSlots but not disksByPath, while Disks() iterates disksByPath.

I think on the destination, vm.Patch calls scsiController.Disks() to GrantVmAccess and since Import does populate disksByPath from the saved host paths, it likely works for the migration path — but the asymmetry seems a little fragile.

Let's confirm rootfs disks are always represented in disksByPath post-import (a test asserting Disks() includes the rootfs would lock this down).

marma-dev · 2026-07-08T13:46:32Z

 type Controller struct {
 	// mu serializes all public operations on the Controller.
-	mu sync.Mutex
+	mu sync.RWMutex


Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

…ume and Patch methods Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

jiechen0826 · 2026-07-09T01:43:11Z

+
+// Save captures the migrating VM's state into a serialized snapshot that the
+// destination host consumes to recreate an equivalent VM.
+func (c *Controller) Save(ctx context.Context) (*anypb.Any, error) {


Really nice that every controller here comes with save/import tests. One gap I noticed: the vm controller's migration surface here (Save/Import/Patch/Resume) and vm_migration.go don't seem to have a corresponding _test.go, while pod/network/process/linuxcontainer/scsi all do.
These are the destination side steps that run right before and after the CreateVM branch in vm.go (the one with the nil MigrationOptions case), so a vm-layer test would exercise that whole sequence. Would it be worth adding one? might have missed them if they're coming in the follow-up PR.

That's a very astute observation! Yes, we would have that as a follow-up PR.
We have an existing PR which adds base VM Controller tests- #2738

Once this one is merged, we will have the mocks we need to accomplish the migration tests. I created a Issue #2812 to track the same.

marma-dev

LGTM! Issue #2812 tracks the VM Controller Tests.

rawahars requested a review from a team as a code owner June 24, 2026 12:02

rawahars added this to the Add Live Migration support to LCOW V2 shim milestone Jun 24, 2026

rawahars force-pushed the lm_save branch 5 times, most recently from 3ae59a0 to e4fc26e Compare June 24, 2026 18:32

rawahars requested a review from jterry75 June 30, 2026 17:09

jterry75 reviewed Jun 30, 2026

View reviewed changes

rawahars mentioned this pull request Jul 2, 2026

Add LCOW live migration source and destination APIs #2803

Open

rawahars force-pushed the lm_save branch from 5e69c1f to 6c82aa8 Compare July 2, 2026 05:45

rawahars force-pushed the lm_save branch from 6c82aa8 to a671117 Compare July 2, 2026 05:52

marma-dev requested changes Jul 8, 2026

View reviewed changes

rawahars added 2 commits July 9, 2026 01:31

address pr feedback

03a529b

Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

refactor: enhance migration state handling and error reporting in Res…

b1421ee

…ume and Patch methods Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>

rawahars force-pushed the lm_save branch from fc92273 to b1421ee Compare July 8, 2026 22:26

jiechen0826 reviewed Jul 9, 2026

View reviewed changes

marma-dev approved these changes Jul 9, 2026

View reviewed changes

jiechen0826 approved these changes Jul 9, 2026

View reviewed changes

rawahars merged commit 3a9e957 into microsoft:main Jul 9, 2026
20 checks passed

rawahars deleted the lm_save branch July 9, 2026 16:40

Uh oh!

Conversation

rawahars commented Jun 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marma-dev left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marma-dev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

marma-dev left a comment •

edited

Loading