FreeBSD bhyve Live Migration: Moving VMs Between Hosts

This project introduces a working prototype for live migration of bhyve guests on FreeBSD 15.1. A running amd64 guest can migrate between two hosts over TCP while continuing to run, only pausing for the final stop-and-copy phase. The development builds upon bhyve's experimental snapshot support, incorporating dirty-page tracking in the vmm kernel module and a migration protocol within bhyve and bhyvectl.

The migration process begins with a pre-copy phase, where the guest's memory is transmitted, followed by a series of repeated scans that capture a bitmap of changed pages and transmit only those pages. Once the remaining dirty set is within the configured downtime target or the round limit is reached, the bhyve guest is paused, and the final dirty pages along with the existing snapshot state for the vCPUs, kernel state, and emulated devices are transmitted to the destination host.

The destination host then validates the VM geometry, receives the memory and state, restores it, and reports readiness. Only after this point does the source host transmit the GO message, which signifies the completion of the migration.

The current implementation includes a warm, stop-and-copy mode as a fallback option and for comparison purposes. Various test environments have successfully completed repeated live migrations, memory-rewrite workloads, and disk-intensive tasks. The observed downtime for a slowly dirtying 512 MiB guest ranged from 138 to 171 ms, while a rapidly dirtying guest may experience a longer pause before converging during the pre-copy phase.

While the prototype demonstrates promising engineering results, it is not a production support promise or a complete correctness proof. The immediate focus of the project is on correctness, compatibility validation, transport safety, and repeatable tests. Some areas requiring attention before upstreaming the code into FreeBSD include compression, zero-page elision, broader device coverage, richer orchestration, and performance tuning.

The source code and test tooling are being published as a WIP RFC to encourage wider FreeBSD and bhyve feedback and to establish a clear roadmap for further development.