SysadminNews

Multikernel Linux Runs Real Kernels Side by Side

On this page
  1. What it does
  2. The numbers, and what they are measuring
  3. Where this is actually interesting
  4. What it is not, yet
  5. Sources and further reading

Multikernel Technologies has published mklinux v7.0-mk2, the first public release of a Linux tree that runs several independent kernels on one physical machine without a hypervisor. A host kernel owns a pool of processors, memory and PCI devices, carves that pool into slices, and boots a separate kernel into each slice through kexec. Every spawned kernel then runs natively on its own cores and its own physical memory. Nothing is emulated and nothing is trapped, which is what produces the numbers: a two process context switch measures 1.37 microseconds against 3.42 microseconds under KVM on the same hardware.

The short answer

Multikernel Technologies released mklinux v7.0-mk2, a Linux 7.0 based tree that boots several independent kernels on one physical machine with no hypervisor involved. A host kernel owns a pool of processors, memory and PCI devices, partitions it, and boots a spawn kernel into each partition through kexec_file_load, tracked under /sys/fs/multikernel/. Each spawned kernel runs natively on its own cores and physical memory. On an Intel Xeon Gold 5418Y, lmbench measures a two process context switch at 1.37 microseconds against 3.42 under KVM, pipe latency at 3.24 against 7.06, and Unix socket latency at 4.81 against 7.48, at a cost of about 19 extra watts.

v7.0-mk2first public multikernel Linux tree
1.37 uscontext switch, against 3.42 under KVM
x86_64the only architecture supported at launch
Answer card: multikernel Linux v7.0-mk2 boots several independent kernels on one machine with no hypervisor, each owning dedicated CPUs, memory and PCI devices, with a two process context switch at 1.37 microseconds against 3.42 under KVM.
Not a hypervisor and not a container. A second kernel, booted onto real cores. PNG

There are normally two answers to running isolated workloads on one machine. Virtual machines give you separate kernels at the cost of a mediation layer. Containers remove the mediation layer at the cost of sharing a kernel. Multikernel Linux proposes a third answer that is odd enough to be worth understanding: separate kernels, no mediation layer, and hardware split between them at boot.

What it does

A host kernel owns a pool of processors, memory and PCI devices. You carve a slice out of that pool and boot a second kernel into it through kexec_file_load(). The spawned kernel comes up on its own cores, in its own physical memory, driving its own assigned devices. Instances are tracked through device tree overlays under /sys/fs/multikernel/.

The phrase the developers use is that nothing is emulated and nothing is trapped, and the only thing shared is what you choose to share. That is a precise description rather than marketing. There is no VM exit path because there is nothing to exit to. There are no second level page tables because the kernel is using real physical memory directly. There is no device emulation because the devices are simply handed over.

The first public tree, mklinux v7.0-mk2, is based on Linux 7.0 and the source sits at github.com/multikernel/linux. Cong Wang announced it on the kernel mailing list, and coverage picked it up between August 25 and the end of the month.

The numbers, and what they are measuring

The release compares against KVM using lmbench on an Intel Xeon Gold 5418Y.

Comparison of lmbench latencies on an Intel Xeon Gold 5418Y: two process context switch 1.37 microseconds on multikernel against 3.42 on KVM, pipe latency 3.24 against 7.06, Unix socket latency 4.81 against 7.48. Lower is better.
Lower is better. These are the paths where hypervisor mediation is most visible. PNG

A two process context switch measures 1.37 microseconds, against 3.42 microseconds under KVM. Pipe latency measures 3.24 microseconds against 7.06. Unix socket latency measures 4.81 microseconds against 7.48. Roughly two times on context switching and a bit more than two times on pipes.

Read those as microbenchmarks, because that is what they are. lmbench deliberately isolates the operations where a hypervisor has to intervene, so it flatters an approach whose entire pitch is removing the intervention. A database or a web tier spends most of its time in code that a hypervisor never touches, and the improvement there will be far smaller. What the numbers do establish honestly is the size of the mediation cost itself, which is useful precisely because most of us carry an assumption about that number rather than a measurement.

There is a cost on the other side of the ledger, and the project publishes it: about 19 additional watts, because the host keeps its processors fully occupied rather than idling them. On a single machine that is noise. Across a rack it is a line item, and it is the kind of detail that gets omitted from this sort of announcement often enough that including it is worth noting.

Where this is actually interesting

The use cases the developers describe are more specific than general purpose consolidation. Giving a machine learning agent direct access to a graphics card without passing it through a hypervisor is one. Replacing a kernel without taking the machine down is another, since a new kernel can be booted into a slice and the workload moved across. Partitioning very high core count servers is a third, and it is the one that will matter most over time: as core counts keep climbing, treating a socket as a single uniform pool becomes a less obvious default than it used to be. Heterogeneous designs where different cores genuinely differ point the same way.

The isolation argument is also a reliability argument rather than a density argument. Containers share a kernel, which means a lock contention problem or a panic is shared by everything on the box. Separate kernels do not have that property. That is a real advantage for workloads where one misbehaving tenant taking down the machine is the failure you are actually trying to prevent, and it is a poor trade for workloads where you wanted to pack four hundred small services onto one host.

What it is not, yet

This is a first public release of an out of tree fork. It supports x86_64 only, although the architecture specific interfaces have been separated so that ARM and RISC-V ports are structurally possible later. Its path into the mainline kernel is uncertain rather than scheduled, and the realistic assessment is that any such move would be well out.

None of which makes it uninteresting. The mainstream direction of the last decade has been to add layers: hypervisors, then containers, then orchestrators on top of both. This goes the other way, and it does so with a mechanism that already exists in the kernel rather than a new one. Anyone who tunes for latency should read the benchmarks, and anyone building a lab this quarter has a cheap experiment available. If you are weighing the alternatives at the layer above, our note on near live container migration in Incus 7.4 covers what the conventional stack has been doing about the same problem, and the Linux 7.3-rc1 merge window shows what is landing upstream while this sits outside it.

Sources and further reading

Frequently asked questions

How is this different from a virtual machine?

A virtual machine runs a guest kernel on emulated or paravirtualised hardware, with the hypervisor mediating access. That mediation costs you the VM exit path, second level page tables and device emulation, and those costs are exactly what show up in latency sensitive workloads. A multikernel instance has none of them. The spawned kernel boots onto real cores and real physical memory through kexec_file_load and drives real PCI devices assigned to it. There is no trap to a layer underneath, because there is no layer underneath. The tradeoff is that you get partitioning rather than the migration, snapshotting and overcommit that a hypervisor exists to provide.

How is it different from containers?

Containers share one kernel. That is their advantage, because it makes them cheap to start and dense to pack, and it is also their limitation: a lock contention problem, a scheduler pathology or a kernel panic is shared by everything on the machine. Multikernel instances do not share a kernel at all. Each one has its own scheduler, its own memory management and its own device drivers, so a fault in one does not propagate to the others. You are paying dedicated CPUs and dedicated memory for that isolation, which is a very different density profile from containers.

What do the published benchmarks actually show?

The release compares multikernel against KVM with lmbench on an Intel Xeon Gold 5418Y. A two process context switch measures 1.37 microseconds against 3.42 microseconds under KVM. Pipe latency measures 3.24 microseconds against 7.06. Unix socket latency measures 4.81 microseconds against 7.48. These are microbenchmarks of the paths where hypervisor mediation is most visible, so they represent a best case for the approach rather than an application level result. There is also a measured power cost of roughly 19 additional watts, because the host keeps its processors fully occupied.

Can I run this in production?

No, and the project does not suggest otherwise. This is the first public release of an out of tree kernel fork, it is tested on x86_64 only, and its path into the mainline kernel is uncertain rather than scheduled. Treat it as something to build in a lab and measure against your own workload, not as a platform choice. The interesting question it lets you answer cheaply is whether the hypervisor overhead you assume you are paying is actually the overhead you are paying, and that is worth an afternoon even if the tree never merges.

What hardware and use cases does it target?

x86_64 is the only architecture officially supported at launch, though the developers separated the architecture specific interfaces so other ports can follow. The use cases they describe are consolidating workloads onto bare metal without a hypervisor tax, giving machine learning agents direct access to a graphics card, replacing a kernel without taking the machine down, and partitioning the very high core count servers that are becoming ordinary. Heterogeneous designs are also mentioned, where different cores could reasonably run different kernels rather than pretending to be one uniform pool.