14.1 How Virtual Machines Share a Host

Virtual machines virtualize hardware interfaces, while containers share the host kernel and use namespaces, resource controls, and filesystem mechanisms to create constrained runtime environments. Both provide isolation, but their boundaries and failure models differ.

The guest OS in a virtual machine thinks it's in full control of the CPU, memory, and hardware, while the actual machine serves multiple guest operating systems. The hypervisor's job isn't simply to "trick" the guest OS, but to build an executable resource model: which operations can run directly, which events must be handed off to the VMM, how guest addresses map to host pages, and who handles device requests.

Linux containers use another boundary, see Namespaces, cgroups, and container isolation. This section only discusses virtual machines with a separate Guest kernel.

1. What exactly does a virtual machine virtualize?

A typical VM will see:

• One or more virtual CPUs (vCPUs);
• An address space for guest physical memory;
• Interrupt controller and timer;
• Virtual devices such as disks, network cards, and consoles;
• Firmware or boot protocol;
• A standalone guest kernel.

The guest process is initially managed by the guest kernel. On the host side, the hypervisor/VMM maps the vCPU to a physical CPU's time, maps the guest physical address to host memory, and handles device access.

Guest App
    ↓ system call
Guest kernel
    ↓ virtual CPU / memory / device
Hypervisor + User-mode VMM
    ↓
Host CPU / memory / device

"Type 1 bare metal" and "Type 2 hosted" setups help get started, but the boundaries of modern implementations aren't always clean. For example, KVM integrates core virtualization capabilities directly into the Linux kernel, while user-space processes like QEMU handle device emulation and management; it's hard to explain this with just a two-layer diagram.

2. CPU Virtualization: Direct Execution and Exit Handling

Ideally, guest arithmetic, branching, and memory instructions execute directly on the physical CPU. VM exits occur only when the hypervisor must intervene, such as with certain controlled instructions, register accesses requiring interception, external interrupts, or device access.

Hardware virtualization extensions provide two execution contexts. To illustrate with Intel terminology:

VMX root operation
The Host/VMM controls the VM here
        │ VM entry
        ▼
VMX non-root operation
Guests can run in their own ring 0.
│ Controlled event triggers VM exit
        ▼
VMM reads exit reason, emulates or handles it, and re-enters Guest

VMX root and non-root operation do not replace traditional ring 0/ring 3. Both operations still have their own privilege levels. The guest kernel can reside in a non-root ring 0, but this does not grant it full control over the host.

The VMM controls the guest state, host recovery state, and exit conditions by managing the configuration structure. Not all "sensitive operations" need to be intercepted on every exit; modern hardware can directly maintain portions of the virtual state, and the VMM selectively intercepts operations based on both security and performance considerations.

Before Hardware Virtualization Extensions

Classic virtualization theory aims for sensitive instructions to trap when privilege levels are insufficient, allowing the VMM to emulate them. Early x86 architectures exhibited behaviors that didn't conform to this simple trap-and-emulate model. Implementers adopted techniques such as binary translation, ring deprivileging, and paravirtualization to rewrite dangerous instructions or encourage guest operating systems to explicitly invoke the hypervisor.

This history shows that hardware extensions controlled execution boundaries and does not mean "virtualization was completely impossible before," nor can a single fixed performance penalty summarize all previous implementations.

3. vCPU must still be scheduled by the Host

In KVM and similar implementations, each vCPU is a host scheduling entity; a bare-metal hypervisor may use its own scheduler. If a VM is configured with 8 vCPUs, it does not mean it permanently occupies 8 physical cores:

• The host might schedule multiple vCPUs to take turns on the same logical CPU;
• vCPU may migrate, compromising cache and NUMA locality;
• After over-allocation of host resources, guest threads will wait for physical CPU;
• When a vCPU holds a guest spinlock and is descheduled, it amplifies spin waiting for other vCPUs;
• The Host still needs to run I/O backends, manage processes, and handle other workloads.

The CPU utilization in Guest only describes the time it observes. Performance diagnostics also need to examine Host scheduling, steal time, vCPU topology, NUMA placement, and over-subscription.

Fixed binding can reduce jitter but decreases scheduling flexibility; whether it's worthwhile depends on latency requirements, interference from other workloads on the same host, and hardware topology.

4. Two-Stage Address Translation

The guest process generates a guest virtual address (GVA). The guest page table translates it into a guest physical address (GPA), which the hypervisor then maps to a host physical address (HPA):

GVA
 │ Guest page table
 ▼
GPA
│ EPT / NPT entries for the second-stage page table
 ▼
HPA

Intel commonly refers to the second-stage page table as EPT, while AMD refers to it as NPT. The CPU's TLB can cache the combined translation, avoiding the need to fully traverse both page table hierarchies on every access.

Shadow Page Tables and Nested Page Tables

Without hardware support for the second stage of translation, the VMM can maintain a shadow page table mapping directly from GVA to HPA. When the guest updates its page table, the VMM must capture and synchronize the shadow mapping, resulting in a complex path and frequent exits.

Nested page tables allow hardware to continue translating Guest Page Table entries during page table walks, reducing many page table maintenance exits, but combined walks on TLB misses are more expensive. Thus, large pages, TLB hit rates, and memory locality remain important.

Don't treat "at most a fixed number of memory accesses" as a performance conclusion. The actual cost depends on page table levels, page size, cache hits, hardware page-walk cache, and architecture.

5. Memory isn't limited to a single address mapping

VM memory management also has to deal with:

Demand paging. The VMM can hold the guest address space and only acquire host physical memory when the guest actually accesses a page.

Memory Overcommitment. The total configured memory across multiple VMs can exceed physical memory, as long as the actual active set doesn't simultaneously reach its limits. Under sudden load spikes, host swapping, failure to reclaim memory, or out-of-memory (OOM) conditions may occur, leading to a sharp deterioration in tail latency.

balloon. The guest-driven balloon consumes and returns a batch of pages, informing the host which guest memory can be reclaimed. It requires guest cooperation, not equivalent to secretly seizing any arbitrary pages.

Huge pages. Huge pages reduce TLB and page table overhead, but make allocation, fragmentation, migration, and deallocation coarser-grained.

NUMA. When a vCPU runs on one socket but its memory resides on a different NUMA node, it increases memory access latency. vCPU placement and memory strategy must be designed together.

Dirty page tracking. Live migration and incremental snapshots need to know which guest pages have changed. The granularity of tracking and write intensity affect whether migration can converge.

6. Three Common Paths of I/O Virtualization

Device Emulation

The VMM presents to the Guest a hardware model already supported by Guest drivers. When the Guest accesses ports or MMIO registers, the VMM simulates the device state machine.

The advantage is compatibility with old Guest OS; the cost is a complex device model that can lead to more exits and user-mode handling upon frequent access. The device model itself is part of the security boundary and requires strict validation of descriptors, lengths, and state transitions provided by the Guest OS.

Paravirtualized Devices

A virtio interface type doesn't pretend to be a traditional physical device; instead, it allows the guest driver to submit descriptors via a shared queue, and the backend processes them in bulk, then notifies the guest.

Guest virtio driver
        ↓ descriptors
shared virtqueue
        ↓ notification
QEMU / vhost / hardware backend
        ↓
physical device

Batching can reduce per-request exits and emulation overhead, but "virtio is zero-copy" is not true. Whether data is copied, and where it queues in the network or block layer, depends on the backend, buffers, and offloading configuration.

Device Passthrough

Mechanisms like VFIO can hand physical devices or SR-IOV virtual functions over to VMs. IOMMU restricts device DMA to authorized host page mappings:

Guest driver ── MMIO ──> device
      │
      └─ DMA address ──> IOMMU translation ──> allowed Host pages

Passthrough shortens the data path but increases difficulty in migration, snapshots, reset isolation, topology, and device sharing. IOMMU is one essential isolation component, and it does not mean that any device firmware is automatically trustworthy.

7. Interrupts and Clocks Need Virtualization Too

After the device completes its request, it must notify the Guest. In traditional approaches, a physical interrupt may arrive at the Host first, then be injected as a virtual interrupt; modern hardware can accelerate some of these paths. If interrupt frequency is high, combining, batching, and polling may be more efficient than handling each interrupt individually.

Time is trickier. After a vCPU is paused by the Host, how should the Guest's wall clock, monotonic clock, and CPU time behave? Is the time source continuous across a migration to a new Host? Hypervisors and Guests typically coordinate through mechanisms like paravirtual clocks. Distributed applications should still use the correct clock semantics and shouldn't treat VM clocks as inherently precise.

8. Compare the Isolation Boundaries

Dimension	Virtual Machine	Linux Container
Kernel	Each Guest has its own kernel	Processes share the Host's kernel
Abstraction	Virtual Hardware	Isolated Kernel Resource View
System Compatibility	Runs different Guest OSs, subject to virtual hardware limitations	User space must be compatible with the Host kernel ABI
Boot path	Firmware/Boot, Guest kernel, User space	Creates isolated and restricted processes
Primary Isolation Boundaries	Hypervisor, Virtual Devices, Management Layer	Host Kernel, Runtime, namespace/cgroup Configuration
Resource baseline	Typically higher	Typically lower

This table cannot conclude that "VMs are definitely secure" or "containers are definitely fast." Virtual device vulnerabilities, management plane permissions, and shared hardware side channels can all cross VM boundaries; containers require deep protections like user namespaces, capabilities, seccomp, and LSMs to limit the host kernel's attack surface.

When choosing, the question is about which kernel boundary, what kind of hardware, and how migration and operations will be handled, not about memorizing boot time or performance percentages.

9. Common Misconceptions

"KVM is a full virtual machine." KVM provides kernel-side vCPU and memory basics. A complete VMM also needs firmware, devices, interrupts, images, migration, and management.

"All Guest instructions pass through the hypervisor." Ordinary instructions are executed directly; only controlled events result in an exit.

"Nested page tables eliminate memory virtualization overhead." They reduce page table synchronization overhead, but combined translation, TLB misses, and NUMA still incur costs.

"Assigning 16 vCPUs gives you 16 CPU cores." vCPUs are schedulable entities, but actual physical CPU time is still allocated by the host.

"Device passthrough is always best." A shorter data path can cost sharing flexibility, migration support, and simpler operations.

10. Summary

A virtual machine's core is controlled sharing:

• The CPU executes guest instructions directly and triggers a VM exit on controlled events;
• vCPU still follows host scheduling, and over-subscription can affect latency;
• GVA reaches HPA via the Guest page table and the second-stage page table;
• Memory also involves balloon, dirty pages, NUMA, and large pages;
• I/O can be selected as device emulation, virtio, or passthrough;
• Independent Guest kernels establish isolation boundaries different from those of containers.

The next section goes to Linux Containers. If you'd like to work directly with KVM file descriptors and ioctl, proceed to the Minimal KVM Experiment.