14.3 Running a Guest Instruction with KVM

Deep within the stack of the tower, there is no complete core of the Earth’s heart, only an empty room, a small chunk of memory, and a vCPU waiting to be started. You’ll manually place a single HLT instruction inside, then watch the CPU transition from Guest mode back to Host mode. This is far simpler than booting a full virtual machine, yet it reveals the barest control surface of a hypervisor.

KVM exposes Linux kernel hardware virtualization capabilities as file descriptors and ioctl. Opening /dev/kvm does not magically deliver a complete VM; the user space must still create the VM and vCPU, register Guest memory, and handle the exit reasons returned on each KVM_RUN. This experiment omits firmware, disk, NIC, interrupt controller, or Guest OS, boundaries are clearly defined in the code.

1. Three-Layer File Descriptor Model of the KVM API

The official API is organized by operation target: ioctl

open("/dev/kvm")
       │
       ├─ system fd
       │   ├─ KVM_GET_API_VERSION
       │   ├─ KVM_CHECK_EXTENSION
       │   └─ KVM_CREATE_VM
       │               │
       │               └─ VM fd
       │                   ├─ KVM_SET_USER_MEMORY_REGION
       │                   └─ KVM_CREATE_VCPU
       │                                  │
       │                                  └─ vCPU fd
       │                                      ├─ KVM_SET_REGS
       │                                      ├─ KVM_SET_SREGS
       │                                      └─ KVM_RUN

Passing a ioctl to a file descriptor at a different layer will result in an error. A virtual machine can have multiple vCPU file descriptors; the virtual machine monitor (VMM) typically runs a separate host thread for each vCPU and handles device and management events separately.

2. Pre-Run Checks

The following conditions must be met:

• An x86 Linux host system;
• The CPU and kernel must enable KVM;
• /dev/kvm must exist;
• The current user must have permission to access it;
• The kernel user-space header files providing linux/kvm.h must be installed.

test -e /dev/kvm && echo "KVM device exists"
ls -l /dev/kvm

If /dev/kvm is not visible inside the container, it may simply be that the device was not passed into the container, this does not indicate that the physical CPU lacks virtualization support. Cloud VMs also require nested virtualization to be enabled in order to use KVM within the guest operating system.

3. Connect the Empty Room, Memory, and vCPU

The three-layer file descriptor hierarchy has now been clearly established. Next, register Guest memory with the VM, place HLT at the entry address, and allow the vCPU to cross over KVM_RUN. The program only validates this shortest path and does not pretend to be a full virtual machine product.

#define _GNU_SOURCE

#include <fcntl.h>
#include <linux/kvm.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <unistd.h>

static void fail(const char *operation) {
    perror(operation);
    exit(EXIT_FAILURE);
}

int main(void) {
#if !defined(__x86_64__) && !defined(__i386__)
    fputs("this example requires an x86 host\n", stderr);
    return EXIT_FAILURE;
#else
    const size_t memory_size = 0x1000;

    int system_fd = open("/dev/kvm", O_RDWR | O_CLOEXEC);
    if (system_fd < 0) {
        fail("open /dev/kvm");
    }

    int api_version = ioctl(system_fd, KVM_GET_API_VERSION, 0);
    if (api_version != KVM_API_VERSION) {
        fprintf(stderr, "unexpected KVM API version: %d\n",
                api_version);
        return EXIT_FAILURE;
    }

    int vm_fd = ioctl(system_fd, KVM_CREATE_VM, 0);
    if (vm_fd < 0) {
        fail("KVM_CREATE_VM");
    }

    /*
     * x86 KVM reserves this high Guest physical address for a
     * three-page TSS region. Set it before creating vCPUs.
     */
    if (ioctl(vm_fd, KVM_SET_TSS_ADDR, 0xfffbd000UL) < 0) {
        fail("KVM_SET_TSS_ADDR");
    }

    uint8_t *memory = mmap(
        NULL,
        memory_size,
        PROT_READ | PROT_WRITE,
        MAP_PRIVATE | MAP_ANONYMOUS,
        -1,
        0
    );
    if (memory == MAP_FAILED) {
        fail("mmap guest memory");
    }
    memory[0] = 0xf4;  /* x86 HLT */

    struct kvm_userspace_memory_region region = {
        .slot = 0,
        .flags = 0,
        .guest_phys_addr = 0,
        .memory_size = memory_size,
        .userspace_addr = (uintptr_t)memory,
    };
    if (ioctl(vm_fd, KVM_SET_USER_MEMORY_REGION, &region) < 0) {
        fail("KVM_SET_USER_MEMORY_REGION");
    }

    int vcpu_fd = ioctl(vm_fd, KVM_CREATE_VCPU, 0);
    if (vcpu_fd < 0) {
        fail("KVM_CREATE_VCPU");
    }

    int run_size = ioctl(system_fd, KVM_GET_VCPU_MMAP_SIZE, 0);
    if (run_size < (int)sizeof(struct kvm_run)) {
        fputs("KVM vCPU mmap region is too small\n", stderr);
        return EXIT_FAILURE;
    }

    struct kvm_run *run = mmap(
        NULL,
        (size_t)run_size,
        PROT_READ | PROT_WRITE,
        MAP_SHARED,
        vcpu_fd,
        0
    );
    if (run == MAP_FAILED) {
        fail("mmap kvm_run");
    }

    struct kvm_sregs special;
    if (ioctl(vcpu_fd, KVM_GET_SREGS, &special) < 0) {
        fail("KVM_GET_SREGS");
    }
    special.cs.base = 0;
    special.cs.selector = 0;
    if (ioctl(vcpu_fd, KVM_SET_SREGS, &special) < 0) {
        fail("KVM_SET_SREGS");
    }

    struct kvm_regs registers = {
        .rip = 0,
        .rflags = 0x2,
    };
    if (ioctl(vcpu_fd, KVM_SET_REGS, &registers) < 0) {
        fail("KVM_SET_REGS");
    }

    puts("entering Guest");
    if (ioctl(vcpu_fd, KVM_RUN, 0) < 0) {
        fail("KVM_RUN");
    }

    if (run->exit_reason != KVM_EXIT_HLT) {
        fprintf(stderr, "unexpected exit reason: %u\n",
                run->exit_reason);
        return EXIT_FAILURE;
    }
    puts("Guest executed HLT");

    if (munmap(run, (size_t)run_size) != 0
            || munmap(memory, memory_size) != 0
            || close(vcpu_fd) != 0
            || close(vm_fd) != 0
            || close(system_fd) != 0) {
        fail("cleanup");
    }
    return EXIT_SUCCESS;
#endif
}

Compile:

cc -std=c17 -O2 -Wall -Wextra kvm-hlt.c -o kvm-hlt
./kvm-hlt
# entering Guest
# Guest executed HLT

When the program cannot open /dev/kvm, it fails explicitly at the first step. Do not modify device permissions to allow all users write access just to make the experiment pass; instead, use the distribution-recommended KVM user group or controlled device authorization.

4. What Changes at Each Step

API Version

KVM_GET_API_VERSION returns the stable version of the KVM API. The official documentation requires applications to reject further execution if this value is not equal to KVM_API_VERSION, rather than speculating about future structural compatibility.

Empty VM

KVM_CREATE_VM returns a VM instance with no vCPU or guest memory initially. The VM file descriptor (fd) serves as the control object for subsequent operations such as registering memory, creating vCPUs, and adding devices.

Guest Memory

mmap first allocates a single page of ordinary memory within the VMM's user address space. KVM_SET_USER_MEMORY_REGION then registers that userspace address as a guest physical address (GPA) starting at zero.

This does not directly expose the entire host virtual address space to the guest. The guest only sees GPA; KVM and the hardware's second-stage page tables are responsible for mapping it to the underlying host pages.

Production-grade VMMs must handle multiple memory slots, read-only regions, dirty page tracking, memory hot-plugging, and memory invalidation ordering. Modifications to registered memory regions must adhere strictly to the official API's concurrency and lifecycle rules.

vCPU Shared Region

KVM_GET_VCPU_MMAP_SIZE returns the size of the shared communication region that the vCPU fd can map. After mapping, struct kvm_run simultaneously holds both pre-execution control fields and post-execution exit information.

This shared mapping avoids copying the full state for every exit. General-purpose registers and special registers are still set via their respective ioctl.

Real-Mode Entry

The program sets the CS base and selector to zero and initializes RIP to zero. The first byte at GPA 0 is HLT, so the guest executes it immediately upon entry. The reserved RFLAGS bit 1 must remain set to 1, so the initial value uses 0x2.

5. KVM_RUN is a loop that handles exit processing

A complete virtual machine monitor (VMM) does not simply call KVM_RUN once. It loops around exit reasons:

for (;;) {
    if (ioctl(vcpu_fd, KVM_RUN, 0) < 0) {
        handle_run_error();
    }

    switch (run->exit_reason) {
    case KVM_EXIT_HLT:
        return;
    case KVM_EXIT_IO:
        emulate_port_io(run);
        break;
    case KVM_EXIT_MMIO:
        emulate_mmio(run);
        break;
    default:
        report_unhandled_exit(run->exit_reason);
        return;
    }
}

This is a structural illustration, not a standalone compilable program. I/O and MMIO handling must verify direction, length, address, and buffer boundaries; treating guest-provided data as trusted structures would turn the device model into an attack vector.

Some hardware events are handled by KVM within the kernel and do not return to user space on every iteration. Whether an exit is processed by the VMM, hardware, or the kernel depends on the architecture, capabilities, and VM configuration.

6. From a Single HLT to a Bootable System

To boot a real Guest, a large number of components are required:

• Memory layout and firmware;
• CPU topology, CPUID, and model-specific registers;
• Interrupt controllers, timers, and clocks;
• Serial ports, disk, network, and graphics devices;
• virtio or emulated device backends;
• Boot images and block device formats;
• vCPU threads, interrupt injection, and event loops;
• Snapshots, migration, error recovery, and management interfaces.

KVM safely hands off vCPUs to hardware for execution and returns control when necessary. QEMU, Cloud Hypervisor, and Firecracker are VMMs that complete the user-space machine model for different target environments, they are not merely command-line wrappers around KVM.

7. Experiment Boundaries

This program only verifies the availability of the minimal x86 KVM path on the current machine. It does not validate:

• Whether guest memory protection satisfies a multi-tenant threat model;
• The security of device emulators;
• The correctness of vCPU concurrency and interrupt handling;
• CPU feature compatibility after migration;
• Performance under host over-subscription;
• Whether any operating system can boot successfully.

If KVM_RUN returns an unexpected reason, first print the reason and then consult the official documentation to inspect the corresponding union field. Do not assume that numeric values represent the same additional information across different architectures.

8. Summary

The minimal KVM program reveals the control skeleton of a VMM:

• A system fd is used to create the VM;
• The VM fd registers guest memory and creates vCPU instances;
• The vCPU fd is set to a running state and enters KVM_RUN;
• kvm_run reports the reason for exit from the shared region;
• User-space handles events that require emulation and then resumes the guest.

The next chapter explores CPU pipeline. Virtualization determines which CPU executes a given instruction, while the microarchitecture dictates how that instruction overlaps with preceding and following instructions.

References

• Linux Kernel KVM API Official Documentation