13.2 File Systems and Crash Recovery

A successful write only means the kernel accepted those bytes; fclose returning success does not mean the data has passed through page cache, device cache, and is durably persisted. The file system must still coordinate data blocks, space allocation metadata, inodes, and directory entries. A power failure could occur between these updates, making crash consistency a core concern in file system design.

How files move from a path to inodes and data blocks is described in File System Structure. This section focuses on a different question: when a file system update is interrupted mid-way by a crash, how does it recover to a consistent, interpretable state, and how should applications define their own persistence boundaries?

1. What Layers Does a Write Go Through

For example, with fprintf, data might pass through the following layers:

Application data
  ↓
C stdio buffer
  ↓  fflush / buffer full / fclose
write system call
  ↓
Kernel page cache
  ↓  write-back / fsync / fdatasync
Block layer, device queues, and device cache
  ↓
Persistent media

Each layer can coalesce and reorder writes. This significantly improves throughput, but it also means that the guarantees of several common operations differ:

Operation	What It Can Indicate	What It Cannot Guarantee
fprintf succeeds	Data has entered the stdio processing pipeline	Data has reached the kernel or persistent storage
fflush succeeds	The stdio buffer has been committed to the underlying stream	Data is permanently stored
write succeeds	The kernel has accepted the reported number of bytes written	The write request is fully satisfied and data has been written to disk
fclose succeeds	stdio has flushed and closed the descriptor	An automatic fsync has been performed
fsync succeeds	The request has synchronized file data and required metadata	The entire business transaction has been atomically completed

write may only write a portion of the data; the program must handle short writes. Asynchronous I/O errors may not be reported until later fsync or close calls. fdatasync synchronizes file data and the metadata needed to retrieve it; it may omit metadata unrelated to content. Savings depend on the filesystem and workload.

The durability guarantees span across the operating system, drivers, and storage devices. A program can only rely on publicly exposed synchronization interfaces and proper configuration. If a device incorrectly reports that its cache is stable, the file system cannot magically fulfill that promise.

2. Why Crashes Leave Inconsistent States

Appending data to a file may require modifying:

• The content of new data blocks;
• Free space bitmaps or other allocation structures;
• The inode's size and block mapping;
• Relevant timestamps;
• In some cases, directory entries or extended attribute metadata.

These changes cannot be assumed to occur as a single, atomic media write. The device might reorder requests, or a single large write could be split across multiple operations. If an inode points to a new block but the new data hasn't yet been durably persisted, a read might return old or invalid content. If space is marked as allocated but no inode references it, that space becomes leaked and unavailable.

Therefore, the file system maintains a set of invariants, such as:

• A block cannot be simultaneously allocated to two files that are not allowed to share it;
• Blocks referenced by an inode must be marked as allocated in the allocation structure;
• Directory entries must point to valid objects;
• Link counts and naming relationships must remain consistent;
• Free space must not overlap with active data.

On crash recovery, the system first restores these structural invariants. It does not automatically guarantee that a sequence of application-level writes has transactional semantics.

3. Journaling: Record Updates Before Their Final Placement

A journaling filesystem organizes metadata modifications into transactions. A simplified flow is:

Prepare transactional changes
    ↓
Write log entries to journal
    ↓
Persist commit record
    ↓
Write changes back to final location (checkpoint)
    ↓
Reclaim corresponding log space

During recovery:

• Transactions without complete commit records are ignored;
• Transactions that have committed but not yet fully written to their final location can be replayed;
• Replay must be designed to be safe even when executed multiple times.

The specific content of log records is not standardized. Metadata logs protect only the file system's structure; data may follow regular write-back paths. Some patterns enforce data commits before the metadata that references it, to prevent exposing stale block contents during recovery. Other patterns allow looser ordering. One cannot assume that application data is durably persisted simply because a journaling filesystem is used.

Logs are not application transactions. Two write entries, one rename, and a database record might belong to different file system transactions. If an application requires "all or nothing," it must still choose a consistency protocol or use a storage system that provides transactional guarantees.

4. Copy-on-Write: Publishing a New State Tree

Copy-on-Write (COW) file systems do not overwrite active blocks in place. Instead, they write modified data and metadata to a new location, then publish a new root or checkpoint:

Old root ──> Old metadata ──> Old data

                    Write new data
                         ↑
New root ──> New metadata ─────┘
  ↑
Final publication

If a crash occurs before publication, the old root continues to describe the previous state. If recovery happens after publication, the system resumes from the new root. A true implementation must also protect the root pointer itself, manage reference counts, handle space reclamation, maintain checksums, and support transaction groups, far more than simply atomically updating an inode pointer.

COW makes snapshotting straightforward: the old and new roots can share unchanged blocks, with only the differing portions consuming additional space. However, snapshots extend the lifetime of old blocks, and random overwrites may end up writing to new locations, potentially increasing fragmentation and write amplification.

Logging and COW are not mutually exclusive. A system can have some components log changes while others use COW, or provide different persistence protocols for different operations. Understanding update ordering is more important than aligning with a particular file system paradigm.

5. Checksum-Based Failure Detection and the Need for Redundancy to Enable Recovery

A block checksum detects many corruptions, but collisions or a consistent incorrect update to both data and checksum can escape detection. To prevent both the data and its checksum from being corrupted by the same erroneous update, end-to-end designs typically place checksum information within metadata that is protected by separate update paths.

When a checksum failure is detected, the system can only automatically recover if a trusted copy exists, such as a mirror, parity reconstruction result, or remote replica. Without redundancy, a checksum can clearly signal that "data is corrupted," but it cannot reconstruct the original bytes.

Periodic scrubbing proactively reads and verifies cold data, exposing latent errors early. It addresses a different problem than backup and recovery drills: scrubbing checks the integrity of current copies, while backups are designed to restore state across time or failure domains.

6. fsck is a recovery tool, not a version-control system

For non-logged file systems or file systems where recovery outcomes remain uncertain after restoration, fsck can scan the global structure and verify invariants. Typical operations include:

• Comparing block references against free space information;
• Verifying that directory entries point to valid objects;
• Correcting determinable link counts;
• Isolating objects that cannot be reassigned;
• Marking bad blocks or unrecoverable structures.

The scanning cost generally increases with the size of the file system and the number of objects. While logged recovery limits common post-crash operations to recent transactions, it does not eliminate the need for consistency checks.

fsck cannot reconstruct lost application data based solely on structural information. If two versions are both structurally valid, it cannot determine which one the user intended to retain; if data blocks have been overwritten, there is no recovery source without a backup.

7. Atomic Visibility and Crash-Consistent Persistence Are Two Different Things

Within the same filesystem, rename can provide atomic name switching for readers: they either see the old file or the new file, never a partial update of a directory entry. But "other processes immediately see the full new version" and "after a power loss or crash, the new version is definitely still there" are not the same guarantee.

In a Linux/POSIX-style environment, a common protocol for replacing a small state file is:

1. Create a temporary file in the target directory;
2. Write all content, handling short writes;
3. fsync the temporary file;
4. Close it;
5. Within the same filesystem, rename the temporary file to the target name;
6. fsync the target directory to persist the name change.

Save this Linux example as durable-replace.c, compile with cc -std=c17 -Wall -Wextra -Werror durable-replace.c -o durable-replace, and run in a dedicated test directory. It replaces state.txt with one writer, assumes a trusted current directory that stays unchanged during the call, and accepts only filenames within that directory. Permission inheritance, multiple writers, and startup cleanup require application policies.

#define _GNU_SOURCE

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

static int write_all(int fd, const void *buffer, size_t length) {
    const unsigned char *cursor = buffer;

    while (length > 0) {
        ssize_t written = write(fd, cursor, length);
        if (written > 0) {
            cursor += (size_t)written;
            length -= (size_t)written;
        } else if (written < 0 && errno == EINTR) {
            continue;
        } else {
            if (written == 0) errno = EIO;
            return -1;
        }
    }
    return 0;
}

enum ReplaceResult {
    REPLACE_FAILED = -1,
    REPLACE_OK = 0,
    REPLACE_PUBLISHED_WITH_ERROR = 1
};

static int durable_replace(const char *target,
                           char *temporary_template,
                           const void *data,
                           size_t length) {
    if (target == NULL || temporary_template == NULL || *target == '\0'
            || strchr(target, '/') != NULL
            || strchr(temporary_template, '/') != NULL) {
        errno = EINVAL;
        return REPLACE_FAILED;
    }
    int directory = open(".", O_RDONLY | O_DIRECTORY);
    if (directory < 0) return REPLACE_FAILED;
    int fd = mkstemp(temporary_template);
    if (fd < 0) {
        int error = errno;
        close(directory);
        errno = error;
        return REPLACE_FAILED;
    }

    if (write_all(fd, data, length) != 0 || fsync(fd) != 0) goto cleanup;
    if (close(fd) != 0) {
        fd = -1;
        goto cleanup;
    }
    fd = -1;
    if (rename(temporary_template, target) != 0) goto cleanup;

    int error = 0;
    if (fsync(directory) != 0) error = errno;
    if (close(directory) != 0 && error == 0) error = errno;
    if (error != 0) {
        errno = error;
        return REPLACE_PUBLISHED_WITH_ERROR;
    }
    return REPLACE_OK;

cleanup:
    {
        int saved_errno = errno;
        if (fd >= 0) close(fd);
        unlink(temporary_template);
        close(directory);
        errno = saved_errno;
        return REPLACE_FAILED;
    }
}

int main(void) {
    static const char payload[] = "version=2\n";
    char temporary[] = "state.txt.tmp.XXXXXX";
    int result = durable_replace("state.txt", temporary,
                                 payload, sizeof payload - 1);
    if (result != REPLACE_OK) {
        perror(result == REPLACE_PUBLISHED_WITH_ERROR
               ? "published; final sync/close failed" : "not published");
        return EXIT_FAILURE;
    }
    puts("state.txt replaced");
    return EXIT_SUCCESS;
}

The return value distinguishes three outcomes: 0 means synchronization and closure succeeded; -1 means publication did not succeed; 1 means rename succeeded but directory synchronization or closure reported an error. In the third case, readers may already see the new file. Failure does not mean the target is unchanged, and deleting it is not a rollback. This example targets local Linux filesystems; remote failures such as NFS errors can make even the rename outcome ambiguous.

The directory is opened before writing, and cleanup preserves the original errno. On Linux, do not retry close on the same fd after an error: the descriptor may already have been reused. Production applications must also address:

• Using directory descriptors and the openat family of interfaces to constrain path resolution;
• Explicitly specifying target permissions, owner, and extended attributes;
• Implementing a concurrent protocol for multiple writers;
• Defining rules for handling erroneous paths and leftover temporary files at startup;
• Falling back to the platform's official persistence mechanism when the filesystem does not support directory fsync.

Do not generalize this protocol into "no bit will ever be lost on any file system or hardware." It instead defines the persistence boundaries that an application should request and verify.

8. Group Commit Shares Synchronization Costs

Persistent synchronization often involves device queuing and cache flushing, both of which introduce fixed overhead that typically exceeds the cost of writing just a few bytes. Databases instead rely on write-ahead logging (WAL) and group commit:

Transaction A ┐
Transaction B ├─> Append individual log records first ─> Single sync operation ─> Acknowledge each transaction separately
Transaction C ┘

This spreads the cost of a single persistence operation across multiple transactions, while still requiring that all relevant log entries have passed the durability boundary before final commit. Specific systems further manage log sequence numbers, checkpoints, data page writes, and replica acknowledgments; file system journals cannot substitute for database WALs because they maintain fundamentally different invariants.

Performance tests cannot simply copy fixed values like "how many microseconds per fsync." The actual result depends on media type, queue depth, virtualization layer, filesystem, mount options, device caching, and synchronization concurrency. Measurements of latency distribution must be taken in the actual target environment, and data durability assumptions must be clearly defined before testing begins.

9. How Meaningful Crash Testing Is Conducted

Simply exiting normally and then checking files won't detect crash windows. More valuable testing involves forcibly terminating processes at different stages of a protocol or simulating power loss, followed by verification of:

• After restart, only the old version or a complete new version is visible;
• The filesystem structure remains mountable and inspectable;
• Temporary files can be reliably identified and safely cleaned up;
• Replay operations are idempotent;
• Errors from synchronous calls propagate upward to the caller;
• Recovery does not inadvertently corrupt previously valid state.

SIGKILL can only simulate a sudden user-space process termination; it cannot fully replicate a full system power-down or loss of device cache. Filesystem-level crash testing typically requires virtual block devices, fault injection, or dedicated power-off test environments, and real data disks should be avoided during experimentation.

10. Summary

Crash consistency imposes constraints on the order of updates:

• stdio, page cache, and device cache correspond to different completion boundaries;
• logging enables replay of recent modifications through committed records;
• COW first constructs a new state, then publishes a new root;
• checksums detect corruption, but recovery still requires redundant copies;
• the atomic visibility of rename does not guarantee persistence after power loss;
• applications must explicitly define fsync files, publish names, and establish synchronization protocols for directories;
• database WAL and filesystem recovery each protect distinct invariants.

The next section discusses Storage Redundancy and Backup: what disk failures RAID can withstand, and why it still cannot replace recoverable backups.

Check the Storage Interfaces and Protocols

• fsync and directory durability
• rename and remote error ambiguity