13.3 Storage Redundancy, Checking, and Backup

The archive has learned how to restore consistency after a power failure. But you see a different kind of disaster at the warehouse entrance: one device has completely failed, another can still read but silently flips several bits, and a directory was manually deleted by an operator. Logs cannot solve these problems because they record write order, not a copy in another failure domain.

RAID, checksum, snapshot, and backup each address a part of this challenge, none can replace the others. Remember this boundary:

Redundancy enables current service to continue running; backup provides a recovery copy in a different time point or failure domain.

1. First, discuss RAID failure models

RAID combines multiple storage devices into a single logical volume. Assume all disks have equal capacity, represented by N for the number of disks and S for the capacity per disk:

Layout	Available Capacity	Fault Tolerance (Disk Failure)	Key Features
RAID 0	N × S	0	Striped, no redundancy
Two-disk RAID 1	S	Up to 1 disk	Mirroring, intuitive layout
RAID 5	(N - 1) × S	Up to 1 disk	Distributed parity
RAID 6	(N - 2) × S	Up to 2 disks	Dual independent parity
RAID 10	N / 2 × S	At most one disk per mirror group	Mirrored groups striped together

The fault tolerance in this table only addresses the model of "complete disk failure." Controller faults, firmware bugs, cabinet power loss, human error, or file system corruption may simultaneously affect multiple disk members.

RAID 10 cannot simply be described as tolerating "any failure of N/2 disks." If all members within a single mirror group fail, the corresponding stripe is lost. Only when failures are spread across different mirror groups can the array survive multiple disk failures.

2. Stripes, Images, and Verification

RAID 0: Only Distribution

Data is spread across multiple devices:

stripe 0: [A0 on disk 0] [A1 on disk 1]
stripe 1: [A2 on disk 0] [A3 on disk 1]

Multiple requests can proceed in parallel, but if any member disk fails, part of the stripe becomes unreadable, and the logical volume is typically unavailable as a whole. It's suitable only for data that can be regenerated at the application layer or when redundancy is already provided above the storage layer. It cannot be considered a fault-tolerant solution.

RAID 1: Duplicate Content

Mirroring writes the same logical data to multiple members. Reads can select from available copies, while writes must adhere to the array’s completion policy. Mirroring allows continued service even if an entire disk fails and provides a second data source for checksum repair.

However, mirroring faithfully replicates erroneous writes from the logical layer. Mistakes like accidental deletion, ransomware encryption, or application-level overwrites quickly propagate to all mirrored members.

RAID 5/6: Recovering from Missing Members Using Parity

RAID 5 stores a distributed parity block alongside each data block in a stripe. When one member fails, the remaining blocks can reconstruct the missing data. RAID 6 uses two independent parity relationships, allowing for the simultaneous failure of two members.

For small data overwrites, the process typically involves:

1. Reading the old data and old parity;
2. Computing new parity;
3. Writing new data and new parity.

If power loss occurs during this process, the new data and parity may become inconsistent, a risk commonly known as the "write hole." Controller caches, logs, full-stripe writes, and integration with the file system can mitigate this, but the presence of parity alone does not guarantee power-loss consistency protocols.

RAID 10: Stripes Across Mirrored Groups

RAID 10 writes each data block into a mirrored group, then distributes those blocks across multiple groups. It avoids the parity update path of RAID 5/6 and is often used for workloads with frequent updates and strict recovery time requirements. The trade-off is that typically only half of the raw capacity is available.

"Better for databases" remains no absolute conclusion. The database’s own replication strategy, cloud block storage guarantees, capacity budget, failure domains, and recovery point objectives may all influence the final choice.

3. Verify Capacity Assumptions with Code

The following function deliberately accepts only disks of equal capacity and encodes layout constraints directly into the code:

def usable_capacity(level, disk_sizes):
    if not disk_sizes or any(size <= 0 for size in disk_sizes):
        raise ValueError("disk sizes must be positive")
    if len(set(disk_sizes)) != 1:
        raise ValueError("this model requires equal-sized disks")

    disks = len(disk_sizes)
    size = disk_sizes[0]
    normalized = level.upper()

    if normalized == "RAID0":
        if disks < 2:
            raise ValueError("RAID0 needs at least two disks")
        return disks * size
    if normalized == "RAID1":
        if disks != 2:
            raise ValueError("this model defines a two-disk mirror")
        return size
    if normalized == "RAID5":
        if disks < 3:
            raise ValueError("RAID5 needs at least three disks")
        return (disks - 1) * size
    if normalized == "RAID6":
        if disks < 4:
            raise ValueError("RAID6 needs at least four disks")
        return (disks - 2) * size
    if normalized == "RAID10":
        if disks < 4 or disks % 2 != 0:
            raise ValueError("RAID10 needs an even number of disks")
        return disks // 2 * size

    raise ValueError(f"unknown level: {level}")


sizes = [8] * 6  # six 8-TB disks; units remain TB in this model
assert usable_capacity("RAID0", sizes) == 48
assert usable_capacity("RAID5", sizes) == 40
assert usable_capacity("RAID6", sizes) == 32
assert usable_capacity("RAID10", sizes) == 24
print("capacity checks passed")

When real arrays mix disks of different capacities, many implementations can only calculate available space based on the smallest disk or partition, potentially leading to wasted capacity. Vendors may also offer layout schemes that differ from traditional RAID levels; capacity formulas must be verified against specific product documentation and configuration guidelines.

4. Still Operating with One Less Drive, But Most Vulnerable During Reconstruction

Once an array enters a degraded state, its redundancy is reduced. During reconstruction, it must read from remaining members and write the recovered data to a replacement drive, a process that takes time and places significant strain on the system. Although the archive may still appear operational, the risk window is substantially larger than under normal conditions.

After a member drive fails, the array transitions into a degraded state. Read requests may need to reconstruct missing data using mirror copies or parity relationships. Once a replacement drive is installed, the system must scan large volumes of existing data and write it to the new drive.

The risks during reconstruction stem from several factors:

• Remaining drives face sustained read loads, increasing the likelihood of latent errors being exposed;
• Business workloads compete with reconstruction for I/O bandwidth;
• The array remains in a degraded state for an extended period, reducing overall resilience;
• A second drive, or another drive within the same failure domain, may fail shortly after;
• Incorrectly replacing a healthy drive can artificially expand the failure scope;
• A hot spare that has never been validated may fail when it's time to take over.

Do not rely on fixed estimates based on capacity (such as "reconstruction takes X hours." Media speed, current system load, controller policies, error retry behavior, and rebuild throttling all influence the actual duration. Monitoring should focus on degradation duration, media errors, rebuild progress, and the health of spare drives) not just whether the array eventually shows optimal.

5. Checksums and Redundancy Must Be Paired

Traditional block devices may return content errors without signaling I/O failure. Without end-to-end checksumming, upper layers cannot distinguish between "read success" and "reading corrupted bytes."

A complete data integrity path requires:

Compute checksums at write time
        ↓
Protect both data and checksums using appropriate update protocols
        ↓
Recompute checksums during read or scrub operations
        ↓
Mismatch detected → retrieve candidate copy from independent redundancy
        ↓
Validate candidate and repair the corrupted copy

Mirroring without checksums means you can't determine which copy is correct when both differ. Checksums without redundancy only detect errors, they cannot repair them. The checksum algorithm must align with the threat model: defending against random media errors and defending against malicious tampering are fundamentally different goals.

6. Why Snapshots Are Not Backup

COW (Copy-on-Write) file systems or storage arrays can create snapshots quickly. Snapshots are well-suited for:

• Rolling back a single erroneous deployment;
• Capturing a short-term consistent view;
• Providing stable read points for backup operations;
• Rapidly cloning test environments.

However, if snapshots and the primary data reside on the same device, array, or management domain, they may both suffer from:

• Complete storage array failure;
• Accidental deletion by an administrator;
• Credential exposure;
• Ransomware that deletes snapshots;
• Facility or account-level outages.

Snapshots are a version retention mechanism. They only become part of a true backup system when the copy is stored in a separate failure domain and is accompanied by clear retention policies, access controls, and defined recovery procedures.

7. Backup Must Be Inferred from Recovery Goals

Start by defining two key objectives:

• RPO (Recovery Point Objective): The maximum amount of data loss you can tolerate;
• RTO (Recovery Time Objective): How quickly services must be restored after a failure.

RPO determines how frequently backups or replicas are created, while RTO dictates the type of recovery medium, level of automation, and extent of pre-warming. Simply stating "backup daily" doesn't clarify whether recovery takes three hours or three days, nor does it confirm that the last successful backup is readable and valid.

A more robust backup design should include:

• Versioning, rather than a single copy that will be overwritten alone;
• Isolated failure domains and independent credentials;
• At least one copy that is resistant to online tampering;
• End-to-end encryption during transfer and at rest, with independently recoverable keys;
• Consistent backup protocols for stateful systems like databases;
• Verification, inventory tracking, and regular recovery drills;
• Clear retention periods, deletion procedures, and compliance requirements.

Recovery drills must actually start services or validate business data, they cannot stop at "backup task completed successfully." A backup that cannot be restored within the RTO provides little value during an incident.

8. What to Ask When Selecting

Don’t start with “RAID 5 or RAID 10.” Begin by answering:

1. What are we protecting against, full disk failure, silent data corruption, node failure, or accidental operations?
2. Is data already replicated across nodes at the application layer?
3. Which failure domains belong to the array, host, power supply, and rack?
4. Which of the following is most critical, random write performance, sequential throughput, tail latency, or capacity?
5. Can performance targets still be met during degradation?
6. How long are rebuild and scrub operations available, and how are they monitored?
7. Who deletes snapshots, and are backup credentials isolated from production environments?
8. When was the last full recovery drill conducted, and did the results meet the required RPO and RTO?

In cloud environments, “disks” may already be replicated at the infrastructure level by the provider. Adding traditional RAID on top of virtual disks can increase complexity and may place seemingly distinct components within the same underlying failure domain. Always base decisions on the current durability, failure domains, and snapshot semantics provided by the cloud service.

9. Summary

RAID, parity, snapshots, and backups address distinct problems:

• RAID 0 offers no fault tolerance; RAID 1 and 10 rely on mirroring; RAID 5 and 6 use parity;
• The available capacity formula fails to account for controller, rack, or human-induced failures;
• During degradation and rebuild phases, redundancy is reduced and latent errors become apparent;
• Parity detects errors, while redundant copies provide a source for recovery;
• Snapshots preserve historical views of the same system but do not inherently span failure domains;
• The reliability of backups must be validated through version isolation, recovery drills, and measurable RPO/RTO metrics.

By now, the file lifecycle (from naming, writing, crash recovery, to device failure) has been fully traced. The next chapter moves into Virtualization and Containers, where we explore how operating systems repackage the real boundaries of CPU, memory, and I/O for different workloads.