9.1 Threads, Data Races, and Mutual Exclusion

Collection, parsing, and disk-writing tasks within an application can collaborate by sharing data through threads. Threads share the process's address space and most resources, enabling more direct communication, but they also introduce competition within the same memory: when two threads modify the same variable simultaneously, what determines the final result?

Processes have isolated address spaces by default, while threads share process resources and are individually scheduled. This lesson establishes the correctness baseline for C/POSIX threads; condition variables, semaphores, and barriers are covered in synchronization patterns.

What Threads Share and What They Preserve

A sensor application can use three threads: data collection, parsing, and disk writing. These threads can directly access the same heap and global objects, eliminating the need to serialize each batch of data through a pipe. The downside is that any invalid pointer can crash the entire process, and unsynchronized conflicting access can cause a data race. Read-only sharing is safe when the object is properly published and remains alive.

Threads within the same process typically share:

• Virtual address space, including code, global variables, and the heap;
• File descriptor table;
• Current working directory, credentials, and other process-level attributes;
• Signal disposition settings.

Each thread maintains its own register context, stack, thread ID, signal mask, scheduling state, and thread-local storage.

"Sharing the heap" does not mean that local variables are automatically private. Automatic local variables are usually stored on the current thread’s stack, but once their address is passed to another thread, that thread can also access them. Conversely, global read-only data is shared, but because it's read-only, it does not introduce write contention.

Arrange Argument Lifetime Before Creating Threads

pthread_create accepts a void * parameter. This pointer must remain valid throughout the worker's usage period. Passing the loop variable's address repeatedly to multiple threads can cause both a data race while the loop changes the index and a dangling pointer after the loop ends:

/* Incorrect: every worker points to the same changing index. */
for (int index = 0; index < thread_count; index++) {
    pthread_create(&threads[index], NULL, worker, &index);
}

The correct approach is to prepare a separate argument object for each worker and release it only after joining. Creation may fail; POSIX thread functions typically return an error number directly, rather than using -1 in conjunction with errno.

#define _POSIX_C_SOURCE 200809L
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

enum { THREAD_COUNT = 4 };

struct WorkerArgument {
    int index;
};

static void *report_index(void *raw_argument) {
    const struct WorkerArgument *argument = raw_argument;
    printf("worker %d\n", argument->index);
    return NULL;
}

int main(void) {
    pthread_t threads[THREAD_COUNT];
    struct WorkerArgument arguments[THREAD_COUNT];
    int created = 0;

    for (int index = 0; index < THREAD_COUNT; index++) {
        arguments[index].index = index;
        int error = pthread_create(&threads[index], NULL,
                                   report_index, &arguments[index]);
        if (error != 0) {
            fprintf(stderr, "pthread_create: %s\n", strerror(error));
            break;
        }
        created++;
    }

    int failed = created != THREAD_COUNT;
    for (int index = 0; index < created; index++) {
        int error = pthread_join(threads[index], NULL);
        if (error != 0) {
            fprintf(stderr, "pthread_join: %s\n", strerror(error));
            failed = 1;
        }
    }
    return failed ? 1 : 0;
}

If all four creations succeed, output order is unspecified, but the argument values must cover 0–3. Save as thread_arguments.c and compile with cc -std=c17 -O2 -Wall -Wextra -Werror -pthread thread_arguments.c -o thread_arguments. pthread_join both waits for completion and recovers the implementation resources of joinable threads; if it's certain that joining won't occur, detached threads can be created, but process shutdown and object lifetime must then be redesigned accordingly.

Race Condition and Data Race Are Not Exactly the Same

A race condition broadly refers to a situation where the outcome depends on the unpredictable timing of events, such as file renames, network requests, or the order of signals. A data race is a more specific case defined within a programming language’s memory model.

In C, conflicting accesses by different threads to the same memory location cause a data race when at least one is a write, at least one is non-atomic, and neither happens before the other. The behavior is undefined. Adjacent nonzero-width bit-fields can share one memory location, so distinct field names alone do not establish independence.

A classic mistake looks like this:

static int counter;

static void *unsafe_worker(void *unused) {
    (void)unused;
    for (int index = 0; index < 1000000; index++) {
        counter++;
    }
    return NULL;
}

counter++ involves a read-modify-write operation and lacks atomic guarantees. The hardware intuition behind a lost update (such as three assembly instructions (read, increment, write) interleaving) can help explain the issue, but it doesn't fully capture what happens in C. The compiler knows that data-race-free behavior is the only valid one and may optimize by merging load/store operations, caching them into registers, or applying other transformations.

Therefore, the erroneous program does not guarantee that "each time it prints a different number less than 2,000,000." It might print the expected value occasionally, or it might exhibit other results due to compiler optimizations. Running the program repeatedly without failure is not proof of correctness. Only race detectors, code reviews, and proper synchronization mechanisms provide meaningful assurance.

A Mutex Provides Mutual Exclusion and Memory Synchronization

When pthread_mutex_lock succeeds, the thread holds the lock exclusively; unlock allows other participants to obtain the lock. All code that accesses the same invariant must follow the same synchronization protocol.

lock(mutex)
  read shared state
  validate / modify the whole invariant
unlock(mutex)

A mutex protects an invariant, rather than a single assignment alone. Suppose a bank transfer must simultaneously update two account balances. Locking each individual store separately could still allow an observer to see an intermediate, inconsistent total.

The unlock operation and the subsequent successful lock establish the required synchronization between POSIX/C threads, ensuring that writes from the first critical section are visible to the next thread that acquires the lock. It's not merely about "preventing the CPU from executing two lines of code simultaneously." Chapter 16 will precisely define this relationship using acquire/release semantics and the C memory model.

A Verified Batched Count

Locking for every increment is correct in principle, but causes all threads to contend for the same cache line. If the application only requires the final total, a thread-private local variable can be used to accumulate values, with a final merge performed at the end:

#define _POSIX_C_SOURCE 200809L
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

enum {
    THREAD_COUNT = 4,
    ITERATIONS_PER_THREAD = 250000
};

static pthread_mutex_t counter_mutex = PTHREAD_MUTEX_INITIALIZER;
static long long counter = 0;

static void fail_pthread(const char *operation, int error) {
    fprintf(stderr, "%s: %s\n", operation, strerror(error));
    abort();
}

static void *count_batch(void *unused) {
    (void)unused;
    long long local = 0;
    for (int index = 0; index < ITERATIONS_PER_THREAD; index++) {
        local++;
    }

    int error = pthread_mutex_lock(&counter_mutex);
    if (error != 0) {
        fail_pthread("pthread_mutex_lock", error);
    }
    counter += local;
    error = pthread_mutex_unlock(&counter_mutex);
    if (error != 0) {
        fail_pthread("pthread_mutex_unlock", error);
    }
    return NULL;
}

int main(void) {
    pthread_t threads[THREAD_COUNT];

    for (int index = 0; index < THREAD_COUNT; index++) {
        int error = pthread_create(&threads[index], NULL, count_batch, NULL);
        if (error != 0) {
            fail_pthread("pthread_create", error);
        }
    }
    for (int index = 0; index < THREAD_COUNT; index++) {
        int error = pthread_join(threads[index], NULL);
        if (error != 0) {
            fail_pthread("pthread_join", error);
        }
    }

    long long expected =
        (long long)THREAD_COUNT * ITERATIONS_PER_THREAD;
    printf("expected=%lld actual=%lld\n", expected, counter);

    int error = pthread_mutex_destroy(&counter_mutex);
    if (error != 0) {
        fail_pthread("pthread_mutex_destroy", error);
    }
    return counter == expected ? 0 : 1;
}

This code merges one million source-level increments into four shared read-modify-write operations. The compiler can fold the local loop into a constant; this demonstrates correctness, not the measured cost of executing a million units of work. The precondition is that intermediate values do not need to be visible to other threads in real time. If the counter also carries semantics such as progress tracking, quota enforcement, or cancellation signaling, batching must not be applied arbitrarily.

A Mutex Does Not Require a System Call for Every Lock

Uncontended POSIX mutexes are typically handled quickly in user space using atomic state operations. When contention occurs and a thread must sleep or wake, the Linux implementation may use futexes. The exact strategy depends on the mutex type, adaptive spinning, and priority inheritance mechanisms.

Therefore, the misconception that "every lock results in a system call and is therefore two orders of magnitude slower" is incorrect. The actual cost of a lock depends primarily on:

• The level of contention and the duration of the critical section;
• Whether ownership crosses cores, leading to cache-line invalidation;
• Whether threads sleep or wake, introducing scheduler latency;
• Whether the lock is fair or employs a priority protocol;
• Whether I/O, memory allocation, or nested locking occurs within the critical section.

The typical optimization path begins with reducing shared state, batching operations, or partitioning access. Replacing a global mutex with custom lock-free code should only be considered after ensuring robust memory reclamation, otherwise, a clearly identified bottleneck may simply be transformed into a hidden bug.

Error Paths Must Also Release Locks

C lacks RAII. When multiple return or goto/error paths exist within a critical section, it's essential to ensure every path properly unlocks the resource. This integration fragment uses one final unlock path. Include <pthread.h> and provide the Store, Record, and business-function declarations first. It assumes a non-robust mutex: EOWNERDEAD from a robust mutex can mean the lock was acquired and its state needs recovery.

static int update_record(struct Store *store, const struct Record *record) {
    int error = pthread_mutex_lock(&store->mutex);
    if (error != 0) {
        return error;
    }

    int result = validate(record);
    if (result == 0) {
        result = write_record(store, record);
    }

    error = pthread_mutex_unlock(&store->mutex);
    return result != 0 ? result : error;
}

Here, validate and write_record are business logic functions that must guarantee they do not violate lock ordering or cause indefinite blocking. If these functions may be interrupted by thread cancellation, additional consideration must be given to how cleanup handlers interact with cancellation points; a simple "unlock before every return" is still insufficient.

pthread_cleanup_push/pop is a macro-based API with lexical scope requirements. Without a clear understanding of cancellation behavior, a safer approach is often to disable deferred cancellation or restrict exit points to well-controlled boundaries.

Python and Java Share Similar Concepts, But Differ in Execution Semantics

Java synchronized and ReentrantLock, as well as POSIX mutexes, can establish mutual exclusion and language-defined visibility, but they differ in fairness, reentrancy, interruptibility, and exception cleanup. Leaving a synchronized block releases its monitor automatically. After successfully locking a ReentrantLock, use try/finally to unlock. Lock does not implement AutoCloseable and cannot be used directly in try-with-resources.

In a GIL-enabled CPython build, the GIL protects interpreter internals, not application invariants spanning multiple operations. Threads may switch between operations, and I/O or C extensions may release the GIL. Since 3.13, CPython also offers free-threaded builds that can disable the GIL; one build’s behavior is not a universal Python guarantee. Shared mutable objects should still be protected using threading.Lock, queues, or higher-level concurrent containers.

Python thread support for CPU-bound pure Python parallelism depends on the implementation and runtime mode; I/O concurrency, native extensions, and multiprocessing each involve different trade-offs. Do not conclude that "because Python has a GIL, there are no data races."

Choosing Between Threads, Processes, and Event Loops

Requirement	Preferred Approach	Trade-offs
Sharing large objects and performing parallel computation	thread / task runtime	Explicit synchronization; weaker failure isolation
Isolating untrusted or error-prone components	process	Inter-process communication (IPC), serialization overhead, resource duplication
Handling a large number of mostly-waiting connections	event loop or structured async	Callback/task lifecycle complexity, backpressure challenges
CPU-intensive tasks	bounded worker pool	Queueing, load balancing, risk of oversubscription
Real-time deadline requirements	RT thread/policy + bounded design	Priority inversion, allocation or I/O constraints

These approaches can be combined. Nginx uses multiple worker processes and an event-based model. A Java server may combine an event loop with a worker pool. Choose based on sharing, blocking behavior, failure domains, and deployment constraints.

Microsecond- or millisecond-level startup costs vary significantly depending on the OS, stack policy, address space size, and security configuration. In practice, workload performance is typically more influenced by task granularity and data movement than by these initial overheads.

Debugging Data Races

Available tools include:

• Clang/GCC ThreadSanitizer to detect many dynamic data races;
• Helgrind or DRD to observe lock operations and memory accesses;
• compiler warnings and static analyzers to identify error-prone paths;
• deterministic stress testing, fault injection, and model checking to expand interleaving coverage.

-fsanitize=thread significantly alters timing, yet data-race reports remain more reliable than the heuristic of "adding a sleep to make the bug disappear." Sanitizers cannot guarantee the absence of races: unexecuted paths, unsupported primitives, and false negatives still require manual review.

The debugging output itself involves locks and I/O, which can alter scheduling, this is a common source of Heisenbugs. Whenever possible, use structured traces, per-thread buffers, or external profilers to avoid casually printf on critical timing paths.

Build and Check a Mutual Exclusion Protocol

1. Explain why the unsafe counter's outcome isn't merely a lost update; it's actually undefined behavior as defined by C standards.
2. Modify the batched counter to merge every 1000 operations, then compare real-time performance, contention levels, and cache transfer overhead.
3. Design a queue invariant that simultaneously maintains size, head, and tail, and clearly identify the scope of the mutex that must protect this invariant.
4. Inject write_record failure in error paths to verify that the mutex is always released, even in failure cases.
5. Run a deliberately data-race-prone version and a correctly synchronized mutex version using ThreadSanitizer, then compare the resulting reports.
6. List why, in a multi-threaded process, after fork, the child process cannot safely invoke arbitrary libraries.

Mutex Is Not Enough

A mutex ensures that only one thread modifies the state at a time, but it doesn't allow consumers to efficiently sleep until the queue is non-empty. It also fails to express a finite number of resources or stage convergence. The next lesson covers condition variables, semaphores, and barriers, focusing on predicates and wake-up protocols.

Check the Interface Contracts

• Java 21 Lock: memory synchronization and try/finally
• CPython free-threading guide
• Nginx process and event model