18.1 From Measurement to Performance Conclusion

Instruction, memory, cache, and concurrency concepts converge in one performance-engineering task: use measurements to find the real bottleneck, then determine whether an optimization helps. “This line looks slow” is a suspicion; reproducible evidence is a conclusion.

A meaningful conclusion specifies: which workload, which metric, under what environment, relative to which baseline, with what level of error margin, and whether the performance bottleneck has shifted after a change.

This chapter establishes the methodology for performance testing. Specific Linux perf sampling and counter usage is detailed in Linux Performance Profiling.

1. First, Turn "Fast" Into a Verifiable Goal

Performance goals across different systems can conflict:

• Batch systems care about throughput per hour and cost;
• APIs focus on p50/p95/p99 latency and timeout rates;
• Interactive applications prioritize frame time and jank;
• Databases track throughput, commit latency, and recovery guarantees;
• Embedded systems also consider power consumption and worst-case latency.

Rewrite "optimize the interface" into a measurable, verifiable goal:

Under production-like data distribution and 500 concurrent connections,
the p99 server latency for checkout drops from 420 ms to 250 ms,
error rates remain unchanged, and per-request CPU time increases by no more than 10%.

This is a target example, not a measured result. Specify the arrival model, duration, and completed work too: 500 connections may be idle and do not imply 500 in-flight requests. Focusing solely on the average can mask a few extremely slow requests; focusing only on p99 might overlook a drop in throughput or a doubling of resource costs.

2. End-to-end Latency Attribution by Layer

The wall-clock latency of a single request can be roughly broken down into:

Waiting in queue
+ CPU execution
+ Memory stalls
+ Lock or scheduler waiting
+ File and network I/O
+ Waiting for downstream services
+ Runtime pauses

This is a diagnostic checklist, not a partition to add up. CPU time normally includes memory stalls while the thread remains scheduled; downstream time may already include network and queueing. Parallel branches of one request contribute through its critical path, while other requests can use the CPU during asynchronous waits. The purpose of attribution is not to present a pie chart and declare causality, but to identify the next actionable measurement.

First, establish three layers of observation:

1. Business layer: request rate, success rate, latency distribution, queue depth;
2. Process layer: CPU time, thread states, allocation, GC activity, system calls;
3. System layer: scheduling, page faults, disk I/O, network I/O, memory pressure, and interference from other processes on the same machine.

A CPU profiler can only explain samples that are actively running on CPU. When threads spend most of their time waiting on locks or network I/O, a pure on-CPU flame graph appears "calm", yet the system is not necessarily bottleneck-free.

3. Performance Loop: Measurement, Hypothesis, Validation

A robust iteration follows this sequence:

Define the goal
  ↓
Fix the workload and environment
  ↓
Establish a repeatable baseline
  ↓
Identify the primary cost
  ↓
Formulate a single, falsifiable hypothesis
  ↓
Make the minimal change
  ↓
Re-measure correctness, performance, and side effects
  ↓
Retain the result or revert

Each iteration should modify only one key factor to reduce attribution confusion. Changes such as compiler version, CPU governor, NUMA placement, dataset, concurrency level, and background tasks can all influence outcomes; each must be documented in the experiment record.

After optimizing a particular component, profiling must be repeated. The previously dominant hotspot may shrink, revealing new bottlenecks such as queueing, locks, or downstream constraints. Performance engineering is not a one-time list of hotspots.

4. Amdahl's Law Gives an Upper Bound on Local Optimization

If a portion of the total runtime accounting for proportion p can be accelerated N times while the rest remains unchanged, the overall speedup is given by:

$$S=\frac{1}{(1-p)+\frac{p}{N}}$$

For example, if 40% of the runtime can be sped up 4 times:

$$S=\frac{1}{0.6+0.4/4}=\frac{1}{0.7}\approx 1.43$$

Speeding up that portion 4x leaves total runtime at 70% of the baseline: a 30% time reduction and about 1.43x speedup, not a 43% time reduction. This helps prevent teams from spending weeks optimizing functions that represent a tiny fraction of the workload.

The function accepts int/float inputs representable as floats, with positive infinity allowed for ideal unlimited local speedup. It rejects bool, NaN, and invalid ranges:

import math

def amdahl_speedup(fraction, local_speedup):
    if type(fraction) not in (int, float) or type(local_speedup) not in (int, float):
        raise ValueError("inputs must be int or float, not bool")
    try:
        fraction, local_speedup = float(fraction), float(local_speedup)
    except OverflowError as error:
        raise ValueError("input exceeds float range") from error
    if not math.isfinite(fraction) or not 0 <= fraction <= 1:
        raise ValueError("fraction must be finite and between 0 and 1")
    if math.isnan(local_speedup) or local_speedup <= 0:
        raise ValueError("local speedup must be positive, with +inf allowed")
    if fraction == 0:
        return 1.0
    if fraction == 1:
        return local_speedup
    return 1 / ((1 - fraction) + fraction / local_speedup)

assert round(amdahl_speedup(0.40, 4), 2) == 1.43
assert amdahl_speedup(0, 100) == 1
assert amdahl_speedup(1, 8) == 8

limit = amdahl_speedup(0.95, float("inf"))
print(f"5% unchanged work caps speedup at {limit:.1f}x")

The Amdahl model assumes a fixed workload size, with known proportions and acceleration costs. In practice, parallelization introduces additional overhead from synchronization, communication, and memory bandwidth contention; the proportion of parallelizable work also shifts as problem size grows. This formula serves as a theoretical upper bound, useful for reasoning about limits, not for capacity planning.

5. Little's Law Connects Concurrency, Throughput, and Latency

For a system in steady state:

$$L=\lambda W$$

• L: the average number of requests in transit within the system;
• \lambda: the average request completion rate;
• W: the average request residency time.

If the steady-state throughput is 1,000 requests per second and the average residency time is 0.2 seconds, then there are approximately 200 requests actively in the system at any given time.

L, λ, and W must use the same system boundary, request population, and observation window. Include failed and cancelled departures consistently. The law does not require Poisson arrivals, and p99 cannot replace mean W.

This relationship serves as a consistency check, but it does not guarantee that achieving a certain level of concurrency will sustain the desired throughput. As the system approaches resource saturation, queue growth will increase W. When input rates remain consistently higher than completion rates, the system ceases to be stable, and the steady-state interpretation of Little's Law no longer holds.

Bounding queues, enforcing deadlines, and implementing backpressure are often more effective at protecting tail latency than optimizing a few lines of code.

6. Percentiles Require Proper Sampling

Average latency fails to capture tail behavior. The p99 percentile indicates that approximately 99% of samples are no greater than that value, but it has boundaries:

• With insufficient sample size, high percentiles become unstable;
• Merging data from different endpoints, payloads, or tenants can blur the true distribution;
• Client latency and server processing time are distinct and often not aligned;
• Timeouts that are dropped from the trace can make percentiles appear artificially better;
• When the load generator itself becomes saturated, the actual request arrival rate drops, skewing results.

Be cautious of coordinated omission: when the intended arrivals are independent of responses, a generator that waits for each slow response omits requests and queueing demand that should have arrived meanwhile. It then understates tail latency. Closed-loop tests are valid when they match the actual usage model. Open-loop arrival rate and closed-loop concurrency models answer different questions; test reports must clearly specify which model was used.

7. Micro-benchmarks Are Prone to Measuring the Wrong Things

Micro-benchmarks are well-suited for measuring the cost of a small operation, but they are not reliable for predicting the performance of a full service. Common pitfalls include:

Compiler Elimination or Folding

When results are not observed, optimizers can eliminate entire loops. If all inputs are constants, the compiler may compute the result at compile time. Prefer using mature benchmark frameworks with black-box or consume-based mechanisms, and inspect the generated assembly. Simply writing results to volatile might suppress certain optimizations and alter the actual code being measured; this is not a universal solution.

Including Setup in the Target Operation

Each round of container reallocation, random number generation, or file reading introduces setup overhead that gets mixed into the operation being measured. If setup is part of the actual production path, it should be preserved. Otherwise, prepare it outside the timing scope and report it separately.

Warmup and Runtime Compilation

JVMs, JavaScript runtimes, and some Python environments interpret, compile, de-optimize, and recompile code. Both cold-start and steady-state performance may be relevant product metrics; do not just say “warm up” without specifying which phase you're measuring.

C/C++ also faces page-fault on first touch, dynamic linking, cache warming, and CPU frequency changes. The absence of JIT does not mean these effects can be ignored; choose whether the experiment measures cold-start or warmed execution.

Unrealistic Input Distributions

Inputs like all zeros, perfectly sorted data, or data that fits neatly into cache can produce performance characteristics that diverge significantly from real-world usage. Datasets must be both reproducible and representative of actual sizes and skew.

Timers and Batch Execution

If the operation being measured is shorter than the timer overhead, execute it in bulk and divide the total by the number of iterations. Always measure an empty baseline. Be aware whether the compiler can merge or vectorize the batch loop. Batch duration divided by iterations is average per-operation cost, not an individual latency distribution or p99. Report repeated measurements and uncertainty, distinguishing iterations, batches, and independent process runs instead of treating correlated samples as independent evidence.

Environmental Noise

CPU migration, frequency scaling, SMT neighbors, NUMA effects, background tasks, temperature fluctuations, and virtualization steal time all introduce variance. A stable environment isn’t about producing clean, pretty numbers; it’s about enabling meaningful comparisons between experiments. Validation in real-world environments must still occur before deployment.

8. Control Experiments Must Preserve Correctness

"Optimized to be faster" might simply mean less work was done. Every performance comparison must first verify:

• Output consistency or adherence to a newly defined semantics;
• Error, timeout, and cancellation paths have not been eliminated;
• Data persistence and synchronization guarantees have not been weakened;
• Memory peaks, CPU usage, and I/O load have not been shifted to other components;
• Input scale and number of completed requests are identical;
• Compilation flags and dependency versions are comparable.

For example, removing fsync may significantly speed up commits, but it alters the data guarantees after a crash. Replacing a mutex with a synchronization-free variable may "improve throughput," yet introduces undefined behavior. Correctness is part of the baseline, not an optional trade-off in a performance optimization table.

9. Optimize Starting from Order of Magnitude

A common sequence of gains is:

1. Remove unnecessary work;
2. Improve algorithmic complexity;
3. Reduce cross-process or network round trips;
4. Batch processing and caching reusable results;
5. Improve data layout and I/O patterns;
6. Reduce lock contention and shared hotspots;
7. Finally, fine-tune at the instruction level.

This is not an absolute checklist. Real-time systems or minimal kernels may start by optimizing at the instruction level; distributed services are often dominated early by network and storage overhead. The principle is to prioritize costs that represent the largest share and have clear, measurable evidence.

Caching is not a universal gain. Caching introduces invalidation, capacity limits, stampedes, consistency issues, and increased memory usage. Batch processing reduces per-unit fixed costs but may increase individual request latency. Any optimization must return to the original SLO.

10. Performance Regression Gatekeeping

CI-friendly gatekeeping should tolerate noise by:

• Selecting a stable, representative subset of benchmarks;
• Fixing machine type, CPU sets, and dependencies;
• Preserving raw samples rather than just aggregated results;
• Running multiple iterations and using statistical comparison;
• Setting thresholds that balance real impact with variance;
• Predefining retest counts and combined decision rules, retaining the first failure instead of rerunning until green;
• Periodically validating benchmarks against production profiles to ensure they still reflect real performance hotspots.

Passing microbenchmarks does not imply end-to-end SLO compliance. A layered gatekeeping approach can simultaneously retain algorithm-level benchmarks, component load tests, and a small number of end-to-end scenarios.

11. A Reproducible Experiment Record

Every optimization must at least preserve:

Objective and metrics:
Code version and build parameters:
Hardware, kernel, runtime:
Dataset and input distribution:
Load model and duration:
Warm-up and sampling window:
Baseline raw results:
Profile evidence:
Changes and assumptions:
New results and variance:
Correctness verification:
CPU/memory/I/O side effects:
Conclusion and scope of applicability:

Without raw data and environment details, a flashy "37% faster" claim is hard to reproduce. Without a clearly defined scope of applicability, it's easy to incorrectly generalize the result to a different CPU, dataset, or request path.

12. Summary

Performance engineering is an evidence chain:

• Define "fast" using business SLOs;
• Break end-to-end latency into measurable components: execution, waiting, and queuing;
• Use Amdahl’s law to estimate how much total latency a local optimization can affect;
• Apply Little’s law to verify that concurrency, throughput, and latency are consistent;
• Avoid benchmarking illusions by using real distributions, accurate load models, and repeated sampling;
• For every change, validate both semantic correctness and resource side effects;
• After optimization, re-profile to expose new bottlenecks.

The next section moves into Linux Performance Profiling, where we apply these methods to practical tools like perf stat, call stack sampling, flame graphs, and off-CPU analysis. After completing this section, Volume 3 will transition readers to Volume 4: Computer Networks.