18.2 Using perf to Build Performance Evidence

As the journey to the core’s exit approaches, the console leaves only a vague message: “The program is slow.” Guessing whether the issue lies in cache, branch prediction, or locking could waste an entire day, even on the most elegant code. Performance instrumentation doesn’t offer conclusions on its own, but it transforms hypotheses into testable assumptions.

Linux perf supports both hardware PMU counters, software events, and tracepoints. It answers two distinct types of questions:

• perf stat Counts how many events occur during a given execution period;
• perf record/report Uses sampling to estimate where events are concentrated, specifically, which instructions and call stacks are most active.

Counters are not call stacks, and sampling does not capture a complete log of every event. Always start by clearly defining your problem and workload, then select the appropriate mode.

1. Four Checks Before Data Collection

Does the Workload Represent the Problem?

Run the program for a sufficient duration to expose real performance bottlenecks, and record the data scale, concurrency levels, and input distribution. Profiling only the startup phase will yield hotspots dominated by dynamic linking and initialization, these are not representative of actual runtime performance.

Is the Binary Symbolizable?

Preserve standalone debug symbols and track the build ID. Avoid deleting the corresponding binary before analysis. Optimized builds can still be used; -g does not equal -O0.

Call stacks also depend on unwind mechanisms. Frame pointers, DWARF CFI, and hardware LBR each have distinct availability and performance trade-offs, no single option works universally across CPU architectures, languages, or production environments.

What Events Are Available on the Current Platform?

perf list

Event names are determined by the capabilities exposed by the CPU PMU, the kernel, and virtualization layers. Never copy raw event codes from another machine. On heterogeneous multi-core platforms, multiple PMUs may exist, and events with the same name must be distinguished by unit.

Permissions and Scope

System configurations restrict regular users from accessing kernel addresses, other processes, or system-wide data. Containers may also lack PMU access privileges. When encountering permission errors, authorize access only to the minimal required scope according to organizational security policies, never disable system-wide protections outright.

2. perf stat: Start with the Big Picture

Begin with the default statistics:

perf stat -r 5 -- ./application arguments

-r 5 Run repeatedly and report variance information, this helps identify obvious instability. When you need specific events, refer to perf list and select accordingly:

perf stat -e cycles,instructions,branches,branch-misses +  -- ./application arguments

Not all platforms support this set of event names. If the command fails or displays <not supported>, first verify PMU availability and permissions.

Read Before You Guess

elapsed time represents end-to-end duration. user/system time helps distinguish between user-space and kernel-space CPU activity, though total CPU time across threads can exceed wall-clock time.

instructions/cycles can be used to compute IPC (instructions per cycle). Low IPC may stem from dependency chains, cache misses, branch mispredictions, front-end starvation, or threads not actually gaining CPU access, it does not alone indicate a memory bottleneck.

branch-misses/branches shows the proportion of branch predictions that failed. Compilers may have transformed source code if into conditional moves, and the event behavior is also influenced by PMU definitions.

cache-misses is a broad event and doesn't necessarily correspond to a specific cache level or type of load. To identify cache hierarchy, TLB behavior, or remote NUMA effects, platform-specific events or memory profiling tools are required.

If the requested events exceed the available hardware counters, perf will multiplex them. Check the runtime proportions and scaling factors in the output; it's not advisable to directly construct high-precision ratios from events sampled at different times.

3. perf record: Where are the hotspots?

Basic CPU sampling workflow:

perf record --call-graph dwarf -- ./application arguments
perf report

perf record writes samples to perf.data, and perf report then aggregates them by symbol, DSO, and call relationship. The default sampling event and frequency are determined by the platform and perf version. For reproducible experiments, explicitly record the command, version, and event used.

If the build preserves the frame pointer, you can assess:

perf record --call-graph fp -- ./application arguments

Frame-pointer unwind overhead is low, but it may result in broken or incorrect stacks when frame pointers are omitted or mixed runtime environments are encountered. DWARF collects user stack snippets and expands them using CFI (Call Frame Information), resulting in higher data volume and overhead. LBR (Last Branch Record) is only available on certain platforms and has limitations on call stack depth and combination options.

Don't interpret business logic just because the stack appears broken. First, validate the unwind quality using known call chains.

4. Inclusive vs. Self Overhead

Assume:

handle_request
  ├─ parse
  └─ query_database
       └─ decode_rows

The inclusive cost in handle_request includes all subcalls; the self cost only accounts for the portion of the sampling points that fall within the current function's execution. A wide parent frame does not imply that the parent function performs heavy computation, it might simply invoke an expensive child function.

When optimizing, trace up the widest call path to identify ownership, and down the path to locate actual consumption:

• A leaf at the top with wide coverage: cost is concentrated within the function itself;
• A middle frame that is wide but has low self cost: it acts as an aggregation entry point;
• The same leaf appearing across multiple paths: determine which caller contributes more significantly;
• Many [unknown] or hexadecimal addresses: first resolve symbols and perform unwinding.

The sampling percentage is an estimate of the selected event, not necessarily equal to the wall-time proportion. Sampling cycles, cpu-clock, or cache events yields different interpretations of the resulting diagram.

5. Flame graphs are another form of call stack projection

If the appropriate stack collapse and FlameGraph scripts are installed, you can:

perf script > samples.perf
stackcollapse-perf.pl samples.perf > samples.folded
flamegraph.pl samples.folded > cpu.svg

Reading principles:

• The horizontal width indicates how many sampled events include that frame;
• The vertical dimension represents call depth, not duration;
• The horizontal position does not typically indicate temporal order;
• Default colors are used primarily for visual distinction and do not represent temperature or severity;
• A wide platform at the top often corresponds to actual on-CPU leaf functions, while broad boxes at the bottom typically represent entry points or aggregated call paths.

Flame graphs sacrifice temporal context. For issues involving periodic jitter, brief spikes, or early startup phases, time-series data, windowed profiling, or trace analysis are often more appropriate.

6. CPU Flame Graphs Don't Show Sleep Time

When a thread is blocked on:

• a mutex or condition variable;
• disk or network I/O;
• a page fault;
• the scheduler's run queue;
• downstream RPC calls;
• a runtime safepoint;

it does not consume user-space CPU time. As a result, on-CPU samples won’t extend across wall time to show the duration of the wait.

First, use business traces, thread states, and system metrics to determine whether the issue is "slow execution" or "long waiting." On the Linux side, scheduler tracepoints, perf sched, perf lock, eBPF tools, or application-level tracing can be employed depending on the scenario. Specific commands and permissions vary by version, start by running help/list locally to identify available data.

Off-CPU analysis requires recording both the start and end of a block, and attributing the wait duration to the corresponding stack trace. A single sample of a sleep function call is insufficient to capture the actual duration of the wait.

7. From CPU Hotspots to Cache and Shared State

After identifying that a loop consumes a large portion of cycles, we refine our hypotheses with more specific assumptions:

Working set exceeds cache capacity. Plot throughput against data scale and look for a breakpoint at a certain capacity threshold, this would indicate that the working set is larger than what fits in cache.

Lack of locality. Compare access patterns across contiguous arrays, pointer chasing, or field splitting in structs, and examine the resulting event timing and memory access sequences.

False sharing. Multiple threads modify different variables that happen to reside in the same coherence unit, causing cache lines to be migrated between cores.

True shared writes. Multiple threads simultaneously modify the same counter or queue head, simple padding won’t eliminate contention. Instead, solutions like data partitioning or algorithmic redesign are required.

perf c2c can help identify cache-to-cache sharing on supported platforms. However, general LLC-load-misses cannot definitively prove false sharing: data might originate from main memory, or the access could simply be a normal cache miss. Moreover, the event semantics vary significantly across architectures.

Fixing false sharing typically involves keeping data local per thread and aggregating results at the end. Padding or alignment is only appropriate when you’ve confirmed that fields actually reside in the same cache line and their lifetimes allow such a layout. Cache line size and structure alignment must be validated on the target platform, never assume a 64-byte cache line is a universal truth in C.

8. Branches, Frontend, and Assembly

If branch miss rates appear suspicious:

1. Use perf report to locate the functions within the sample set;
2. Use perf annotate to align the source code with the assembly output;
3. Verify that the source code branches still exist;
4. Examine the input distribution and layout of hot execution paths;
5. After modification, compare cycles, instructions, branch misses, and total execution time side by side.

A branch-free version may execute more instructions, and conditional moves can extend dependency chains. __builtin_expect primarily assists the compiler with layout and static decision-making, it does not directly encode probabilities into hardware predictors.

Frontend issues may also stem from instruction cache capacity, decoding bandwidth, or excessive inlining. A wide function in a flame graph does not imply that further inlining will yield better performance; code bloat can cause hot paths to displace each other from instruction cache.

9. Sampling Bias and Measurement Overhead

Sampling is not a disturbance-free observation:

• Sampling at too high a frequency increases interrupt, record, and unwind overhead;
• A too-small buffer results in dropped samples;
• Short functions may be misaligned due to skid or event precision, being grouped with nearby instructions;
• Dynamically generated code requires proper JIT symbol support;
• Container PID or mount namespace configurations can affect symbol path resolution;
• System-wide profiling may capture sensitive call stacks and memory addresses;
• Changes in program phase can cause aggregate metrics to mask localized issues.

Evaluate overheads first in a test environment. Production-level sampling should limit duration, frequency, event types, target PID or cgroup, and data access permissions, and must comply with privacy and operations procedures.

When comparing two versions, use the same sampling methodology. If the observed performance improvement is comparable to the profiler’s own overhead or runtime variance, the conclusions are not robust.

10. From "Slow Code" to a Reviewable Evidence Chain

There are many readings from monitoring tools, but troubleshooting shouldn’t start with random events. First, fix the workload and symptoms. Then, narrow down from the overall system view to specific hotspots, call stacks, cache behavior, or scheduling patterns. At every step, you must be able to explain exactly what the next command is answering.

SLO degradation
  │
  ├─ Is CPU saturated?
  │    ├─ Yes: perf stat → record/report → annotate
  │    └─ No: examine queueing, locks, I/O, downstream services, and runtime behavior
  │
  ├─ What constraints are limiting performance?
  │    ├─ Instruction set or algorithmic complexity
  │    ├─ Branch or front-end bottlenecks
  │    ├─ Cache, TLB, or NUMA effects
  │    ├─ Shared cache lines or locking contention
  │    └─ System calls or device I/O
  │
  └─ Minimal change → re-run correctness, SLO, resource usage, and profiling

Tool selection follows your hypotheses, not the need to explain every line of the default perf stat. An anomalous counter is typically just the entry point for the next experiment.

11. Summary

perf turns performance intuition into verifiable evidence:

• stat provides a running event ledger;
• record/report uses sampling to pinpoint event-concentrated code and call paths;
• frame-pointer, DWARF, and LBR are different methods for constructing call graphs;
• flame graph width reflects the selected sample, not an automatically derived wall-time percentage;
• on-CPU profiling does not reveal full wait states involving locks, I/O, or queuing;
• cache misses do not automatically prove false sharing, more specific event data and memory layout evidence are required;
• events, PMU, and permissions vary by machine; start with perf list and native documentation.

By this point, Volume 3 has taken us from coding, CPU behavior, memory, processes, to performance validation. The next stop is Volume 4: Computer Networks.

References

• perf stat manual
• perf record manual
• perf report manual