14.1 Network Performance Measurement and Bottleneck Identification

"The API is slow" is not an actionable problem. It doesn't specify which users are affected, which percentile of latency is involved, or which stage of the request flow is slow. It also fails to distinguish between failed and successful requests. The first step in performance optimization isn't tweaking parameters; it's establishing a set of repeatable, comparable, and explainable measurements.

Learning Objectives

• Distinguish between latency, throughput, goodput, concurrency, and loss;
• Accurately explain the phased timing of curl;
• Design experiments that include baselines, variables, and rollback conditions;
• Describe user experience using percentiles and error rates;
• Narrow down symptoms to specific stages: DNS, connection, TLS, first byte, or transmission.

Performance Is Not a Number

Metric	Answers the Question	Common Misconceptions
latency	How long does a single operation take?	Only reporting averages, hiding long-tail delays
throughput	How much work is completed or how many bytes are transferred per unit time?	Confusing it with goodput
goodput	The actual rate of useful application data	Ignoring retransmissions, headers, and duplicate data
concurrency	How many operations are simultaneously being processed or waiting?	Equating it directly with throughput
error/timeout rate	What percentage of requests fail or time out?	Only measuring latency for successful requests
saturation	Which resource is approaching its capacity boundary?	Seeing low CPU usage and assuming there's no queueing

User experience typically requires percentiles like p50, p95, and p99, rather than an average alone. An average of 100 ms could come from "all requests taking about 100 ms" or "99% taking 10 ms and 1% taking 9.01 seconds." The engineering responses to these two scenarios are fundamentally different.

Timeouts and errors must be reported separately. If a performance script only calculates percentiles for successful responses, a system with more failures and timeouts will appear faster, because the remaining successful requests look better in aggregate.

What Components Make Up Request Time

A new HTTPS request might go through the following stages:

name resolution
  -> route/address selection
  -> TCP or QUIC connection
  -> TLS handshake
  -> request queued/sent
  -> edge/proxy/application/upstream work
  -> first response byte
  -> response body transfer

After connection reuse, DNS, TCP, and TLS may no longer occur. When passing through an HTTP proxy, additional stages (such as travel to the proxy and from the proxy to the destination) appear. HTTP/3, which is built on QUIC, cannot be interpreted using the concept of "TCP latency." Therefore, any timing breakdown must simultaneously record the protocol version, whether connection reuse is in effect, proxy involvement, and the location of the test.

Use curl to obtain a stage snapshot

curl --disable --silent --show-error --output /dev/null \
  --connect-timeout 3 --max-time 15 \
  --write-out $'url=%{url_effective}\nremote_ip=%{remote_ip}\nhttp_version=%{http_version}\nresponse_code=%{response_code}\nnum_connects=%{num_connects}\nnum_redirects=%{num_redirects}\nnamelookup=%{time_namelookup}\nconnect=%{time_connect}\nappconnect=%{time_appconnect}\npretransfer=%{time_pretransfer}\nstarttransfer=%{time_starttransfer}\ntotal=%{time_total}\nsize_download=%{size_download}\nspeed_download=%{speed_download}\n' \
  https://example.com/

The timing values from curl are typically accumulated from the start of transmission to a specific event point, rather than representing isolated stage durations. For a direct, new TCP + TLS connection without redirects or reuse, the following approximations can be made:

DNS               = time_namelookup
TCP after DNS     = time_connect - time_namelookup
TLS after TCP     = time_appconnect - time_connect
pre-first-byte    = time_starttransfer - time_pretransfer
body after first  = time_total - time_starttransfer

This is only an approximate breakdown for a specific request pattern:

• Redirects introduce multiple requests; simultaneously record num_redirects;
• Proxies alter the connection destination;
• Connection reuse may bring connection phases close to zero;
• For non-TLS requests, time_appconnect is typically zero;
• HTTP/3 connection timing semantics differ from TCP;
• time_starttransfer encompasses request transmission, network round-trips, proxy processing, and server-side handling, and thus cannot be directly labeled as "backend execution time."

Check curl --version and the matching official man page. As the first option, --disable skips the default curl configuration file; environment proxies can still apply, so record the actual path. This command does not follow redirects. HTTP 4xx/5xx also do not make curl exit nonzero by default; inspect the status code too.

One Measurement Is Not a Baseline

This script requires curl 7.70.0 or later for %{json}. It counts a completed transfer with HTTP 2xx as successful; add checks when the application expects a particular status or response content. Every run prints raw metrics and exit information, preserving available timing data on failure:

from __future__ import annotations

import json
import statistics
import subprocess
import time

FORMAT = "%{json}"

def sample(url: str, count: int = 10) -> None:
    rows: list[dict[str, object]] = []
    failures: list[str] = []

    for run in range(count):
        started_at = time.time()
        completed = subprocess.run(
            [
                "curl", "--disable", "--silent", "--show-error",
                "--output", "/dev/null", "--connect-timeout", "3",
                "--max-time", "15", "--write-out", FORMAT, "--url", url,
            ],
            text=True,
            capture_output=True,
            check=False,
        )
        record: dict[str, object] = {
            "run": run,
            "started_at": started_at,
            "returncode": completed.returncode,
            "stderr": completed.stderr.strip(),
        }
        try:
            row = json.loads(completed.stdout)
        except json.JSONDecodeError:
            record["raw_write_out"] = completed.stdout
            failures.append(f"run={run} invalid write-out; rc={completed.returncode}")
        else:
            record["metrics"] = row
            code = int(row["http_code"])
            if completed.returncode == 0 and 200 <= code < 300:
                rows.append(row)
            else:
                failures.append(f"run={run} rc={completed.returncode} http={code}")
        print(json.dumps(record, ensure_ascii=False))
        time.sleep(0.2)

    totals = [float(row["time_total"]) for row in rows]
    print(f"success={len(rows)} failure={len(failures)}")
    if totals:
        print(f"median={statistics.median(totals):.6f}s")
        print(f"min={min(totals):.6f}s max={max(totals):.6f}s")
    for failure in failures:
        print(failure)

if __name__ == "__main__":
    sample("https://example.com/")

Using curl’s own JSON avoids turning a failed connection’s status 000 into an invalid JSON number. Each iteration starts a separate curl process and establishes connections and TLS anew, though system DNS caches may be reused. This does not benchmark connection reuse or control caches and background traffic. The median covers successful samples only and must be read alongside failures. Ten samples cannot reliably characterize the tail, so the program does not report p95.

A slow request in this serial probe postpones the next one and reduces offered load. Capacity tests must specify a fixed arrival rate or fixed concurrency model and record the actual request rate; this lightweight probe cannot establish peak capacity.

Cold Paths and Hot Paths Must Be Separated

At a minimum, distinguish between:

• Cold DNS cache and hot DNS cache;
• New connections and connection reuse;
• TLS full handshake and session resumption;
• CDN miss, revalidation, and hit;
• Application cold start and steady-state operation;
• Small responses and large responses;
• Single requests and concurrent requests to the same target.

If these samples are mixed together, the observed distribution changes may simply reflect shifts in mixing ratios, rather than actual improvements or degradation in system performance.

Active Probing, Real User Monitoring, and Server Tracing

These three observation methods complement each other:

Method	Advantages	Limitations
synthetic probe	Controllable conditions, enables consistent baselines	Cannot capture the full range of user networks and devices
RUM	Reflects actual user distribution and behavior	Affected by browser APIs, privacy settings, sampling, and client-side conditions
server trace/metrics	Enables identification of queuing, application performance, and downstream dependencies	Cannot see all client-side DNS resolution and access paths

When curl shows a slow first byte, use a trace ID to stitch together timing data across edge services, load balancers, applications, databases, and external dependencies. Relying solely on client-side TTFB cannot conclusively determine that the database is slow.

The Relationship Between Latency, Throughput, and Concurrency

Little's Law in a stable system states:

L = λW

• L: the average number of concurrent requests in the system;
• λ: the average request rate (throughput);
• W: the average time a request spends in the system (residency time).

If the throughput is 1000 requests per second and the average residency time is 0.2 seconds, then there are approximately 200 requests actively in the system on average. This metric doesn't identify where the bottleneck lies, nor does it guarantee the system is in a steady state; however, it helps validate whether capacity figures agree. All three quantities need the same system boundary, request population, and time range. Do not combine successful-completion rate with an in-flight count that includes timed-out work, or substitute p99 for mean residence time.

Near saturation, queueing latency often grows nonlinearly. In such cases, increasing client concurrency may only result in more timeouts and retries, without improving actual throughput.

What Should Be Included in an Experiment Record

hypothesis: The tail of the first-byte latency stems from queuing in an upstream connection pool
scope: region=..., endpoint=..., payload=...
baseline: p50/p95/p99, error rate, throughput, saturation
change: Adjust only one connection pool limit
controls: Identical traffic model, dataset, protocol, and cache state
duration: Covers both warm-up and steady-state phases
success: p99 latency decreases without worsening error rate or downstream saturation
rollback: Immediate rollback if error rate or resource thresholds are exceeded
evidence: raw results, dashboard metrics, traces, version, timestamp

"Feeling faster after the change" cannot be verified or traced back during regression testing.

Lesson Summary

Performance optimization begins with measurable, repeatable data. The phase timing from curl offers a client-side perspective and must be interpreted in conjunction with connection reuse, protocol behavior, proxies, and server-side tracing. The next lesson will examine transport-layer constraints (specifically RTT, bandwidth, window size, packet loss, and queueing) to determine what truly limits throughput.

Resource Entry

• curl: Write-out variables and options.
• RFC 2330: Framework for IP Performance Metrics;
• RFC 7312: Advanced Stream and Sampling Framework for IPPM;
• RFC 5481: Packet Delay Variation Applicability Statement.