14.2 Transport Performance: RTT, Windows, Loss, Queues, and Connection Reuse

Increasing socket buffer sizes, switching congestion control algorithms, or adjusting the initial window may improve performance, yet they could also increase queuing, bursts, and memory consumption. Before making any adjustments, determine whether the connection is constrained by RTT, congestion window, receive window, application data supply, or bandwidth limitations.

The Four Sources of Delay

The time it takes for a packet to traverse a path can be roughly broken down into:

propagation + serialization + queueing + processing

• propagation: the signal travels through the medium, dependent on distance and path length;
• serialization: the time required to place each bit of the packet onto the link;
• queueing: waiting for link or device processing, which can grow sharply when load approaches capacity;
• processing: handling by protocols, encryption, forwarding, and application logic.

Serializing 1500 bytes over a 10 Mbit/s link takes approximately:

1500 × 8 / 10,000,000 = 1.2 ms

A full line-rate calculation would also account for link-layer overhead; this example only illustrates the order of magnitude.

Bandwidth-delay product

The bandwidth-delay product (BDP) is the product of the bottleneck bandwidth and the round-trip time (RTT), approximating the amount of data that would fill the path:

BDP = bandwidth × RTT

For 1 Gbit/s bandwidth and an 80 ms RTT:

1,000,000,000 bit/s × 0.08 s = 80,000,000 bit
80,000,000 / 8 = 10,000,000 byte ≈ 9.54 MiB

To estimate data needed to fill the path without extra queueing, use a near-unloaded RTT. Sizing larger buffers from an RTT already inflated by congestion can increase the queue further.

If the sender’s congestion window or receiver's advertised window remains significantly smaller than the BDP, a single connection may never fully utilize the link. However, setting the buffer size to the BDP does not guarantee achieving the full bandwidth. Application data generation rates, congestion control algorithms, packet loss, pacing mechanisms, receiver processing capacity, and intermediate network devices can all act as limiting factors.

Congestion Windows, Receive Windows, and Buffers

• congestion window, cwnd: an estimate of network capacity maintained by the sender for congestion control;
• advertised receive window, rwnd: the amount of receive buffer space advertised by the receiver;
• application buffer: the buffer configuration between the application and the kernel for sending and receiving data.

With both windows expressed in bytes, in-flight data is roughly bounded by min(cwnd, rwnd). Newly sendable data must also subtract bytes already in flight. Actual transmission is also constrained by pacing, application write operations, and protocol implementation details. The socket buffer and the wire-visible window are not simply equivalent; Linux additionally accounts for management overhead and performs autotuning.

For example, a 64 KiB window at 80 ms RTT gives an ideal single-flow bound of 65,536 × 8 / 0.08 = 6.55 Mbit/s, far below 1 Gbit/s. Linux ss commonly reports cwnd in segments, so multiply by MSS to estimate bytes.

TCP Window Scale is negotiated during the handshake. Changing settings later does not alter the scaling already negotiated on an existing connection.

Observe Linux TCP States

ss -tin dst 203.0.113.10

The output fields vary by kernel version. Common information includes RTT estimation, cwnd, MSS, retransmissions, pacing rate, delivery rate, and congestion control algorithms. Before interpreting the data, record the environment:

uname -r
ss -V
sysctl net.ipv4.tcp_congestion_control
sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem

These commands are used to retrieve TCP state information. Do not copy the example values directly to all hosts: behavior can be influenced by container namespaces, cgroups, kernel version, memory pressure, and default settings of the operating system distribution.

Why a Single Connection Fails to Reach Full Throughput

Check the following evidence:

1. Is the application continuously producing data, or is it constrained by disk I/O, CPU load, locks, or upstream throttling?
2. Is the cwnd × MSS / RTT throughput estimate close to the observed rate?
3. Is the receive window (rwnd) becoming the bottleneck? Is the receiving application reading data promptly?
4. Are there retransmissions, RTOs, or gaps/reordering indicated by SACK blocks? Negotiated SACK support alone does not establish packet loss.
5. How does the pacing or delivery rate compare to the interface’s capacity?
6. Can multi-stream aggregation improve throughput? If so, single-flow control or window limits deserve closer inspection.
7. Are there policy-based rate limits, traffic shaping, VPNs, tunnels, or cloud service quotas in place?

The formula provides only an upper-bound approximation. ACK strategies, offloading, application bursts, and measurement window duration all introduce short-term fluctuations in observed values.

Packet Loss Is Not the Only Congestion Signal

Traditional loss-based algorithms rely on packet loss as the primary congestion indicator; ECN can signal congestion without actually dropping packets; BBR-style algorithms model congestion based on delivery rate and RTT. ECN needs endpoint negotiation, packets marked as ECN-capable, network congestion marking, and feedback that makes the sender respond. A router marking packets alone is insufficient. Loss can also result from wireless errors or other non-congestion causes. Algorithm choice affects throughput, queue behavior, and fairness with other flows, and depends heavily on the specific kernel implementation.

You cannot universally assert that "BBR is always faster" or "CUBIC is always fair." A proper evaluation must cover at least the following scenarios:

• Short-lived versus long-lived flows;
• Varying RTT, bandwidth, and loss conditions;
• Fairness when coexisting with existing traffic;
• Interaction with bottleneck qdisc and AQM (Active Queue Management);
• CPU, memory usage, and retransmission behavior;
• The specific version of the algorithm implemented in the target kernel.

Production deployment requires gradual rollout, continuous monitoring, and rollback capability. Direct global changes to sysctl should be avoided.

Pacing, Queue, and Bufferbloat

A sudden burst of the entire cwnd creates a short queue at the bottleneck. Pacing distributes transmissions over a period of time, reducing burstiness. Fair queuing and AQM mechanisms (such as FQ-CoDel) can isolate flows and take action (like packet dropping or ECN marking) before the queue becomes excessively long.

Increasing queue or buffer size can absorb short bursts, but it may also lead to bufferbloat: throughput may appear normal, yet interaction latency rises significantly under load. Testing should simultaneously monitor both throughput and loaded latency, rather than relying solely on "the download saturates the link."

Connection Establishment and Reuse

TCP and TLS

Establishing a new TCP connection requires a handshake. TLS 1.3 full handshakes typically send application data earlier than TLS 1.2 full handshakes, and session resumption can further reduce handshake overhead. The number of handshake round trips depends on version, early data, client authentication, and transport. TLS 1.3 resumption without 0-RTT can reduce cryptographic or certificate work while still taking a 1-RTT handshake; resumption does not automatically mean zero round trips.

Optimization priorities are generally:

1. Avoid unnecessary new connections;
2. Reuse connections while bounding pool size and connection lifetime;
3. Correctly configure TLS 1.3 with session resumption;
4. Reassess mechanisms like TCP Fast Open or 0-RTT, which involve trade-offs in security and compatibility.

Two independent curl processes typically do not share a connection. To demonstrate reuse, the same client instance should send multiple consecutive requests, and reuse must be verified through traces or num_connects, rather than by removing Connection: close alone.

HTTP/2 enables multiplexing of streams over a single connection, but TCP packet loss can still block subsequent bytes on that connection. HTTP/3 moves stream reliability into QUIC, eliminating cross-stream transport-layer head-of-line (HOL) blocking. However, it remains subject to congestion control, path loss, stream ordering, and endpoint processing limitations.

TFO and Early Data

TCP Fast Open allows data to be carried in the SYN packet under conditions like the presence of a cookie; the benefits depend heavily on initial connection success and intermediate device compatibility. TLS 1.3 0-RTT sends early data that is replayable (only suitable for requests explicitly designed to allow replay) and requires server-side protection. Data carried in a TFO SYN has replay concerns too. An HTTP method name or an informal claim that an endpoint is read-only does not establish harmless repeated execution.

Neither TFO nor early data is "just turn the switch and save one RTT." Success rates, fallback behavior, replay boundaries, and intermediate network behavior must all be measured and understood.

MTU and MSS

"The small response works, the large one fails" often points to PMTUD/PLPMTUD or tunnel MTU. When diagnosing, distinguish between:

• Link MTU;
• Path MTU;
• TCP MSS;
• IPv4 fragmentation and IPv6 source fragmentation;
• Whether ICMP "Packet Too Big" or "Fragmentation Needed" messages are returned;
• The impact of NIC offload on packet capture behavior on the local host.

MSS clamping can alleviate TCP issues at certain tunnel boundaries, but it does not fix UDP or QUIC problems and should not replace the thorough investigation of path MTU and ICMP behavior.

iperf3 Measurement Boundaries

Run iperf3 only between endpoints you control or have permission to test. Install the tools and replace 192.0.2.10 with the actual server address. The default control port is TCP 5201; these examples measure client-sent and reverse-direction traffic separately:

# server
iperf3 --server

# client: one TCP test
iperf3 --client 192.0.2.10 --time 20

# reverse direction
iperf3 --client 192.0.2.10 --reverse --time 20

It measures the generated traffic between test hosts under specific parameters, not the actual application goodput. Running multiple parallel streams may mask issues in a single stream or saturate shared network links. UDP tests must clearly specify the target bitrate and simultaneously explain loss, jitter, and received rate. Do not assume that "running TCP and UDP simultaneously" is the default or standard bandwidth test.

Checks Before a Change

• [ ] Confirmed that the bottleneck is in transport, not in application data supply or downstream components;
• [ ] Have baseline metrics for RTT, throughput, packet loss, retransmissions, queue depth, and memory usage prior to change;
• [ ] Modify only one primary variable at a time;
• [ ] Understand the parameter's scope, whether it applies to socket, route, namespace, or host level;
• [ ] Test under the target kernel and actual RTT/loss distribution conditions;
• [ ] Implement gradual rollout, alerting, and rollback mechanisms;
• [ ] Simultaneously evaluate short flows, long flows, and coexisting traffic.

Lesson Summary

Transmission performance depends on the joint effect of several factors: BDP (bandwidth-delay product), window size, application data supply rate, packet loss, pacing, and queue behavior. Kernel parameters are not better simply because they are larger; congestion control algorithms have no universal champion that operates independently of network conditions. The next lesson shifts the focus to HTTP, caching, CDN, and service capacity.

Specification Entry Points

• RFC 7323: TCP Extensions for High Performance;
• RFC 5681: TCP Congestion Control;
• RFC 7413: TCP Fast Open;
• RFC 8985: RACK-TLP Loss Detection;
• RFC 9002: QUIC Loss Detection and Congestion Control;
• RFC 7567: Active Queue Management;
• iperf3: Options and measurement modes.
• RFC 8446: TLS 1.3 handshake and early data.
• RFC 3168: ECN.