14.2 Transmission Performance: RTT, Window, Packet Loss, Queuing, and Connection Multiplexing

The network is connected, yet cross-ocean transfers still fall short of target throughput. Ah Hua insists we first identify the performance bottlenecks using evidence from RTT, window sizes, and queue behavior.

Increasing socket buffer sizes, switching congestion control algorithms, or adjusting the initial window may improve performance, yet they could also increase queuing, bursts, and memory consumption. Before making any adjustments, determine whether the connection is constrained by RTT, congestion window, receive window, application data supply, or bandwidth limitations.

The Four Sources of Delay

The time it takes for a packet to traverse a path can be roughly broken down into:

propagation + serialization + queueing + processing

• propagation: the signal travels through the medium, dependent on distance and path length;
• serialization: the time required to place each bit of the packet onto the link;
• queueing: waiting for link or device processing, which can grow sharply when load approaches capacity;
• processing: handling by protocols, encryption, forwarding, and application logic.

Transmitting a 1500-byte frame over a 10 Mbit/s link, serialization alone requires approximately:

1500 × 8 / 10,000,000 = 1.2 ms

A full line-rate calculation would also account for link-layer overhead; this example only illustrates the order of magnitude.

Bandwidth-delay product

The bandwidth-delay product (BDP) is the product of the bottleneck bandwidth and the round-trip time (RTT), approximating the amount of data that would fill the path:

BDP = bandwidth × RTT

For 1 Gbit/s bandwidth and an 80 ms RTT:

1,000,000,000 bit/s × 0.08 s = 80,000,000 bit
80,000,000 / 8 = 10,000,000 byte ≈ 9.54 MiB

If the sender's congestion window or receiver's advertised window remains significantly smaller than the BDP, a single connection may never fully utilize the link. However, setting the buffer size to the BDP does not guarantee achieving the full bandwidth. Application data generation rates, congestion control algorithms, packet loss, pacing mechanisms, receiver processing capacity, and intermediate network devices can all act as limiting factors.

Three Different Windows

• congestion window, cwnd: an estimate of network capacity maintained by the sender for congestion control;
• advertised receive window, rwnd: the amount of receive buffer space advertised by the receiver;
• application buffer: the buffer configuration between the application and the kernel for sending and receiving data.

The amount of data in transit on the sender side is roughly bounded by min(cwnd, rwnd), but actual transmission is also constrained by pacing, application write operations, and protocol implementation details. The socket buffer and the wire-visible window are not simply equivalent; Linux additionally accounts for management overhead and performs autotuning.

TCP Window Scale is negotiated during the handshake. After the connection is established, the scaling capability can be modified, but such changes cannot retroactively alter the previously negotiated window scaling for existing connections.

Observe Linux TCP States

ss -tin dst 203.0.113.10

The output fields vary by kernel version. Common information includes RTT estimation, cwnd, MSS, retransmissions, pacing rate, delivery rate, and congestion control algorithms. Before interpreting the data, ensure that:

uname -r
ss -V
sysctl net.ipv4.tcp_congestion_control
sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem

These commands are used to retrieve TCP state information. Do not copy the example values directly to all hosts: behavior can be influenced by container namespaces, cgroups, kernel version, memory pressure, and default settings of the operating system distribution.

Why a Single Connection Fails to Reach Full Throughput

Check the following evidence:

1. Is the application continuously producing data, or is it constrained by disk I/O, CPU load, locks, or upstream throttling?
2. Is the number of cwnd × MSS / RTT events on the order of the observed throughput?
3. Is the receive window (rwnd) becoming the bottleneck? Is the receiving application reading data promptly?
4. Are there retransmissions, SACKs, RTO events, or sustained packet loss?
5. How does the pacing or delivery rate compare to the interface’s capacity?
6. Can multi-stream aggregation improve throughput? If so, the focus should shift from single-stream control or window sizing to broader flow management.
7. Are there policy-based rate limits, traffic shaping, VPNs, tunnels, or cloud service quotas in place?

The formula provides only an upper-bound approximation. ACK strategies, offloading, application bursts, and measurement window duration all introduce short-term fluctuations in observed values.

Packet Loss Is Not the Only Congestion Signal

Traditional loss-based algorithms rely on packet loss as the primary congestion indicator; ECN can signal congestion without actually dropping packets; BBR-style algorithms model congestion based on delivery rate and RTT. The choice of algorithm impacts throughput, queue behavior, and fairness with other flows, and depends heavily on the specific kernel implementation.

You cannot universally assert that "BBR is always faster" or "CUBIC is always fair." A proper evaluation must cover at least the following scenarios:

• Short-lived versus long-lived flows;
• Varying RTT, bandwidth, and loss conditions;
• Fairness when coexisting with existing traffic;
• Interaction with bottleneck qdisc and AQM (Active Queue Management);
• CPU, memory usage, and retransmission behavior;
• The specific version of the algorithm implemented in the target kernel.

Production deployment requires gradual rollout, continuous monitoring, and rollback capability. Direct global changes to sysctl should be avoided.

Pacing, Queue, and Bufferbloat

A sudden burst of the entire cwnd creates a short queue at the bottleneck. Pacing distributes transmissions over a period of time, reducing burstiness. Fair queuing and AQM mechanisms (such as FQ-CoDel) can isolate flows and take action (like packet dropping or ECN marking) before the queue becomes excessively long.

Increasing queue or buffer size can absorb short bursts, but it may also lead to bufferbloat: throughput may appear normal, yet interaction latency rises significantly under load. Testing should simultaneously monitor both throughput and loaded latency, rather than relying solely on "the download is full."

Connection Establishment and Reuse

TCP and TLS

Establishing a new TCP connection requires a handshake. TLS 1.3 full handshakes typically send application data earlier than TLS 1.2 full handshakes, and session resumption can further reduce handshake overhead. The actual round-trip time (RTT) depends on version, resumption, certificate validation, client authentication, and the transport protocol used.

Optimization priorities are generally:

1. Avoid unnecessary new connections;
2. Limit connection reuse to bounded, predictable patterns;
3. Correctly configure TLS 1.3 with session resumption;
4. Reassess mechanisms like TCP Fast Open or 0-RTT, which involve trade-offs in security and compatibility.

Two independent curl processes typically do not share a connection. To demonstrate reuse, the same client instance should send multiple consecutive requests, and reuse must be verified through traces or num_connects, rather than by removing Connection: close alone.

HTTP/2 enables multiplexing of streams over a single connection, but TCP packet loss can still block subsequent bytes on that connection. HTTP/3 moves stream reliability into QUIC, eliminating cross-stream transport-layer head-of-line (HOL) blocking. However, it remains subject to congestion control, path loss, stream ordering, and endpoint processing limitations.

TFO and Early Data

TCP Fast Open allows data to be carried in the SYN packet under conditions like the presence of a cookie; the benefits depend heavily on initial connection success and intermediate device compatibility. TLS 1.3 0-RTT sends early data that is replayable (only suitable for requests explicitly designed to allow replay) and requires server-side protection.

Neither TFO nor early data is "just turn the switch and save one RTT." Success rates, fallback behavior, replay boundaries, and intermediate network behavior must all be measured and understood.

MTU and MSS

"The small response works, the large one fails" often points to PMTUD/PLPMTUD or tunnel MTU. When diagnosing, distinguish between:

• Link MTU;
• Path MTU;
• TCP MSS;
• IPv4 fragmentation and IPv6 source fragmentation;
• Whether ICMP "Packet Too Big" or "Fragmentation Needed" messages are returned;
• The impact of NIC offload on packet capture behavior on the local host.

MSS clamping can alleviate TCP issues at certain tunnel boundaries, but it does not fix UDP or QUIC problems and should not replace the thorough investigation of path MTU and ICMP behavior.

iperf3 Measurement Boundaries

iperf3 must run on both ends that you control or have explicit authorization to use:

# server
iperf3 --server

# client: one TCP test
iperf3 --client 192.0.2.10 --time 20

# reverse direction
iperf3 --client 192.0.2.10 --reverse --time 20

It measures the generated traffic between test hosts under specific parameters, not the actual application goodput. Running multiple parallel streams may mask issues in a single stream or saturate shared network links. UDP tests must clearly specify the target bitrate and simultaneously explain loss, jitter, and received rate. Do not assume that "running TCP and UDP simultaneously" is the default or standard bandwidth test.

Change Before Checklist

• [ ] Confirmed that the bottleneck is in transport, not in application data supply or downstream components;
• [ ] Have baseline metrics for RTT, throughput, packet loss, retransmissions, queue depth, and memory usage prior to change;
• [ ] Modify only one primary variable at a time;
• [ ] Understand the parameter's scope, whether it applies to socket, route, namespace, or host level;
• [ ] Test under the target kernel and actual RTT/loss distribution conditions;
• [ ] Implement gradual rollout, alerting, and rollback mechanisms;
• [ ] Simultaneously evaluate short flows, long flows, and coexisting traffic.

Lesson Summary

Transmission performance depends on the joint effect of several factors: BDP (bandwidth-delay product), window size, application data supply rate, packet loss, pacing, and queue behavior. Kernel parameters are not better simply because they are larger, congestion control algorithms have no universal champion that operates independently of network conditions. The next lesson shifts the focus to HTTP, caching, CDN, and service capacity.

Specification Entry Points

• RFC 7323: TCP Extensions for High Performance;
• RFC 5681: TCP Congestion Control;
• RFC 7413: TCP Fast Open;
• RFC 8985: RACK-TLP Loss Detection;
• RFC 9002: QUIC Loss Detection and Congestion Control;
• RFC 9260 is not a TCP tuning specification; when reviewing, be mindful of protocol names and their applicable scope.