3.2 TCP Sequence Numbers, ACKs, and Flow Control

TCP's job is to ensure that the receiving application ultimately sees a continuous stream of hello world. It assigns byte-level sequence numbers, sends ACKs to report contiguous received ranges, buffers out-of-order data, and retransmits after detecting loss. The important boundary is that TCP numbers bytes, not application messages or the packet ordinal shown by a capture tool.

This lesson follows a data stream after a connection has been established. Connection setup and teardown are covered in 3.1 TCP Connection Lifecycle; how the sender adjusts cwnd during network congestion will be discussed in Chapter 10.

1. TCP is a reliable byte stream

TCP provides applications with a bidirectional, in-order byte stream. For a successfully established connection, it performs the following:

• Uses a checksum to detect many, though not all, transmission bit errors;
• Uses sequence numbers to identify data positions;
• Confirms ranges of consecutively received data;
• Retransmits lost data;
• Reassembles data in order before delivering it to the application;
• Limits the amount of data sent based on the receiver's available buffer space.

It does not preserve the message boundaries defined by the application, nor does it guarantee that the remote application has processed the data. In the event of a process crash, a connection reset, or prolonged network interruption, unacknowledged data may still be lost. When applications require assurances such as "the order has been committed to the database," they must implement their own acknowledgment mechanisms and handle retries and idempotency explicitly.

2. What a TCP Header Tells You

Without options, the shortest TCP header is 20 bytes:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |          source port          |       destination port        |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                        sequence number                        |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                    acknowledgment number                     |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | Data  |       |C|E|U|A|P|R|S|F|                               |
 | Offset| Rsrvd |W|C|R|C|S|S|Y|I|            Window             |
 |       |       |R|E|G|K|H|T|N|N|                               |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |           checksum            |         urgent pointer        |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                  options, then application data ...           |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

When troubleshooting data transmission, start by examining these fields:

Field	Purpose
source/destination port	Together with IP addresses, identifies the endpoint
sequence number	Usually the first payload byte’s sequence; with SYN, it is the SYN sequence and data starts one later
acknowledgment number	When ACK is present, indicates the next expected sequence number from the remote side
data offset	Length of the TCP header, in 32-bit words
flags	Status bits including SYN, ACK, FIN, RST, PSH, and ECN/urgent indicators
window	The advertised receive window from this end to the peer, possibly scaled via window scaling
checksum	Error detection value covering the TCP header, payload, and IP pseudo-header
options	Negotiated or feedback information such as MSS, window scale, SACK, and timestamp

The header diagram shows the logical structure, not the full wire capture. Actual packet layouts may be altered by IPv4/IPv6 headers, TCP options, offload mechanisms, or tunneling.

This is the RFC 9293 base header: a 4-bit data offset, 4 reserved bits, and 8 control bits. It does not treat the historical experimental NS bit as a fixed field. Data offsets from 5 to 15 represent headers of 20 to 60 bytes; extensions require their own negotiation rules.

3. Sequence number is a byte coordinate

Suppose a packet capture tool displays the first byte of data in a direction as a relative sequence number 0. If an application writes 11 bytes, TCP might split it into:

segment A: seq=0, len=5   -> bytes [0, 5)   "hello"
segment B: seq=5, len=1   -> bytes [5, 6)   " "
segment C: seq=6, len=5   -> bytes [6, 11)  "world"

Splitting into three segments is just the result of this particular transmission. After retransmissions, changes in MSS, offloading, or congestion state, the boundaries could shift entirely. The receiving application sees only a single 11-byte stream.

The sequence number on the wire is a 32-bit unsigned value, computed modulo 2^32. Packet capture tools often display relative sequence numbers for easier human readability. Both SYN and FIN packets consume one sequence number each, even if they don't carry application data bytes.

MOD = 1 << 32
start, size = MOD - 4, 8
next_wire_sequence = (start + size) % MOD
assert next_wire_sequence == 4
print(start, next_wire_sequence)  # 4294967292 4

Here the wrapped value 4 is newer than 4294967292. TCP sequence comparisons need modular arithmetic and window constraints, not ordinary integer <; the model below therefore excludes wrapping. Capture tools commonly number a relative SYN as 0 and the first data byte as 1. This lesson uses a data-origin zero for convenient arithmetic.

4. ACK Report on the Next Gap

TCP ACKs are typically cumulative. When a receiver sends ack=6, it means all bytes before sequence number 6 have been received, and the next expected byte is 6.

Let the middle space arrive before the preceding word:

Received seq=5, len=1  -> ack=0   # [0,5) still missing
Received seq=6, len=5  -> ack=0   # subsequent data can be buffered, but the gap remains at 0
Received seq=0, len=5  -> ack=11  # gap closed, continuous range advances

Multiple identical ACKs do not mean the receiver has received nothing. It may have buffered subsequent bytes, but the cumulative ACK cannot cross the earliest gap.

An ACK does not indicate that the peer application has read or committed the data. It only confirms the TCP layer’s reception state.

5. Using a Simple Model to Illustrate Out-of-Order Reassembly

The following is a runnable Python 3 example. It simulates non-overlapping, non-wrapping segments to observe cumulative ACK behavior; it is not a real TCP implementation.

class Receiver:
    def __init__(self):
        self.next_sequence = 0
        self.pending = {}
        self.delivered = bytearray()

    def receive(self, sequence, payload):
        if sequence < 0 or not payload:
            raise ValueError("need nonnegative sequence and nonempty payload")
        end = sequence + len(payload)
        if end <= self.next_sequence:
            return self.next_sequence
        if sequence < self.next_sequence:
            raise ValueError("partial overlap is outside this model")
        if sequence in self.pending and self.pending[sequence] == payload:
            return self.next_sequence
        for start, chunk in self.pending.items():
            if sequence < start + len(chunk) and start < end:
                raise ValueError("overlapping ranges are outside this model")

        self.pending[sequence] = payload
        while self.next_sequence in self.pending:
            chunk = self.pending.pop(self.next_sequence)
            self.delivered.extend(chunk)
            self.next_sequence += len(chunk)
        return self.next_sequence

receiver = Receiver()

assert receiver.receive(5, b" ") == 0
assert receiver.receive(6, b"world") == 0
assert receiver.receive(0, b"hello") == 11
assert receiver.delivered == b"hello world"

print(receiver.delivered.decode())
print("cumulative ACK:", receiver.next_sequence)

Run:

python3 tcp-reassembly.py

Expected output:

hello world
cumulative ACK: 11

The model ignores data wholly inside the delivered range, rejects partial overlap and conflicting pending ranges, and does not compare old duplicate content. Real TCP also needs overlap handling, sequence wrap, receive windows, SACK, memory limits, and malformed-input handling. Appending to delivered is only a model operation; it says nothing about actual application reads.

6. SACK tells the sender what it received afterward

With cumulative ACKs alone, the sender knows the earliest gap but lacks a report of later received ranges. Selective Acknowledgment (SACK) allows the receiver to report in the TCP option several received non-contiguous blocks.

For example:

cumulative ACK: 0
SACK block: [5, 11)

The sender can use that report to prioritize [0,5). Permission is directional: a data receiver may send SACK blocks only after receiving SACK-permitted in its peer’s SYN. The option advertises willingness to process SACK feedback; both endpoints do not have to advertise it before either direction can use SACK.

SACK provides feedback without altering the application's perceived in-order byte stream; if the gap at sequence number 0 isn't filled, subsequent bytes are typically not available for the application to read.

A SACK report does not permit the sender to discard its retransmission copy: the receiver can discard previously reported out-of-order data. The sender must retain the data until a cumulative ACK covers it.

7. When to Retransmit

The sender cannot wait indefinitely, nor can it retransmit immediately after every delayed ACK. TCP primarily infers packet loss from two types of evidence:

Retransmission Timeout (RTO). The implementation estimates RTO based on measured round-trip time (RTT) and its variability. If no acknowledgment is received before the timer expires, the sender retransmits and adjusts the timer accordingly. Hardcoding a fixed "network average RTT" fails to adapt to differences between local-area networks and cross-continental links.

ACK/SACK feedback. If multiple ACKs consistently point to the same gap, or if SACK indicates that subsequent data ranges have been received, the sender can detect a likely loss before the RTO expires. This classic mechanism is known as fast retransmit; modern loss detection algorithms also use richer SACK and timing information.

Retransmission means "the sender re-sends a range of sequence numbers," and it does not guarantee that packet boundaries match those of the original transmission. When analyzing retransmissions in packet captures, one must also account for capture loss, offloading, and the specific observation point.

For a first RTT sample of 100 ms and a second of 140 ms, RFC 6298 updates variation using the old SRTT before updating SRTT itself. This calculation uses 1 ms clock granularity and the RFC’s recommended 1-second RTO floor:

srtt = 0.100
rttvar = srtt / 2
sample = 0.140
rttvar = 0.75 * rttvar + 0.25 * abs(srtt - sample)
srtt = 0.875 * srtt + 0.125 * sample
raw_rto = srtt + max(0.001, 4 * rttvar)
rto = max(1.0, raw_rto)
print(f"SRTT={srtt:.4f}, RTTVAR={rttvar:.4f}")
print(f"raw RTO={raw_rto:.3f}, floored RTO={rto:.3f}")
# SRTT=0.1050, RTTVAR=0.0475
# raw RTO=0.295, floored RTO=1.000

This does not predict every system’s ss output: minimum timer policies can differ. Karn’s rule excludes ambiguous RTT samples from retransmitted data unless mechanisms such as timestamps remove that ambiguity. Backoff after an RTO expiry is a separate step.

8. Flow Control Protects the Receiver

The receiver's buffer is finite. If the application does not consume data promptly, the TCP receive buffer fills; the receiver advertises the remaining acceptable sequence space through the receive window (rwnd).

A simplified relationship helps understand the upper bound of data that can be in transit:

allowed flight ≈ min(rwnd, cwnd)

rwnd protects the receiving endpoint, while cwnd protects the network path. The former belongs to flow control, and the latter to congestion control. The effective limit uses the smaller value, but the two windows answer different questions.

Ignoring details such as SACK recovery, with rwnd=64 KiB, cwnd=32 KiB, and 24 KiB already in flight, roughly 8 KiB more new data can enter flight. This is not a promise that each send can write 32 KiB: the local send buffer can queue bytes, and a shrinking window cannot recall bytes already sent.

When the receive window drops to zero, the sender is no longer allowed to send new ordinary data. The implementation retains a persist state and probes for window re-opening to prevent both sides from entering a permanent wait state after a window update is lost. If the application remains inactive for an extended period, the protocol ultimately requires a resource ceiling and connection management strategy.

9. Why Window Scale is Negotiated in SYN

The TCP header's window field is only 16 bits. On high-bandwidth, high-RTT paths, the required in-flight data can exceed 65,535 bytes, making the default window size insufficient. The Window Scale option allows each direction to declare a shift count, which is then applied to shift the window field left by that many bits during subsequent transmissions.

Several edge cases are common sources of error:

• The option can only be negotiated in the SYN segment;
• Each direction independently declares its own scale value, which may differ;
• The window field in the SYN packet itself is not subject to scaling;
• Scaling expands the representable window size without increasing the socket buffer size;
• Packet capture tools may display the scaled window value, or show both the raw value and the scale factor simultaneously.

RFC 7323 limits the maximum shift count to 14. Increasing system buffer sizes does not mean that every connection will immediately consume that memory; actual buffer allocation and autotuning depend on the specific implementation.

Window Scale has a different negotiation rule from SACK-permitted: both endpoints must send Window Scale in their SYNs before scaling is enabled in either direction. Each scale factor remains fixed for the connection.

10. Bandwidth-Delay Product Explains Why Throughput Gets Stuck

The amount of data a path can carry simultaneously is approximately equal to the bandwidth-delay product (BDP):

BDP = bottleneck bandwidth × RTT

If the bottleneck is 100 Mbit/s and the RTT is 80 ms, the path's BDP is roughly 1 MB. If a single connection can only maintain significantly less data in transit (regardless of whether packets are lost) the path remains underutilized.

bandwidth_bits_s = 100_000_000
rtt_s = 0.080
bdp_bytes = bandwidth_bits_s * rtt_s / 8
window_bytes = 65_535
window_limited_bits_s = window_bytes * 8 / rtt_s
print(f"BDP: {bdp_bytes:.0f} bytes")
print(f"Window ceiling: {window_limited_bits_s / 1_000_000:.4f} Mbit/s")
print("Scaled window:", 8192 << 7, "bytes")
# BDP: 1000000 bytes
# Window ceiling: 6.5535 Mbit/s
# Scaled window: 1048576 bytes

Here MB is decimal: 1,000,000 bytes. One MiB is 1,048,576 bytes. The 6.5535 Mbit/s value is an ideal window/RTT ceiling; protocol overhead and other limits can reduce application throughput further.

This calculation provides only an order-of-magnitude estimate, not a guaranteed throughput. Protocol overhead, application data generation rate, cwnd, rwnd, queuing, packet loss, and receiver processing all influence the actual result. The correct approach is to measure the window size and RTT first, then determine whether buffer limitations are present.

11. MSS, MTU, and Segment Boundaries

The Maximum Segment Size (MSS) option indicates the largest TCP payload segment size that the sender is willing to receive. It is typically derived from the interface or path MTU, but does not include the IP or TCP headers.

An MTU-derived MSS advertisement subtracts fixed IP/TCP headers. Actual segments must also allow for IP/TCP options and obey the peer MSS and path MTU. An IPv4 MTU of 1500 can therefore yield an advertised MSS of 1460 while 12 bytes of TCP options leave only 1448 data bytes. IPv6, tunnels, and smaller path MTUs alter the budget as well; advertised MSS and actual segment payload are distinct values.

TCP segmentation offload (TSO/GSO) may cause the sender to generate logical segments larger than the path MSS. On the receiving end, GRO (Generic Receive Offload) may combine multiple packets before handing them to the protocol stack. When analyzing wire packet sizes, it's essential to capture at an appropriate point in the network path, or to account for the effects of offload.

12. Nagle, Delayed ACK, and Small Writes

The Nagle algorithm attempts to reduce the number of small segments by buffering new small writes when a connection already has unacknowledged data. Delayed ACK allows the receiver to temporarily wait, potentially combining ACKs or piggybacking them on data sent in the reverse direction.

These strategies can interact and increase latency in certain request/response patterns, but you cannot universally enable TCP_NODELAY simply upon observing small packets:

• Low-latency interactive protocols may prefer more packets for faster transmission;
• Bulk transfers typically prioritize throughput and packet efficiency;
• Applications that manage their own buffers, TLS records, or runtime writers can shift write boundaries;
• Specific behaviors of delayed ACK and Nagle are implementation-specific decisions.

First, capture packets to determine where the waiting occurs in the stack, then adjust socket options accordingly. A single send may be small, but it does not prove that the wire carries a packet of the same size.

13. What a Checksum Can Detect

A TCP checksum covers the TCP header, the payload, and a pseudo-header constructed from IP address and protocol fields. It is designed to detect random transmission errors and can also identify cases where packets are delivered to the wrong endpoint.

It does not provide cryptographic integrity. An attacker who can modify a packet can recalculate the checksum to evade detection. Protection against tampering and identity authentication is handled by security protocols such as TLS and IPsec.

Seeing "checksum incorrect" when capturing packets on the sender's host does not necessarily mean the packet is corrupted. Hardware offload of checksum computation may fill in the final checksum value after the capture point, so the diagnosis must take into account both the capture location and the offload state.

14. TCP's Head-of-Line Blocking

The receiver has received subsequent segments but must still wait for the earliest gap to be filled before delivering a continuous stream to the application. This is known as transport-layer head-of-line blocking.

HTTP/2 enables multiplexing of many streams within a single TCP connection, yet if one TCP sequence number range is lost, all subsequent bytes across all streams are blocked. QUIC isolates reliability to individual streams specifically to reduce the scope of this blocking. Chapter 11 examines the details.

TCP has no knowledge of which bytes belong to an image, an RPC, or a heartbeat. No matter how sophisticated the application-level framing, it cannot overcome the gaps in the underlying byte stream.

15. Using ss and tcpdump to Gather Evidence

On Linux, you can first inspect the internal statistics of sockets:

ss -tin

The fields in ss vary across kernel and iproute2 versions. Common information includes RTT, RTO, congestion control algorithm, MSS, window scale, retransmission, and delivery rate. Do not treat a single line of output from one version as a stable, cross-platform API.

A local experiment can be performed with loopback packet capture:

Port 8000 below is an example. Chapter 2 binds port 0 by default; for this exercise, change it to an available local port 8000 and start capture before running the program. If another port is chosen, update the filter too. The short connection can finish before a diagnostic tool starts.

sudo tcpdump -i lo -nn -tttt 'tcp port 8000'

When observing, align the three timelines:

1. When the application calls send, recv, and shutdown;
2. How the kernel socket state and queue evolve over time;
3. What actual SYN, ACK, data, retransmissions, FIN, or RST packets are captured at the capture point.

A capture observes one point on the path. It cannot show application processing after the server kernel receives a segment, and the TCP flows on either side of a proxy must be examined separately.

16. Common Misconceptions

"ACK means the packet was processed by the application." An ACK only confirms the TCP sequence number range, it does not indicate that the business transaction has been completed.

"One TCP send corresponds to one TCP receive." TCP does not retain application-level write boundaries; it only manages byte streams.

"Larger windows mean faster performance." Effective throughput is still constrained by cwnd, round-trip time (RTT), packet loss, application processing speed, and the bottleneck bandwidth.

"Duplicate ACKs always indicate network packet loss." Reordering, duplicate segments, and capture artifacts can produce similar observations. These must be evaluated in conjunction with SACK and packet capture from both ends.

"TCP checksum prevents tampering." It provides error detection, not authentication or cryptographic integrity.

"The PSH flag defines a message boundary." PSH signals to the receiver that data should be processed promptly, but it does not establish a universal message boundary for the application.

17. Exercises and Verification

Exercise One: Change Arrival Order

Modify the previous Receiver example so that world, a space, and hello arrive in sequence. Verification conditions:

• The first two ACK responses returned are both 0;
• The final response returns 11;
• The delivered state remains hello world.

Exercise Two: Compare Application and Packet Capture

Run the socket example from Chapter 2 on your local machine, while simultaneously using tcpdump to capture loopback traffic. Verification conditions:

• Identify the three segments involved in the handshake;
• Distinguish between the application's two sendall calls and the actual segment boundaries;
• Locate a FIN or RST packet and explain how the program closes the connection using that signal.

If packet capture requires administrative privileges and cannot be executed, save the application logs and note the boundary points for verification, do not fabricate packet output.

18. From Byte Stream to Application Protocol

Out-of-order arrival now has a clear interpretation: later bytes can be buffered while the cumulative ACK stays at the gap. Filling that gap advances the contiguous range. TCP acknowledgment describes reception by the transport stack; business completion still needs an application response.

Next up is HTTP and Web Servers, where continuous bytes finally regain their meaning as request lines, headers, and bodies. If you want to track how the window shrinks during traffic congestion, head directly to Chapter 10: TCP Congestion Control.

References

• RFC 9293: Transmission Control Protocol
• RFC 7323: TCP Extensions for High Performance
• RFC 6298: Computing TCP's Retransmission Timer
• RFC 2018: TCP Selective Acknowledgment Options