Software Systems Atlas

English

中文

English

中文

Theme

VOL 04 / CH 08 / LESSON 03

8.3 MTU, IP Fragmentation, and Path MTU Discovery ​

The MTU is the maximum size of a network-layer packet that a single link or interface can carry. Path MTU (PMTU) is the minimum MTU along the source-to-destination path; the reverse path can differ and needs separate assessment. These values are not equivalent to application message sizes, nor to a required chunk size for each application send() call.

1. Separate frame, IP packet, TCP segment, and application write ​

text
application bytes
    ↓ stream/message framing
TCP segment or UDP datagram
    ↓ IP header
IP packet
    ↓ link header/trailer
Ethernet/Wi-Fi/tunnel frame

Ethernet typically has an IP MTU of 1500, but jumbo frames, PPPoE, VPNs/overlay networks, and cloud encapsulation can alter the effective MTU. You cannot infer that a host's IP MTU is 2304 simply from a quoted Wi-Fi frame-body limit of 2304: the 802.11 MAC frame overhead, LLC/SNAP headers, aggregation, and bridge translation operate at a different boundary than the IP interface MTU.

TCP can split a byte stream into segments, and segmentation offload can cause packet capture on a host side to reveal logical packets larger than the wire MTU. When observing traffic, always note the capture point and the NIC offload state.

2. IPv4 Fragmentation Depends on DF and Packet Size ​

When an IPv4 router receives a packet larger than the outgoing link's MTU:

  • If DF (Don't Fragment) is 0, the router may fragment the packet;
  • If DF is 1, the router cannot fragment it and instead discards the packet, sending back an ICMP Destination Unreachable message (Type 3, Code 4) indicating "Fragmentation Needed," which includes the next-hop MTU information.

Fragments are reassembled at the original IP datagram’s destination, not at ordinary forwarding routers along the way. A tunnel’s outer-IP destination may itself be a router; distinguish the inner and outer layers. If any single fragment is lost, the original datagram cannot be reconstructed. Additionally, middleboxes, NAT devices, and firewalls often handle fragments in unpredictable or unreliable ways. Modern transports therefore generally avoid relying on in-path IPv4 fragmentation.

For example, fragment a 4000-byte IPv4 datagram with no IP options onto an MTU-1500 link. Its original payload is 3980 bytes. Nonfinal fragment payloads must be multiples of eight bytes, and Fragment Offset uses eight-byte units:

FragmentIP payloadTotal with 20-byte headerOffset / MF
1148015000 / 1
214801500185 / 1
310201040370 / 0

All three share the Identification value; MF=0 marks the last fragment. Offset 185 refers to byte 1480 of the original payload, not byte 185.

3. IPv6 Routers Do Not Perform In-Path Fragmentation ​

When an IPv6 router encounters a packet that is oversized, it discards the packet and returns an ICMPv6 "Packet Too Big" message (Type 2). Unlike IPv4, where a router can fragment a packet with DF=0, IPv6 routers do not perform fragmentation. Only the source host can use the Fragment Extension Header to split the packet into fragments.

IPv6 mandates that each link support a minimum MTU of 1280 bytes. If a link cannot natively carry such packets, the link layer must provide fragmentation and reassembly beneath the IPv6 layer. The requirement for a 1280-byte minimum MTU does not mean that all IPv6 paths will have a PMTU strictly limited to 1280 bytes.

4. Classic PMTUD Relies on ICMP Feedback ​

IPv4 PMTUD typically sends DF (Don't Fragment) packets and reduces its PMTU estimate based on ICMP Fragmentation Needed messages. In IPv6, PMTUD updates its estimate using ICMPv6 Packet Too Big messages. The PMTU estimate must have a lifetime and be capable of being re-raised after path changes, otherwise a single outdated bottleneck can permanently constrain the packet size across a connection.

Equal-cost multipath routing, route changes, and tunneling can result in different PMTU values across different flows or times. Therefore, PMTU should not be treated as a permanent constant between two hosts.

ICMP messages must undergo validity checks, rate limiting, and policy filtering. However, indiscriminately dropping all ICMP or ICMPv6 messages can undermine PMTUD functionality.

5. MTU black hole is a potential cause of "small packets work, large packets fail" ​

Typical failure chain:

  1. The source sends an oversized, non-fragmentable packet;
  2. The router or tunnel endpoint drops the packet and generates an ICMP Too Big / Fragmentation Needed message;
  3. The ICMP message is dropped by an intermediate policy, or the sender fails to associate it with the corresponding flow;
  4. The sender does not reduce packet size, and retransmissions continue to fail.

Symptoms may include a successful TCP handshake and successful small requests, but timeouts when accessing certificate chains, large responses, or uploads. However, the same symptoms can also stem from application-level timeouts, proxy body size limits, packet loss, or flow-control bugs, so the observation that "large files fail" alone is not sufficient to conclude that an MTU black hole is present.

Evidence should include:

  • The interface or tunnel MTU and routing configuration;
  • Packet capture data showing packet size, DF flag, IPv6 usage, and retransmissions;
  • Whether ICMP "Too Big" messages are generated and received;
  • Results from probes using different packet sizes;
  • Whether the issue resolves consistently when the MTU or MSS is reduced.

6. PLPMTUD Does Not Rely on ICMP as the Sole Signal ​

Packetization Layer PMTUD (PLPMTUD) involves the transport or application layer sending probes of varying sizes and inferring the supported packet size based on acknowledgment behavior or packet loss, without depending entirely on ICMP responses.

However, packet loss may simply indicate network congestion, not necessarily an MTU limitation. The algorithm must perform state tracking, confirmation checks, fallback base size selection, and black-hole detection, going beyond a simple binary search of packet size. RFC 4821 gives PLPMTUD guidance applicable to TCP, while datagram-based transport and application layers can implement Datagram PLPMTUD as specified in RFC 8899. QUIC also has its own requirements for datagram size validation and path checks.

7. TCP MSS and MSS Clamping ​

The MSS option in SYN (including SYN-ACK) advertises how much TCP data that endpoint can receive, excluding IP/TCP headers. From the effective MTU, subtract fixed headers: 20+20 bytes for IPv4 or 40+20 for IPv6. RFC 6691 says not to pre-deduct possible options from the advertised MSS; the sender reduces actual data length for its options or extension headers.

For IPv4 MTU 1500, MSS can be 1460. A data packet carrying 12 bytes of TCP options then has room for 1448 data bytes. Peer MSS and discovered PMTU both constrain segmentation; MSS does not prescribe application send() sizes.

MSS clamping reduces the advertised MSS when a packet passes through a router or firewall during the SYN phase, and can serve as a mitigation technique for tunnel or PMTUD black holes. It only affects TCP traffic and does not resolve issues with UDP, ICMP, or other IP protocols. MSS clamping should not be used as a substitute for properly configured link MTU settings and ICMP policies. Arbitrarily setting the MSS clamp to an extremely small value increases packet and CPU overhead.

8. Tunnel Overhead: Must Make an Explicit Budget ​

After an inner packet enters a tunnel, additional outer IP, UDP/GRE/ESP, and tunnel-specific headers or tags are added. The exact overhead depends on whether IPv4 or IPv6 is used, the encryption mode, supported options, and the specific implementation, so general assumptions like "all VPNs add 50–70 bytes" should not be used as a basis for configuration.

A simple budget calculator can force configurations to explicitly list the overhead at each layer:

python
def inner_mtu(outer_mtu: int, overheads: list[tuple[str, int]]) -> int:
    if outer_mtu <= 0:
        raise ValueError("outer_mtu must be positive")
    total = 0
    for name, size in overheads:
        if size < 0:
            raise ValueError(f"negative overhead for {name}")
        total += size
    result = outer_mtu - total
    if result <= 0:
        raise ValueError("overhead leaves no room for an inner packet")
    if result < 1280:
        print("warning: result is below the IPv6 minimum link MTU")
    return result

# Use sizes from the actual tunnel format; these are illustrative values.
headers = [("outer IP", 20), ("UDP", 8), ("tunnel data header/tag", 32)]
assert inner_mtu(1500, headers) == 1440

Here outer_mtu starts at the outer IP header; do not subtract the already-excluded outer Ethernet header again. Specify the carried payload too: an Ethernet tunnel must budget for the inner Ethernet header. Add variable padding, nested tunnels, and provider limits from the actual format, using maximum overhead when necessary. Offload can change host capture appearance but does not remove wire encapsulation bytes.

A result below 1280 is a planning warning, not a usable IPv6 link MTU configuration by itself. It needs adaptation below IPv6 or changes to encapsulation or the outer path. Validate the final budget end to end.

9. Linux Diagnostics Tools ​

bash
ip link show
ip route get 203.0.113.10
tracepath 203.0.113.10

# IPv4: forbid fragmentation; ICMP payload 1472 + IPv4 20 + ICMP 8 = 1500
ping -4 -M do -c 3 -W 2 -s 1472 203.0.113.10

Replace 203.0.113.10 with an actual endpoint you control. Here ping -M do respects the locally known PMTU, so “message too long” can be a local rejection before the probe reaches any remote host. Command options and privileges vary by operating system, and ping success does not guarantee that TCP/UDP uses the same path or policy. When IPv4 headers include options, the 1472 calculation must also be adjusted. Experiments are conducted only on endpoints the user is authorized to test, and are combined with packet capture and counter data.

10. Acceptance Questions ​

  1. What is the difference between IPv4 DF=0 and DF=1 when encountering a small outgoing MTU?
  2. Why do IPv6 routers not perform in-path fragmentation?
  3. How do classic PMTUD and PLPMTUD differ in their feedback source?
  4. Why can MSS clamping not resolve the UDP MTU black hole problem?
  5. Provide at least three pieces of evidence that support the conclusion of an MTU black hole.

References ​

Built with VitePress | Software Systems Atlas

Copyright 2026 Steven Yang