8.3 MTU, IP Fragmentation, and Path MTU Discovery
A series of large packets vanish on a certain stretch of the route, while smaller ones pass through unimpeded. Ah Hua checks the maximum packet size allowed at each hop along the way.
The MTU is the maximum size of a network-layer packet that a single link or interface can carry. The Path MTU (PMTU) is the minimum of the MTUs across all hops between source and destination. These values are not equivalent to application message sizes, nor to the TCP send() chunk size that TCP uses in each transmission.
1. Separate frame, IP packet, TCP segment, and application write
application bytes
↓ stream/message framing
TCP segment or UDP datagram
↓ IP header
IP packet
↓ link header/trailer
Ethernet/Wi-Fi/tunnel frameEthernet typically has an IP MTU of 1500, but jumbo frames, PPPoE, VPNs/overlay networks, and cloud encapsulation can alter the effective MTU. You cannot infer that a host's IP MTU is 2304 simply because the Wi-Fi frame body is capped at 2304: the 802.11 MAC frame overhead, LLC/SNAP headers, aggregation, and bridge translation operate at a different boundary than the IP interface MTU.
TCP can split a byte stream into segments, and segmentation offload can cause packet capture on a host side to reveal logical packets larger than the wire MTU. When observing traffic, always note the capture point and the NIC offload state.
2. IPv4 Fragmentation Depends on DF and Packet Size
When an IPv4 router receives a packet larger than the outgoing link's MTU:
- If DF (Don't Fragment) is 0, the router may fragment the packet;
- If DF is 1, the router cannot fragment it and instead discards the packet, sending back an ICMP Destination Unreachable message (Type 3, Code 4) indicating "Fragmentation Needed," which includes the next-hop MTU information.
Fragmentation occurs only at the final destination, where the fragments are reassembled. Intermediate routers do not reassemble packets along the path. If any single fragment is lost, the original datagram cannot be reconstructed. Additionally, middleboxes, NAT devices, and firewalls often handle fragments in unpredictable or unreliable ways. For these reasons, modern transport protocols typically avoid relying on in-path IPv4 fragmentation.
3. IPv6 Routers Do Not Perform In-Path Fragmentation
When an IPv6 router encounters a packet that is oversized, it discards the packet and returns an ICMPv6 "Packet Too Big" message (Type 2). Unlike IPv4, where a router with DF=0 can fragment the packet on the fly, IPv6 routers do not perform fragmentation. Only the source host can use the Fragment Extension Header to split the packet into fragments.
IPv6 mandates that each link support a minimum MTU of 1280 bytes. If a link cannot natively carry such packets, the link layer must provide fragmentation and reassembly beneath the IPv6 layer. The requirement for a 1280-byte minimum MTU does not mean that all IPv6 paths will have a PMTU strictly limited to 1280 bytes.
4. Classic PMTUD Relies on ICMP Feedback
IPv4 PMTUD typically sends DF (Don't Fragment) packets and reduces its PMTU estimate based on ICMP Fragmentation Needed messages. In IPv6, PMTUD updates its estimate using ICMPv6 Packet Too Big messages. The PMTU estimate must have a lifetime and be capable of being re-raised after path changes, otherwise a single outdated bottleneck can permanently constrain the packet size across a connection.
Equal-cost multipath routing, route changes, and tunneling can result in different PMTU values across different flows or times. Therefore, PMTU should not be treated as a permanent constant between two hosts.
ICMP messages must undergo validity checks, rate limiting, and policy filtering. However, indiscriminately dropping all ICMP or ICMPv6 messages can undermine PMTUD functionality.
5. MTU black hole is a potential cause of "small packets work, large packets fail"
Typical failure chain:
- The source sends an oversized, non-fragmentable packet;
- The router or tunnel endpoint drops the packet and generates an ICMP Too Big / Fragmentation Needed message;
- The ICMP message is dropped by an intermediate policy, or the sender fails to associate it with the corresponding flow;
- The sender does not reduce packet size, and retransmissions continue to fail.
Symptoms may include a successful TCP handshake and successful small requests, but timeouts when accessing certificate chains, large responses, or uploads. However, the same symptoms can also stem from application-level timeouts, proxy body size limits, packet loss, or flow-control bugs, so the observation that "large files fail" alone is not sufficient to conclude that an MTU black hole is present.
Evidence should include:
- The interface or tunnel MTU and routing configuration;
- Packet capture data showing packet size, DF flag, IPv6 usage, and retransmissions;
- Whether ICMP "Too Big" messages are generated and received;
- Results from probes using different packet sizes;
- Whether the issue resolves consistently when the MTU or MSS is reduced.
6. PLPMTUD Does Not Rely on ICMP as the Sole Signal
Packetization Layer PMTUD (PLPMTUD) involves the transport or application layer sending probes of varying sizes and inferring the supported packet size based on acknowledgment behavior or packet loss, without depending entirely on ICMP responses.
However, packet loss may simply indicate network congestion, not necessarily an MTU limitation. The algorithm must perform state tracking, confirmation checks, fallback base size selection, and black-hole detection, going beyond a simple binary search of packet size. TCP provides guidance for PLPMTUD, while datagram-based transport and application layers can implement Datagram PLPMTUD as specified in RFC 8899. QUIC also has its own requirements for datagram size validation and path checks.
7. TCP MSS and MSS Clamping
The TCP MSS option declares in the SYN packet the receiver's desired maximum TCP payload size. The typical IPv4 default upper bound without options is IP MTU - 20-byte IPv4 header - 20-byte TCP header, but the presence of IPv4 options, IPv6 extension headers, TCP options, and tunnel overhead can alter this budget. Therefore, the statement "MSS is always equal to MTU minus 40" is not a universal rule.
MSS clamping reduces the advertised MSS when a packet passes through a router or firewall during the SYN phase, and can serve as a mitigation technique for tunnel or PMTUD black holes. It only affects TCP traffic and does not resolve issues with UDP, ICMP, or other IP protocols. MSS clamping should not be used as a substitute for properly configured link MTU settings and ICMP policies. Arbitrarily setting the MSS clamp to an extremely small value increases packet and CPU overhead.
8. Tunnel Overhead: Must Make an Explicit Budget
After an inner packet enters a tunnel, additional outer IP, UDP/GRE/ESP, and tunnel-specific headers or tags are added. The exact overhead depends on whether IPv4 or IPv6 is used, the encryption mode, supported options, and the specific implementation, so general assumptions like "all VPNs add 50–70 bytes" should not be used as a basis for configuration.
A simple budget calculator can force configurations to explicitly list the overhead at each layer:
def inner_mtu(outer_mtu: int, overheads: list[tuple[str, int]]) -> int:
if outer_mtu <= 0:
raise ValueError("outer_mtu must be positive")
total = 0
for name, size in overheads:
if size < 0:
raise ValueError(f"negative overhead for {name}")
total += size
result = outer_mtu - total
if result < 1280:
print("warning: result is below the IPv6 minimum link MTU")
return result
# The value must come from what you actually use tunnel format, The following is just an algorithm example.
headers = [("outer IP", 20), ("UDP", 8), ("tunnel data header/tag", 32)]
assert inner_mtu(1500, headers) == 1440However, the calculator won't automatically account for NIC offload, nested tunnels, or provider network limits. The resulting overhead values must be validated through end-to-end observation.
9. Linux Diagnostics Tools
ip link show
ip route get 203.0.113.10
tracepath 203.0.113.10
# IPv4: Not allowed fragment, ICMP payload 1472 + IPv4 20 + ICMP 8 = 1500
ping -4 -M do -s 1472 203.0.113.10Command options and privileges vary by operating system, and ping success does not guarantee that TCP/UDP uses the same path or policy. When IPv4 headers include options, the 1472 calculation must also be adjusted. Experiments are conducted only on endpoints the user is authorized to test, and are combined with packet capture and counter data.
10. Acceptance Questions
- What is the difference between IPv4 DF=0 and DF=1 when encountering a small outgoing MTU?
- Why do IPv6 routers not perform in-path fragmentation?
- How do classic PMTUD and PLPMTUD differ in their feedback source?
- Why can MSS clamping not resolve the UDP MTU black hole problem?
- Provide at least three pieces of evidence that support the conclusion of an MTU black hole.