12.2 Packet Capture and Protocol Analysis: Choose Your Observation Point, Then Interpret the Data

Packet capture can answer the question, "What packets did this observation point actually see?" But it cannot automatically reveal what happened across the entire end-to-end flow. Choosing the wrong interface, capturing packets at different positions before and after NAT, or overlooking network card offloading can all yield seemingly contradictory results, each of which may be factually accurate in its own context.

Learning Objectives

• Select packet capture points that align with fault assumptions;
• Distinguish between capture filters and Wireshark display filters;
• Extract evidence from TCP connection establishment, retransmissions, termination, and TLS handshake phases;
• Identify observation boundaries introduced by offload, namespaces, proxies, and encryption;
• Securely store and share pcap files.

First, Draw the Observation Points

A single request might traverse the following path:

client process
  -> client namespace / host
  -> NAT or proxy
  -> load balancer
  -> server host / container namespace
  -> server process

Capturing a SYN packet emitted by the client only confirms that the packet reached the client-side packet capture point; if no packet is seen on the server side, the issue could lie in the middle network layers, or it might be a misidentified interface or namespace. The strongest form of validation comes from simultaneous packet captures at both ends within the same time window, correlated using five-tuple information, TCP sequence numbers, and timestamps.

Before beginning, record the following:

• UTC time and clock synchronization status;
• Client and server host addresses and ports;
• Container, Pod, and network namespace boundaries, as well as proxy boundaries;
• Address transformations before and after NAT or load balancing;
• The trace ID or unique request header used to reproduce the request.

Using tcpdump for bounded packet capture

Only capture packets from systems and traffic you own or have explicit authorization to diagnose. Start by narrowing the scope of interfaces, hosts, ports, duration, and file size:

sudo tcpdump -i eth0 -nn -s 0 \
  'host 203.0.113.10 and tcp port 443' \
  -c 500 -w incident-443.pcap

Parameter meanings:

• -i eth0: Explicitly specify the interface to monitor; on Linux, any is convenient for initial screening, but link-layer information and behavior may differ from the actual interface;
• -nn: Skip hostname and service name resolution to reduce additional traffic and ambiguity;
• -s 0: Capture full packets according to tcpdump's current semantics; if sensitive data is involved, use a smaller snaplen sufficient for diagnosis only;
• -c 500: Stop after capturing a specified number of packets;
• -w: Save raw capture data rather than parsed text output.

For long-duration captures, rotate files and limit the total number of files to prevent disk exhaustion:

sudo tcpdump -i eth0 -nn -s 256 \
  'host 203.0.113.10 and tcp port 443' \
  -G 60 -W 5 -w 'incident-%Y%m%dT%H%M%S.pcap'

Different tcpdump versions may vary in the details of how -G, -W, and file name rotation interact. Before going into production monitoring, test these behaviors on the target version in a controlled environment.

Two Filters Are Not the Same Syntax

tcpdump's capture filter typically uses BPF syntax to decide which packets are captured and written to a file during acquisition:

host 203.0.113.10 and tcp port 443
tcp[tcpflags] & (tcp-syn|tcp-ack) != 0

Wireshark's display filter operates after packets have been captured, filtering what is displayed in the interface, and uses a different syntax:

ip.addr == 203.0.113.10 && tcp.port == 443
tcp.flags.syn == 1
tcp.analysis.retransmission
tls.handshake.type == 1

Pasting a display filter into tcpdump or a BPF filter into Wireshark's display filter box will result in failure or produce unexpected behavior.

Connection Establishment Phase: Judging by Package, Not by Error String

Typical TCP connection establishment:

client -> server  SYN
server -> client  SYN, ACK
client -> server  ACK

Several evidence patterns:

Client Packet Capture	Server Packet Capture	More Accurate Interpretation
SYN retransmitted with no response	No SYN observed	Request never reached the server's observation point, or the server captured the packet at the wrong location
SYN retransmitted with no response	SYN observed, no SYN-ACK received	Server path, policy, resource, or kernel handling requires further investigation
RST received	Possible RST sent	Host or intermediate device actively rejected the connection
SYN-ACK received, but client continues to send SYN	Server repeatedly sends SYN-ACK	Client failed to acknowledge the response, or the final ACK was lost in the path

Do not conclude with certainty that a packet loss occurred at a specific device based solely on a single client-side capture. Dual-end evidence narrows the scope to between the two observation points, but still does not identify which intermediate device is at fault.

Retransmission and Out-of-Order: Analyzer Insights Are Inferences

Wireshark's tcp.analysis.retransmission, fast_retransmission, and out_of_order are generated based on the current capture and analysis state, they are not kernel logs from the sender. Packet loss, snaplen settings, packet splitting, timestamp precision, and observation points can all influence the analyzer's conclusions.

When determining retransmissions, at least verify the following:

• The five-tuple and direction of the flow;
• TCP sequence number ranges and ACK values;
• SACK blocks;
• Whether the same payload reappears;
• Whether packet loss occurred within the capture file itself;
• Whether both endpoints observe the same segment.

When capturing from the sender side, TSO/GSO may cause the capture to show packets larger than the actual line MTU. On the receiver side, GRO/LRO may merge multiple segments into one. Additionally, checksums might be filled in by the network interface card after the packet has been captured, leading to a local capture showing "checksum incorrect" while the actual transmitted packet is valid. To accurately assess the frame structure in production, capture on the peer endpoint or at an intermediate TAP, or temporarily inspect offload settings in a controlled experiment. Avoid modifying NIC offload features in production environments without thorough evaluation.

TLS: What You Can See, What You Can't

Without a session key, application data in TLS 1.2 or 1.3 remains encrypted. Packet captures still reveal:

• IP addresses, ports, packet sizes, direction, and timing;
• Partial TCP or QUIC transmission behavior;
• In TLS ClientHello, unencrypted fields such as SNI, ALPN, and supported versions, unless protected by ECH;
• Server certificate visibility depends on TLS version and handshake phase; in TLS 1.3, most handshake messages are encrypted after the ServerHello.

Thus, "Follow TCP Stream" on HTTPS by default only yields TLS records and cannot reconstruct the underlying HTTP traffic.

In controlled testing environments, supported clients can export session secrets via SSLKEYLOGFILE, enabling Wireshark to decrypt the traffic. Key logs are equivalent to sensitive decryption material for the session and must be stored strictly under access control and promptly destroyed. Not all runtime environments or clients support this environment variable.

For QUIC/HTTP/3, standard TCP analysis is not applicable. Instead, analysis should prioritize combining client key logs, implementation-provided qlog files, connection IDs, server-side logs, and application-level traces.

NAT, Proxies, and Load Balancing

It's normal for IP addresses to change along the network path:

• Client-side SNAT (source network address translation) causes the server to see a public IP address as the source;
• DNAT (destination network address translation) or load balancers map a virtual address to a backend server's actual IP;
• Reverse proxies terminate the client connection and establish a new, independent upstream connection;
• Protocols like PROXY protocol or trusted forwarding headers can pass through the original client IP, but only if both ends are properly configured and validated.

Therefore, the statement "the server sees a source IP that isn't the user's IP" cannot be used directly to conclude that NAT configuration is wrong. Instead, draw out the endpoint of each connection segment and clearly identify which segment you're observing.

A Reusable Packet Capture Workflow

1. Document the hypothesis to be validated, such as "The client's SYN packet reached the load balancer but not the backend."
2. Select observation points that can distinguish between the two possible outcomes, and synchronize them to a common timestamp.
3. Reproduce the scenario using the minimal possible filter and a short capture window.
4. Preserve the original pcap file, the command used, tool version, and host location.
5. First verify that the packet exists, then examine timing, sequence numbers, and protocol fields.
6. Align the packet capture with connection tables, firewall counters, load balancer logs, and application traces.
7. Draw a narrowly scoped conclusion and clearly identify the intervals that remain unobserved.

Data Security

PCAP files may contain sensitive information such as cookies, authorization tokens, query parameters, internal IP addresses, DNS names, and unencrypted business data. Before sharing, ensure that:

• The capture scope and snaplen are restricted at the source;
• Data is stored under controlled conditions with minimal required permissions;
• Text-based replacement of binary PCAP data is not used directly for "de-identification";
• When necessary, use dedicated tools to generate sanitized copies and verify their integrity;
• Retention periods are clearly defined and temporary key logs are securely deleted.

Lesson Summary

Packet capture gives a sensor's view from one point in the network, not an omniscient view. First, choose the observation point, then interpret the protocol; first verify the raw packet, then consult the analyzer's hints. In the next lesson, these tools will be organized into three safe, repeatable local experiments.

Standards and Documentation Entry Points

• tcpdump manual page and pcap-filter manual page;
• Wireshark User's Guide: capture filters, display filters, TCP analysis;
• RFC 9293 (TCP), RFC 8449 (TLS 1.3), RFC 9000 (QUIC).