12.1 From Symptoms to Evidence: Layered Network Diagnostics

Network failures rarely tell you directly where the problem lies. A "can't load" message in the browser or a "timeout" in a client app is just a symptom. Effective troubleshooting isn't about running a series of commands in sequence; it's about forming a falsifiable hypothesis first, then gathering just enough evidence to confirm or reject it.

Learning Objectives

After completing this lesson, you will be able to:

• Rewrite vague symptoms into fault descriptions that include scope, time, and phase;
• Distinguish the evidence boundaries across DNS, routing, TCP, TLS, and HTTP stages;
• Correctly interpret the outputs of ping, traceroute, ss, dig, and curl;
• Use controlled experiments to narrow down fault scope, rather than jumping to conclusions based on a single command.

Define the Failure Boundary Clearly

Before entering any command in the terminal, answer these five questions:

1. Who is failing? A single user, a host, an availability zone, or all callers?
2. What is being accessed? A domain name, a set of IPs, a port, or the entire network?
3. When did it start? Has it been consistently failing, intermittently failing, or only occurring during peak loads?
4. At which stage is the failure occurring? DNS resolution, connection establishment, TLS handshake, first byte, or during data transfer?
5. What changed recently? DNS configuration, certificate, routing, firewall rules, proxy settings, deployment, or capacity?

"Site is slow" is not actionable. "Access to api.example.test via the Beijing office network completes DNS resolution in 20 ms, but TLS handshake increases from 40 ms to 3 seconds; mobile network access is unaffected" generates multiple testable hypotheses.

Layering Is Not a Fixed Command Sequence

The value of layer-by-layer diagnosis lies in isolating variables, not in demanding that every troubleshooting session begin with checking the network cable. If monitoring has already confirmed successful DNS resolution, leverage that evidence directly. If only one host is failing, begin by comparing its configuration to that of a healthy host.

Stage	Core Question	Common Evidence
Local and Link Layer	Are interfaces, addresses, and default routes present?	ip link, ip address, ip route
Name Resolution	What actual addresses does the application resolve to?	getent ahosts, resolvectl, dig
Path	Which interface, source address, and next-hop does the kernel use?	ip route get, ping, traceroute, mtr
Transport	Is the service listening? What happens after the SYN?	ss, nc, packet capture
TLS	Do SNI, certificate chain, time, and ALPN values match?	openssl s_client, curl -v
HTTP	Are the status code, redirects, first byte, and response body behaving as expected?	curl, server logs, trace ID

Step 1: Check Local Path

203.0.113.10 below is a documentation address, not a working test server. Replace it with the authorized target for this investigation before sending probes. On Linux, inspect the interfaces, addresses, and routing decision:

ip -brief link
ip -brief address
ip route show
ip route get 203.0.113.10

ip route get shows the kernel's current routing decision for this destination, typically including the output interface, next hop, and source address. It does not prove remote reachability, nor does it indicate that packets have actually been sent.

If the application runs inside a container or network namespace, the routing check must be performed within the application's network namespace. The host machine may have a default route, but that does not guarantee the container has the same routing configuration.

Step 2: Verify the Actual Name Resolution Used by the Application

Direct DNS queries with dig are useful for inspecting records and authority chains; however, applications typically operate through the system resolver and may be influenced by /etc/hosts, NSS, systemd-resolved, search domains, and caching mechanisms. First establish what the application receives, then compare DNS answers. getent follows the system name-service path; it does not reproduce browser DoH, custom resolvers, or application caches.

# Follow the configured system name-service path
getent ahosts example.com

# View DNS Responses and TTL
dig example.com A
dig example.com AAAA

# Compare with the specified recursive resolver
dig @1.1.1.1 example.com A

# Iterate from the root zone; this does not reproduce the application resolver cache
dig +trace example.com

Key points to compare:

• The response code is NOERROR, NXDOMAIN, or SERVFAIL;
• Whether NOERROR has an empty answer. Check SOA/NS records in the authority section to distinguish NODATA from a referral; an empty answer alone does not prove a negative response;
• Whether both A and AAAA records exist, and whether the application prefers IPv6;
• Whether the CNAME chain ultimately resolves to an IP address;
• Whether different resolvers return different answers or TTLs;
• Whether DNSSEC validation fails, split DNS views, or negative caching are involved.

Running only dig +short will omit critical evidence such as response codes, TTLs, and authority information, making it unsuitable for a complete diagnosis.

Step 3: Explain ICMP and Path Discovery

What Does ping Prove?

ping -c 5 203.0.113.10

Receiving an Echo Reply proves that, at the time of the test, there exists a path between the source and destination that allows ICMP echo requests and responses to travel back and forth. It does not prove that TCP port 443 is operational or that the application is healthy.

The absence of an Echo Reply may result from several factors: the destination being unreachable, packet loss along the path, ICMP being filtered or rate-limited, or the destination explicitly disabling ICMP responses. Therefore, "ping failure" alone cannot be taken as definitive evidence of a network outage.

Round-Trip Time (RTT) includes the forward path, processing at the destination, and the return path. Internet routing can be asymmetric, so an increased RTT on a single hop does not necessarily mean that that forward link is the bottleneck.

Limits of traceroute and mtr

traceroute incrementally increases TTL/Hop Limit and uses the ICMP Time Exceeded messages returned by routers to identify a set of potential intermediate nodes:

traceroute -n 203.0.113.10
mtr --report --report-cycles 20 --no-dns 203.0.113.10

When interpreting results, follow these three rules:

1. If a single hop shows * while subsequent hops and the destination respond normally, this typically indicates that the node did not respond to the probe or is rate-limiting ICMP replies.
2. High reported loss at one hop, without loss at subsequent hops, does not prove forwarding loss; routers may prioritize business traffic and defer or drop ICMP packets due to lower priority.
3. ECMP, load balancing, and path changes may cause each probe to traverse a different route. What is observed is evidence of probe paths, not a complete network topology.

Only when an anomaly begins at a specific hop and persists through subsequent nodes and the destination (especially when the probe traffic closely mirrors the actual business traffic in terms of protocol, source address, and destination port) should the location near that hop be considered suspect.

Step 4: Distinguishing Between Listening, Rejecting, and Timeout

On the server side, inspect the listening socket:

ss -lntp
ss -lnt 'sport = :443'

ss -lntp only indicates that a process within the current network namespace is listening on a local address and port. Listening 127.0.0.1:443 does not mean the external interface is accessible; internal container listening does not imply that the host has published the port.

From the client side, test the TCP connection:

nc -vz -w 3 example.com 443

Common observations:

Phenomenon	Direct Evidence	Possible Causes
Immediate Connection refused	Typically receives RST or is directly rejected by the local system	No listener, incorrect port mapping, device actively rejects the connection
Connection phase timeout	No connection established within the expected time	Silent drop, routing issues, return path problems, remote endpoint unresponsive, SYN queue or resource pressure
Connection succeeds but read timeout	TCP handshake completes, application data fails to arrive in time	Service processing is slow, upstream latency, protocol-level waiting, response dropped mid-way

These are potential explanations, not definitive root causes based solely on error messages. To differentiate among them, combine client-side packet captures, server-side listening status, firewall counter logs, and server-side application logs.

Step 5: Separate TLS from HTTP

Use curl to log timing at each stage:

curl --silent --show-error --output /dev/null \
  --connect-timeout 3 --max-time 10 \
  --write-out $'remote_ip=%{remote_ip}\nhttp_code=%{http_code}\nnamelookup=%{time_namelookup}\nconnect=%{time_connect}\nappconnect=%{time_appconnect}\nstarttransfer=%{time_starttransfer}\ntotal=%{time_total}\n' \
  https://example.com/

• time_namelookup: Time from start to DNS resolution complete;
• time_connect: Time from start to transport connection established;
• time_appconnect: Time from start to completion of TLS/application-layer handshake;
• time_starttransfer: Time from start to receipt of the first response byte;
• time_total: Total time from start to end of transmission.

--connect-timeout covers the connection phase, including DNS, transport, and TLS handshakes; it is not a TCP-only timer. --max-time bounds this transfer. Without --fail, HTTP 404/500 responses can still produce curl exit code 0, so also check the status code and application response.

These are cumulative durations. Approximate duration for a given phase must be calculated by subtracting adjacent timestamps. The meaning of each phase can change due to connection reuse, proxies, HTTP/3, or version differences. Record curl --version alongside the actual negotiated protocol to avoid treating these formulas as universal truths across environments.

Bypass DNS but Preserve Host and SNI

curl --noproxy '*' --connect-timeout 3 --max-time 10 \
  --resolve example.com:443:203.0.113.10 https://example.com/

Here, --noproxy '*' explicitly tests a direct backend connection so a proxy cannot take over resolution; this does not reproduce a path that requires a proxy. --resolve provides an address for a specific host:port, while the original domain name remains in the URL. This setup is ideal for testing a backend endpoint while preserving the HTTP Host header and TLS SNI. Directly requesting https://203.0.113.10/ often changes virtual host selection and certificate validation, making it unsuitable as a valid control case.

Verify TLS

openssl s_client \
  -connect example.com:443 \
  -servername example.com \
  -verify_hostname example.com -verify_return_error \
  -alpn h2,http/1.1 \
  -showcerts </dev/null

-verify_return_error stops the handshake on certificate verification errors. Otherwise, s_client can report an error and continue, so a successful process exit does not prove successful verification. -alpn offers protocols; inspect the server selection. Check that the certificate is within its validity period, that the hostname matches, that the server sends required intermediate certificates, and that the final verify return code is correct. -servername explicitly sets SNI; without it, multi-tenant services might return a certificate from a different site.

curl -k or --insecure can only be used to determine whether a failure is related to certificate validation; they cannot serve as a fix, nor should they be included in production scripts.

A Table of Executable Evidence

Known Fact	Next Step to Verify	Conclusion to Hold Off
The system resolver returned no address	Compare getent, dig, the selected resolver, and local configuration	"The authoritative DNS is down"
ping succeeds, port 443 times out	Use nc and curl to test the port, perform SYN packet capture on both ends	"The server is operational"
TCP connection succeeds, TLS handshake fails	Verify SNI, timestamps, certificate chain, and protocol version	"The network is fine, this must be a code bug"
TLS succeeds, first byte is slow	Investigate server-side tracing, upstream dependencies, and queueing	"Bandwidth is insufficient"
mtr shows packet loss at an intermediate node, target does not lose packets	Extend observation period and compare with business metrics	"The router is dropping 30% of business packets"

Lesson Summary

Work through a cycle of hypothesis, experiment, evidence, and conclusion. A single output usually eliminates only a few possibilities. Define the failure scope and phase, then change one variable at a time and record what the evidence actually supports.

The next lesson will cover packet capture: how to select observation points, write effective filters, and avoid misinterpreting NIC offload effects or analyzer hints as facts about traffic on the wire.

Standards and Documentation Entry Points

• Linux ip-route(8), ss(8) manual pages;
• BIND 9 Administrator Reference Manual: dig;
• curl command manual;
• OpenSSL s_client manual;
• RFC 792 (ICMPv4), RFC 4443 (ICMPv6).