12.3 Repeatable Network Diagnostics Experiments

This lesson introduces faults only on the local loopback interface. The goal is not to memorize output from a specific machine, but to practice defining phases, setting deadlines, preserving evidence, and drawing limited conclusions.

Experiment Boundaries

• Use only 127.0.0.1 and local temporary ports;
• Do not scan public networks or unauthorized hosts;
• Do not modify firewalls, routing, or system DNS;
• Do not use curl -k to suppress certificate errors;
• Each background process must record its PID and terminate cleanly upon exit.

These examples use Linux, bash, GNU timeout, and Python 3.11+. Open a dedicated bash session and save the scripts and logs in a new temporary directory. A startup failure exits that session; its exit trap stops the service it created. Run the cleanup commands when each experiment ends.

Experiment 1: Distinguishing Refusals, Connection Success, and Response Timeout

Create stall_server.py:

import socket
import time

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener:
    listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    listener.bind(("127.0.0.1", 0))
    listener.listen()
    host, port = listener.getsockname()
    print(port, flush=True)

    connection, peer = listener.accept()
    with connection:
        print(f"accepted={peer}", flush=True)
        time.sleep(10)

Run it and read the port assigned by the operating system:

: >stall.log
python3 stall_server.py >stall.log 2>&1 &
server_pid=$!
trap 'kill "$server_pid" 2>/dev/null || true' EXIT

for _ in {1..50}; do
  server_port=$(sed -n '1p' stall.log)
  test -n "$server_port" && break
  sleep 0.1
done
if ! [[ "$server_port" =~ ^[0-9]+$ ]]; then
  cat stall.log >&2
  exit 1
fi

First, verify that the service is listening:

ss -lnt "sport = :$server_port"

Then request the service that "accepts the connection but does not return HTTP":

curl --noproxy '*' --verbose --connect-timeout 1 --max-time 2 \
  "http://127.0.0.1:$server_port/"

Expected result: TCP connects quickly, but no HTTP response arrives before the total deadline; curl exits with code 28. The timeout occurs after connection establishment. --noproxy '*' sends traffic directly over loopback so proxy environment variables cannot change the experiment path.

Stop the service and make the same request to the same port:

kill "$server_pid"
wait "$server_pid" 2>/dev/null || true
trap - EXIT

curl --noproxy '*' --verbose --connect-timeout 1 --max-time 2 \
  "http://127.0.0.1:$server_port/"

On a typical local TCP stack, a port with no listener quickly returns Connection refused, and curl exits with code 7. This behavior differs from the previous case where the connection was established but no HTTP response was received.

This experiment does not simulate silent SYN packet drop. Connection timeouts depend on routing, firewalls, and retransmission timers, conditions that cannot be reliably reproduced with arbitrary public addresses.

Experiment 2: Using Packet Capture to Verify Error Phases

Restart the server from Experiment 1 before capturing, then repeat the two curl requests and the intervening server shutdown. Copy the new port number into server_port in the capture terminal; separate terminals do not share shell variables. Terminal A captures only that temporary loopback port:

sudo timeout --signal=INT 8s tcpdump -i lo -nn -s 0 \
  "tcp port $server_port" -c 20 -w loopback-lab.pcap

Once tcpdump prints listening on lo, terminal B runs the requests and shutdown. Capture stops after 20 packets or 8 seconds. Exit code 124 from GNU timeout is expected when the time limit expires; still check for tcpdump capture errors. Read the packet summary:

tcpdump -nn -r loopback-lab.pcap

Answer the following questions:

1. Were SYN packets sent in both requests?
2. Which request completed the three-way handshake?
3. Was an RST packet observed in the rejection scenario?
4. In the response timeout scenario, did the client send any HTTP request bytes?
5. Which end issued the FIN or RST? How does this relate to the curl deadline?

The packet structure on the loopback interface differs slightly from that on a standard Ethernet interface, but TCP state transitions remain applicable for this experiment.

Experiment 3: Measuring a Local HTTP Request

Save this as http_server.py. It returns fixed text without exposing files from the working directory:

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        body = b"ok\n"
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; charset=utf-8")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

with ThreadingHTTPServer(("127.0.0.1", 0), Handler) as server:
    print(server.server_port, flush=True)
    server.serve_forever()

Start the server and read its port:

: >http.log
python3 http_server.py >http.log 2>&1 &
http_pid=$!
trap 'kill "$http_pid" 2>/dev/null || true' EXIT
for _ in {1..50}; do
  http_port=$(sed -n '1p' http.log)
  test -n "$http_port" && break
  sleep 0.1
done
if ! [[ "$http_port" =~ ^[0-9]+$ ]]; then
  cat http.log >&2
  exit 1
fi

Execute five requests:

for run in {1..5}; do
  curl --noproxy '*' --silent --show-error --output /dev/null \
    --connect-timeout 1 --max-time 2 \
    --write-out "run=$run code=%{http_code} connect=%{time_connect} starttransfer=%{time_starttransfer} total=%{time_total}\n" \
    "http://127.0.0.1:$http_port/"
done

In this local HTTP/1.x experiment, DNS and TLS are not involved. Nevertheless, variability still arises from first-request latency, thread creation, scheduling, and service logging. Do not claim an optimization is universally effective based on just five samples; this experiment is designed to train you in measuring specific fields and interpreting phases.

Clean up at the end:

kill "$http_pid"
wait "$http_pid" 2>/dev/null || true
trap - EXIT

A Shared TCP/TLS Deadline in Python

This program measures DNS separately, then gives TCP connection establishment and the TLS handshake one shared budget. Starting the next phase does not refill the clock. TLS still verifies the certificate chain and hostname; the optional cafile explicitly trusts a test CA without disabling verification. Errors preserve the failed phase and completed measurements. Save this as diagnose.py and pass the target explicitly. For local tests, use a temporary TLS server whose certificate covers the target IP address or hostname.

from __future__ import annotations

import argparse
import math
import socket
import ssl
import sys
import time
from dataclasses import dataclass

@dataclass(frozen=True)
class Stage:
    name: str
    milliseconds: float
    detail: str

class DiagnosisError(RuntimeError):
    def __init__(self, stage: str, completed: list[Stage], cause: OSError):
        super().__init__(f"{stage}: {cause}")
        self.stage = stage
        self.completed = tuple(completed)

def diagnose(host: str, port: int = 443, timeout: float = 3.0,
             *, cafile: str | None = None) -> list[Stage]:
    if not isinstance(host, str) or not host.strip():
        raise ValueError("host must be a nonempty string")
    if type(port) is not int or not 1 <= port <= 65535:
        raise ValueError("port must be an integer in 1..65535")
    if type(timeout) not in (int, float) or not 0 < timeout < math.inf:
        raise ValueError("timeout must be positive and finite")
    context = ssl.create_default_context(cafile=cafile)
    context.set_alpn_protocols(["http/1.1"])
    stages: list[Stage] = []
    phase = "dns"
    try:
        started = time.monotonic()
        addresses = socket.getaddrinfo(
            host, port, type=socket.SOCK_STREAM, proto=socket.IPPROTO_TCP,
        )
        if not addresses:
            raise OSError("resolver returned no addresses")
        stages.append(Stage("dns", (time.monotonic() - started) * 1000,
                            f"answers={len(addresses)}"))

        deadline = time.monotonic() + timeout
        def remaining() -> float:
            seconds = deadline - time.monotonic()
            if seconds <= 0:
                raise TimeoutError("TCP/TLS deadline exceeded")
            return seconds

        family, socktype, proto, _, sockaddr = addresses[0]
        phase = "tcp"
        started = time.monotonic()
        with socket.socket(family, socktype, proto) as raw:
            raw.settimeout(remaining())
            raw.connect(sockaddr)
            stages.append(Stage("tcp", (time.monotonic() - started) * 1000,
                                f"peer={sockaddr}"))
            phase = "tls"
            started = time.monotonic()
            with context.wrap_socket(raw, server_hostname=host,
                                     do_handshake_on_connect=False) as tls:
                tls.settimeout(remaining())
                tls.do_handshake()
                stages.append(Stage(
                    "tls", (time.monotonic() - started) * 1000,
                    f"version={tls.version()} alpn={tls.selected_alpn_protocol()}",
                ))
    except OSError as error:
        raise DiagnosisError(phase, stages, error) from error
    return stages

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("host")
    parser.add_argument("--port", type=int, default=443)
    parser.add_argument("--timeout", type=float, default=3.0)
    parser.add_argument("--cafile")
    args = parser.parse_args()
    try:
        completed = diagnose(args.host, args.port, args.timeout,
                             cafile=args.cafile)
    except DiagnosisError as error:
        for stage in error.completed:
            print(stage)
        print(error, file=sys.stderr)
        raise SystemExit(1)
    for stage in completed:
        print(stage)

It still has clear limitations:

• It only attempts the first address returned by getaddrinfo(), without implementing Happy Eyeballs;
• Synchronous getaddrinfo() has no hard deadline here. TLS context preparation is also outside the budget, so timeout does not bound the entire function;
• It does not send any HTTP requests;
• A single sample cannot represent long-term performance;
• alpn=None means ALPN was not negotiated. This probe offers only http/1.1, so it cannot determine whether the server supports HTTP/2;
• Fix input and CA-file configuration errors before measuring.

Teaching programs should explicitly state these boundaries rather than treating a successful run as evidence of production-grade probe capability.

About Port Scanning

nmap is useful for asset discovery and fault verification, and may trigger alerts, violate authorization boundaries, or place undue pressure on vulnerable services. Scan only targets explicitly under your management or formally authorized, and agree on source address, target network range, port range, time window, and scan rate.

Even if the scan results indicate open, closed, or filtered, these are classifications based on response detection, not proof of application health. Production services require protocol-level requests, authentication, and server-side validation for confirmation.

Acceptance Checklist

• [ ] Can separate connect timeout from response timeout;
• [ ] Can use ss to prove the local listening address, rather than vaguely stating "a port is open";
• [ ] Can identify SYN, SYN-ACK, ACK, and RST/FIN packets from a local pcap capture;
• [ ] Can describe the interface, namespace, time window, and filter used in a single packet capture;
• [ ] Can explain that curl's timing values are cumulative;
• [ ] Can provide at least one alternative explanation for each conclusion that hasn't been ruled out.

Chapter Summary

Chapter 12 established a complete diagnostic chain: first defining symptoms and stages, then progressively narrowing down the root cause through name resolution, routing, transmission, TLS, and HTTP evidence at each layer. When deeper investigation is needed, bounded packet captures should be performed at the appropriate observation points. Finally, local experimentation is used to verify whether one truly can distinguish between different error stages.

The next chapter moves to IPv6. While the same diagnostic approach remains valid, new variables arise from address selection, neighbor discovery, ICMPv6, and dual-stack fallback.

Experiment References

• Python socket and ssl;
• curl command manual.