12.3 Repeatable Network Diagnostics Experiment

This lesson introduces faults only on the local loopback interface. The goal is not to memorize output from a specific machine, but to practice defining phases, setting deadlines, preserving evidence, and drawing limited conclusions.

Experiment Boundaries

• Use only 127.0.0.1 and local temporary ports;
• Do not scan public networks or unauthorized hosts;
• Do not modify firewalls, routing, or system DNS;
• Do not use curl -k to suppress certificate errors;
• Each background process must record its PID and terminate cleanly upon exit.

Command options may vary across systems. The following shell examples are based on Linux/bash; Python requires version 3.11 or later.

Experiment 1: Distinguishing Refusals, Connection Success, and Response Timeout

Create stall_server.py:

import socket
import time


with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener:
    listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    listener.bind(("127.0.0.1", 0))
    listener.listen()
    host, port = listener.getsockname()
    print(port, flush=True)

    connection, peer = listener.accept()
    with connection:
        print(f"accepted={peer}", flush=True)
        time.sleep(10)

Run it and read the port assigned by the operating system:

python3 stall_server.py >stall.log 2>&1 &
server_pid=$!

for _ in {1..50}; do
  server_port=$(sed -n '1p' stall.log)
  test -n "$server_port" && break
  sleep 0.1
done
test -n "$server_port"

First, verify that the service is listening:

ss -lnt "sport = :$server_port"

Then request the service that "accepts the connection but does not return HTTP":

curl --verbose --connect-timeout 1 --max-time 2 \
  "http://127.0.0.1:$server_port/"

Expected result: TCP connects successfully very quickly, and the HTTP response does not arrive within the total deadline. The timeout occurs after the connection is established.

Stop the service and make the same request to the same port:

kill "$server_pid"
wait "$server_pid" 2>/dev/null || true

curl --verbose --connect-timeout 1 --max-time 2 \
  "http://127.0.0.1:$server_port/"

In a typical native local TCP stack, an unbound port usually returns Connection refused immediately. This behavior differs from the previous case where the connection was established but no HTTP response was received.

This experiment does not simulate silent SYN packet drop. Connection timeouts depend on routing, firewalls, and retransmission timers, conditions that cannot be reliably reproduced with arbitrary public addresses.

Experiment 2: Using Packet Capture to Verify Error Phases

Terminal A captures packets on the loopback interface, focusing only on the temporary port:

sudo tcpdump -i lo -nn -s 0 "tcp port $server_port" -c 20 -w loopback-lab.pcap

Terminal B executes both "service runtime request" and "service stopped request." Then it reads the response summary:

tcpdump -nn -r loopback-lab.pcap

Answer the following questions:

1. Were SYN packets sent in both requests?
2. Which request completed the three-way handshake?
3. Was an RST packet observed in the rejection scenario?
4. In the response timeout scenario, did the client send any HTTP request bytes?
5. Which end issued the FIN or RST? How does this relate to the curl deadline?

The packet structure on the loopback interface differs slightly from that on a standard Ethernet interface, but TCP state transitions remain applicable for this experiment.

Experiment 3: Measuring a Local HTTP Request

Start the standard library HTTP server:

python3 -m http.server 0 --bind 127.0.0.1 >http.log 2>&1 &
http_pid=$!

The log formatting across different Python versions is not suitable as a stable port-discovery interface. To ensure reproducible experiments, use the following minimal service that outputs its port:

from http.server import ThreadingHTTPServer, SimpleHTTPRequestHandler


server = ThreadingHTTPServer(("127.0.0.1", 0), SimpleHTTPRequestHandler)
print(server.server_port, flush=True)
server.serve_forever()

Save it as http_server.py and run it:

kill "$http_pid" 2>/dev/null || true
wait "$http_pid" 2>/dev/null || true

python3 http_server.py >http.log 2>&1 &
http_pid=$!
for _ in {1..50}; do
  http_port=$(sed -n '1p' http.log)
  test -n "$http_port" && break
  sleep 0.1
done
test -n "$http_port"

Execute five requests:

for run in {1..5}; do
  curl --silent --show-error --output /dev/null \
    --write-out "run=$run code=%{http_code} connect=%{time_connect} starttransfer=%{time_starttransfer} total=%{time_total}\n" \
    "http://127.0.0.1:$http_port/"
done

In this local HTTP/1.x experiment, DNS and TLS are not involved. Nevertheless, variability still arises from first request latency, filesystem caching, scheduling, and service logging. Do not claim an optimization is universally effective based on just five samples; this experiment is designed to train you in measuring specific fields and interpreting phases.

Clean up at the end:

kill "$http_pid"
wait "$http_pid" 2>/dev/null || true

A Python Diagnostics Tool with Deadline

The following program records DNS, TCP, and TLS handshakes. It is not a monitoring system, but it is more suitable than a timeout-free socket.connect() as a teaching scaffold.

from __future__ import annotations

import socket
import ssl
import time
from dataclasses import dataclass


@dataclass(frozen=True)
class Stage:
    name: str
    milliseconds: float
    detail: str


def diagnose(host: str, port: int = 443, timeout: float = 3.0) -> list[Stage]:
    stages: list[Stage] = []

    started = time.monotonic()
    addresses = socket.getaddrinfo(
        host,
        port,
        type=socket.SOCK_STREAM,
        proto=socket.IPPROTO_TCP,
    )
    stages.append(Stage(
        "dns",
        (time.monotonic() - started) * 1000,
        f"answers={len(addresses)}",
    ))

    family, socktype, proto, _, sockaddr = addresses[0]
    started = time.monotonic()
    raw = socket.socket(family, socktype, proto)
    raw.settimeout(timeout)
    try:
        raw.connect(sockaddr)
        stages.append(Stage(
            "tcp",
            (time.monotonic() - started) * 1000,
            f"peer={sockaddr}",
        ))

        context = ssl.create_default_context()
        started = time.monotonic()
        with context.wrap_socket(raw, server_hostname=host) as tls:
            stages.append(Stage(
                "tls",
                (time.monotonic() - started) * 1000,
                f"version={tls.version()} alpn={tls.selected_alpn_protocol()}",
            ))
    except BaseException:
        raw.close()
        raise

    return stages


if __name__ == "__main__":
    for stage in diagnose("example.com"):
        print(stage)

It still has clear limitations:

• It only attempts the first address returned by getaddrinfo(), without implementing Happy Eyeballs;
• The DNS call itself lacks an independent Python socket deadline;
• It does not send any HTTP requests;
• A single sample cannot represent long-term performance;
• Exceptional output needs further structuring in real-world tools.

Teaching programs should explicitly state these boundaries rather than treating a successful run as evidence of production-grade probe capability.

About Port Scanning

nmap is useful for asset discovery and fault verification, and may trigger alerts, violate authorization boundaries, or place undue pressure on vulnerable services. Scan only targets explicitly under your management or formally authorized, and agree on source address, target network range, port range, time window, and scan rate.

Even if the scan results indicate open, closed, or filtered, these are classifications based on response detection, not proof of application health. Production services require protocol-level requests, authentication, and server-side validation for confirmation.

Acceptance Checklist

• [ ] Can separate connect timeout from response timeout;
• [ ] Can use ss to prove the local listening address, rather than vaguely stating "a port is open";
• [ ] Can identify SYN, SYN-ACK, ACK, and RST/FIN packets from a local pcap capture;
• [ ] Can describe the interface, namespace, time window, and filter used in a single packet capture;
• [ ] Can explain that curl's timing values are cumulative;
• [ ] Can provide at least one alternative explanation for each conclusion that hasn't been ruled out.

Chapter Summary

Chapter 12 established a complete diagnostic chain: first defining symptoms and stages, then progressively narrowing down the root cause through name resolution, routing, transmission, TLS, and HTTP evidence at each layer. When deeper investigation is needed, bounded packet captures should be performed at the appropriate observation points. Finally, local experimentation is used to verify whether one truly can distinguish between different error stages.

The next chapter moves to IPv6. While the same diagnostic approach remains valid, new variables arise from address selection, neighbor discovery, ICMPv6, and dual-stack fallback.