6.3 Network Detection and Resilience: IDS, Traffic Analysis, and DDoS
Network control defines what should happen; detection systems observe what actually happens. With increasingly prevalent encrypted traffic, detection can no longer rely solely on deep payload inspection. Connection metadata, DNS queries, identity information, endpoint behavior, and application telemetry must be correlated to enable effective detection.
IDS and IPS Have Different Failure Modes
A network IDS commonly observes mirrored traffic and alerts; false positives mainly consume analyst capacity. IPS is placed directly in the traffic path and can actively block; misfires can directly cause availability incidents. Only rules with high confidence and fast rollback capabilities should be used for automatic blocking; all others should first be observed, correlated, and manually verified.
Signature detection excels at identifying known patterns, while anomaly detection looks for behavior that deviates from a baseline. An anomaly does not necessarily mean malicious activity: scheduled releases, batch jobs, and holiday traffic can all shift the baseline. Every high-priority detection must include asset scope, attack hypothesis, expected false positives, and defined response actions.
Metadata Can Still Answer Key Questions
Even when TLS payloads are invisible, flow and endpoint data can still provide answers to critical questions such as:
Which host first connected to a new destination?
Which machine suddenly scanned multiple ports?
Which workload consistently uploads to non-standard regions?
Are DNS queries showing high entropy or newly registered domains?
Is the same identity simultaneously accessing from impossible devices or locations?The visibility and value of TLS fingerprints, SNI, certificates, and traffic sizes depend on the observation point. Passive sensors generally cannot read certificates in an encrypted TLS 1.3 handshake, and ECH hides the inner ClientHello/SNI. Endpoints and terminating proxies offer different views; client updates also change fingerprints. These should be treated as contextual signals, not standalone justifications for arbitrary blocking.
Recording controlled application metadata at TLS termination points is generally more reliable than broadly decrypting traffic. Full-scale TLS interception introduces new root CA trust chains, private keys, privacy risks, compatibility issues, and single points of failure, risks that must be strictly scoped and limited to specific use cases.
Detection-as-Code Requires Test Data
Rules must be versioned and include:
- Data sources and required fields;
- ATT&CK technique or internal attack assumption;
- Thresholds and aggregation windows;
- Allowed business exceptions;
- Owner, severity, and runbook;
- Positive, negative, and historical replay test samples.
The absence of data from a source should not be interpreted as "zero events, therefore secure." Monitoring must include sensor coverage, log latency, field parsing failures, and rule execution status.
DDoS is a problem of capacity and state exhaustion
An attack can exhaust bandwidth, connection tables, TLS handshake CPU, threads, queues, database connections, or expensive business operations. Layered defense:
Upstream scrubbing/CDN + Anycast traffic distribution
→ L4 SYN/connection protection
→ L7 rate limiting and identity quotas
→ Application-level concurrency, queues, and timeouts
→ Dependency isolation and fallbacksLimiting by source IP alone harms users behind NAT and can be circumvented by distributed sources. Instead, combine account, API key, device, tenant, operational cost, and global capacity, and apply different budgets for anonymous versus authenticated traffic.
Rate limiters themselves must make trade-offs around distributed consistency: overly strong consistency can become a bottleneck, while fully local limits allow the aggregate accepted quota to grow with instance count. At 200 requests/second per instance, four instances can admit about 800 requests/second if traffic reaches them evenly; eight can admit about 1,600. Algorithm-specific burst allowances are additional. The key is to limit the most expensive resources after minimal, low-cost filtering, such as screening traffic before expensive operations like password hashing, complex queries, or file transcoding.
Anycast distributes traffic across ingress points; it does not identify or scrub attacks itself. Once the upstream link is saturated, a fast origin-side limiter cannot restore that capacity. Also restrict paths that bypass the CDN to reach the origin directly.
Response Must Preserve Evidence and Control Blast Radius
After a high-confidence event is triggered, actions such as revoking short-lived credentials, isolating workloads, tightening egress rules, switching to read-only mode, blocking malicious destinations, or enhancing authentication assurances should be executed. Define scope, duration, and authorization for each action. Do not automatically lift isolation merely because a timer expires: require conditions such as removed persistence and restored trustworthy identities, extending isolation when necessary.
When ongoing harm permits, preserve volatile evidence before restarting or clearing state. If immediate containment is necessary, record the action and lost evidence rather than delaying indefinitely for collection. Preserve relevant evidence across key flows, DNS records, identities, processes, image digests, and memory/disk artifacts, ensuring time synchronization is maintained. Network teams see IPs, identity teams see subjects, and application teams see traces. Only through shared correlation and a unified asset inventory can the full attack chain be reconstructed.
Exercise List
- Simulate an application host transmitting to a new external domain, how long until detection triggers?
- Simulate internal port scanning: does segmentation block the traffic, and does the IDS correctly identify the initiating entity?
- Disable one sensor, does the platform generate an alert for missing data?
- Replay a normal peak traffic event, does any rule incorrectly block traffic?
- Perform load testing on an expensive anonymous interface, on which layer does saturation first occur?
- After isolation, is the recovery and cleanup of temporary policies fully auditable?
Run these exercises against owned test endpoints with synthetic data and agreed traffic limits. The next chapter brings verifiable security requirements into software development.