4.2 HTTP/1.1 Framing and the Minimal Server

A recv may return only half of POST /echo HT; a later call may contain the rest of the header plus the beginning of another request. If an implementation treats one recv as one HTTP message, parsing depends on network segmentation, an accident the application does not control.

HTTP/1.1 defines message boundaries on a TCP byte stream. A parser must incrementally read the start-line, field section, and content, while strictly enforcing time and memory limits. If proxies or origin servers interpret these boundaries differently, they can create openings for request smuggling.

This lesson first establishes a clear contract for message parsing, then implements a minimal server that supports only HTTP/1.1, Content-Length, and one request per connection. The feature set is intentionally narrow, with each boundary explicitly defined and discussed.

1. A Single recv Has No Protocol Significance

The following approach is invalid:

request = connection.recv(65536)
method, target, version = request.split(b"\r\n", 1)[0].split()

recv returns "how many bytes are currently available," not "the next HTTP message":

• The start-line or \r\n\r\n may be split across multiple recv calls;
• The body may not have arrived in full;
• Headers and body may arrive in a single chunk;
• On persistent connections, the next request might already be in the buffer;
• The peer might send only a partial message and then pause;
• EOF could occur before the declared message length is reached.

Therefore, the parser must maintain a buffer that persists across recv calls, along with a clear read state.

2. The Boundary of an HTTP/1.1 Request

For a standard request, first locate:

request-line CRLF
field-line CRLF
...
CRLF

The final empty line marks the end of the header section. Whether a message body exists is not determined by heuristic assumptions such as "POST requests usually have a body," but rather by the framing field.

In this lesson, we accept:

Content-Length: 11

We do not implement:

Transfer-Encoding: chunked

When a feature is unsupported, we must explicitly reject the request. We cannot fall back to "keep reading from the socket until no data is available." On persistent connections, this would simultaneously break message boundary detection and connection liveness.

3. Parser Sets Resource Limits

Before writing code, establish the contract:

Item	Teaching Server Policy
HTTP version	Accepts only HTTP/1.1
request-target	Accepts an ASCII origin-form subset; checks escapes, controls, and fragments
header section	Maximum 16 KiB, including the trailing empty line
body	Maximum 1 MiB
Host header	Exactly once: nonempty reg-name or bracketed IPv6, with optional port
Content-Length	At most one occurrence; must be a non-negative decimal integer
Transfer-Encoding	Not supported; if present alongside Content-Length, returns 400
timeout	Shared 5-second header/body deadline after accept; separate 5-second write timeout
connection reuse	After response, sends Connection: close

These numbers are configuration values for this experiment, not global constants of HTTP. In production servers, limits should be chosen based on business requirements, proxy chain behavior, and resource budgets, and should distinguish between oversized headers, oversized content, and read timeouts.

This is a teaching subset, not a fully conforming HTTP/1.1 server. RFC 9112 requires recipients to decode chunked, which this example deliberately omits. It also rejects Host extensions such as IPvFuture and IPv6 zone identifiers, does not route virtual hosts, and does not decode URL paths.

4. Why Ambiguous Framing Is Dangerous

Suppose the frontend proxy splits a request according to Content-Length, while the backend prioritizes splitting according to Transfer-Encoding. The same byte stream could be partitioned into different messages at each end, allowing an attacker to smuggle hidden requests into the backend.

RFC 9112 establishes strict rules for framing conflicts. This experiment adopts a narrower, more conservative approach:

• If both Transfer-Encoding and Content-Length appear: return 400 and terminate;
• If any Transfer-Encoding is detected: return 501 and terminate;
• If Content-Length is repeated: return 400 and terminate;
• If the header name, whitespace, or control characters are invalid: return 400 and terminate.

The teaching code is not part of the proxy chain, yet it still avoids lenient, speculative parsing. Every hop in the message path must agree on the same message boundaries.

5. A Runnable Minimal Server

Save the following as mini_http11.py, using Python 3.10+. It supports:

• GET /health;
• HEAD /health;
• POST /echo;
• 404, 405, and several parser errors;
• reassembly of header/body from arbitrary TCP segments.

import re
import socket
import time
import logging
from ipaddress import IPv6Address
from email.utils import formatdate
from dataclasses import dataclass

MAX_HEADER_SECTION = 16 * 1024
MAX_BODY = 1024 * 1024
TOKEN = re.compile(rb"^[!#$%&'*+\-.^_\x60|~0-9A-Za-z]+$")

TARGET = re.compile(rb"/(?:[A-Za-z0-9._~!$&'()*+,;=:@/?-]|%[0-9A-Fa-f]{2})*")

def valid_host(value):
    # This subset accepts reg-name or bracketed IPv6, with an optional port.
    if value.startswith("["):
        match = re.fullmatch(r"\[([0-9A-Fa-f:.]+)\](?::([0-9]*))?", value)
        if not match:
            return False
        try:
            IPv6Address(match[1])
        except ValueError:
            return False
        port = match[2]
    else:
        host, separator, port = value.partition(":")
        if not re.fullmatch(r"(?:[A-Za-z0-9._~!$&'()*+,;=-]|%[0-9A-Fa-f]{2})+", host):
            return False
        if separator and (not port.isascii() or (port and not port.isdecimal())):
            return False
    digits = (port or "").lstrip("0") or "0"
    return len(digits) <= 5 and int(digits) <= 65535

class DeadlineReader:
    def __init__(self, connection, deadline):
        self.connection = connection
        self.deadline = deadline
        self.method = None

    def recv(self, length):
        remaining = self.deadline - time.monotonic()
        if remaining <= 0:
            raise TimeoutError("request deadline exceeded")
        self.connection.settimeout(remaining)
        return self.connection.recv(length)

class HTTPError(Exception):
    def __init__(self, status, message):
        super().__init__(message)
        self.status = status
        self.message = message

@dataclass(frozen=True)
class Request:
    method: str
    target: str
    version: str
    headers: dict[str, list[str]]
    body: bytes

def receive_header_section(connection):
    buffer = bytearray()
    marker = b"\r\n\r\n"

    while marker not in buffer:
        if len(buffer) >= MAX_HEADER_SECTION:
            raise HTTPError(431, "header section too large")

        chunk = connection.recv(
            min(4096, MAX_HEADER_SECTION - len(buffer))
        )
        if chunk == b"":
            raise HTTPError(400, "EOF before header section")
        buffer.extend(chunk)

    marker_at = buffer.find(marker)
    return bytes(buffer[:marker_at]), bytes(buffer[marker_at + 4:])

def receive_exact(connection, initial, length):
    body = bytearray(initial[:length])
    while len(body) < length:
        chunk = connection.recv(min(4096, length - len(body)))
        if chunk == b"":
            raise HTTPError(400, "EOF before complete request content")
        body.extend(chunk)
    return bytes(body)

def parse_request(connection):
    head, buffered_after_head = receive_header_section(connection)
    lines = head.split(b"\r\n")
    if not lines or not lines[0]:
        raise HTTPError(400, "missing request line")

    request_parts = lines[0].split(b" ")
    if len(request_parts) != 3 or any(not part for part in request_parts):
        raise HTTPError(400, "malformed request line")

    method_bytes, target_bytes, version_bytes = request_parts
    if not TOKEN.fullmatch(method_bytes):
        raise HTTPError(400, "invalid method")

    try:
        method = method_bytes.decode("ascii")
        target = target_bytes.decode("ascii")
        version = version_bytes.decode("ascii")
    except UnicodeDecodeError as error:
        raise HTTPError(400, "non-ASCII request line") from error

    connection.method = method
    if version != "HTTP/1.1":
        raise HTTPError(400, "only HTTP/1.1 is supported")
    if not TARGET.fullmatch(target_bytes):
        raise HTTPError(400, "only origin-form targets are supported")

    headers: dict[str, list[str]] = {}
    for raw_line in lines[1:]:
        if not raw_line:
            raise HTTPError(400, "unexpected empty field line")
        if raw_line[:1] in (b" ", b"\t"):
            raise HTTPError(400, "obsolete folded field")
        if b":" not in raw_line:
            raise HTTPError(400, "field without colon")

        name_bytes, value_bytes = raw_line.split(b":", 1)
        if not TOKEN.fullmatch(name_bytes):
            raise HTTPError(400, "invalid field name")

        value_bytes = value_bytes.strip(b" \t")
        if any(
            byte < 32 and byte != 9 or byte == 127
            for byte in value_bytes
        ):
            raise HTTPError(400, "control character in field value")

        name = name_bytes.decode("ascii").lower()
        value = value_bytes.decode("latin-1")
        headers.setdefault(name, []).append(value)

    hosts = headers.get("host", [])
    if len(hosts) != 1 or not valid_host(hosts[0]):
        raise HTTPError(400, "need exactly one valid Host")

    content_lengths = headers.get("content-length", [])
    transfer_encodings = headers.get("transfer-encoding", [])
    if transfer_encodings and content_lengths:
        raise HTTPError(400, "ambiguous message framing")
    if transfer_encodings:
        raise HTTPError(501, "Transfer-Encoding is not supported")
    if len(content_lengths) > 1:
        raise HTTPError(400, "duplicate Content-Length")

    body_length = 0
    if content_lengths:
        raw_length = content_lengths[0]
        if not raw_length.isascii() or not raw_length.isdecimal():
            raise HTTPError(400, "invalid Content-Length")
        digits = raw_length.lstrip("0") or "0"
        limit = str(MAX_BODY)
        if len(digits) > len(limit) or (len(digits) == len(limit) and digits > limit):
            raise HTTPError(413, "request content too large")
        body_length = int(digits)

    if "expect" in headers:
        raise HTTPError(417, "Expect is not supported")

    body = receive_exact(
        connection,
        buffered_after_head,
        body_length,
    )
    return Request(method, target, version, headers, body)

REASONS = {
    200: "OK",
    400: "Bad Request",
    404: "Not Found",
    405: "Method Not Allowed",
    408: "Request Timeout",
    413: "Content Too Large",
    417: "Expectation Failed",
    431: "Request Header Fields Too Large",
    500: "Internal Server Error",
    501: "Not Implemented",
}

def encode_response(status, body=b"", headers=None, head_only=False):
    fields = {
        "Date": formatdate(usegmt=True),
        "Content-Type": "text/plain; charset=utf-8",
        "Connection": "close",
    }
    if headers:
        fields.update(headers)
    fields["Content-Length"] = str(len(body))

    start = f"HTTP/1.1 {status} {REASONS[status]}\r\n"
    field_lines = "".join(
        f"{name}: {value}\r\n"
        for name, value in fields.items()
    )
    head = (start + field_lines + "\r\n").encode("ascii")
    return head if head_only else head + body

def application(request):
    path = request.target.partition("?")[0]

    if path == "/health":
        if request.method in {"GET", "HEAD"}:
            return 200, b"healthy\n", {}
        return 405, b"method not allowed\n", {
            "Allow": "GET, HEAD",
        }

    if path == "/echo":
        if request.method == "POST":
            return 200, request.body, {
                "Content-Type": "application/octet-stream",
            }
        return 405, b"method not allowed\n", {
            "Allow": "POST",
        }

    return 404, b"not found\n", {}

def handle_connection(connection, read_budget=5):
    reader = DeadlineReader(connection, time.monotonic() + read_budget)
    try:
        request = parse_request(reader)
        status, body, headers = application(request)
    except TimeoutError:
        status, body, headers = 408, b"request timeout\n", {}
    except HTTPError as error:
        status = error.status
        body, headers = (error.message + "\n").encode("utf-8"), {}
    except OSError:
        return
    except Exception:
        logging.exception("request handling failed")
        status, body, headers = 500, b"internal error\n", {}

    response = encode_response(
        status, body, headers, head_only=reader.method == "HEAD",
    )
    try:
        connection.settimeout(5)
        connection.sendall(response)
    except OSError:
        # The connection owner closes it; this client cannot stop the listener.
        return

def serve(host="127.0.0.1", port=8080):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener:
        listener.setsockopt(
            socket.SOL_SOCKET,
            socket.SO_REUSEADDR,
            1,
        )
        listener.bind((host, port))
        listener.listen(128)
        print(f"listening on http://{host}:{port}")

        while True:
            connection, peer = listener.accept()
            with connection:
                connection.settimeout(5)
                handle_connection(connection)
                print("served", peer)

if __name__ == "__main__":
    try:
        serve()
    except KeyboardInterrupt:
        print("\nstopped")

To start:

python3 mini_http11.py

Verify in another terminal:

curl -i http://127.0.0.1:8080/health
curl -i -X POST --data-binary 'hello world' \
  http://127.0.0.1:8080/echo
curl -i -X DELETE http://127.0.0.1:8080/health

Expected responses: 200, 200, and 405; the 405 response includes Allow: GET, HEAD.

Also run curl -I http://127.0.0.1:8080/health: HEAD advertises Content-Length: 8 but sends no healthy\n content. Once a request is recognized as HEAD, later parse errors also omit the error body. Every Expect is rejected with 417, avoiding a client waiting for 100 Continue while the server waits for content.

6. Key Decisions in a Parser

Why header values are decoded using latin-1

Latin-1 is used to map bytes directly to code points, avoiding silent UTF-8 substitution that could alter the original octets. This does not imply that all field values have ISO-8859-1 semantics. Before using any field value, the parser must still follow the ABNF specification or the library API for that field to ensure correct interpretation.

Why duplicate Content-Length is rejected

RFC 9112 allows normalization of repeated Content-Length values only when they are valid and identical, but security-sensitive parsers can safely reject such cases. The teaching server does not serve as a compatibility proxy and prioritizes determinism over leniency.

Length handling strips leading zeros, checks significant digits against the limit, and only then converts to int. Thousands of digits yield 413 rather than hitting Python’s integer-conversion limit and becoming 500. Long leading-zero forms remain valid within the 16 KiB header limit.

Why extra bytes in the buffer are not processed

receive_header_section may encounter bytes that come after the body of a message. The current server processes only one request per connection and closes the connection after sending a response, so it does not treat these extra bytes as part of another message. To support persistent connections, the parser would need to return the remaining bytes and continue parsing in the next round; discarding them would lose stream state.

Why all responses use Connection: close

The DeadlineReader shares one absolute read deadline; a successful short read does not restart it. Sending has a separate 5-second timeout. This single-threaded server still handles one connection at a time. The deadline excludes time in the accept queue and does not bound resources held by all queued clients.

HTTP/1.1 supports persistent connections by default, but "support" means the server must correctly parse multiple requests in sequence, send responses in order, set idle timeouts, and decide safely whether to reuse the connection after an error. This experiment explicitly closes the connection to simplify the state machine and avoid misleading users into thinking persistent connections are supported when they are not.

7. Testing Boundary Conditions with Fragmented Input

Small client writes alone do not guarantee short reads. A ReadCap adapter therefore limits each server read to 3 bytes, forcing split CRLF and length-field paths. On typical Unix systems socketpair uses no TCP port; implementations elsewhere can differ. This test uses no external network. Save it as test_mini_http11.py beside the server file.

import socket
import threading

from mini_http11 import handle_connection

class ReadCap:
    def __init__(self, connection):
        self.connection = connection

    def recv(self, length):
        return self.connection.recv(min(length, 3))

    def settimeout(self, seconds):
        self.connection.settimeout(seconds)

    def sendall(self, data):
        self.connection.sendall(data)

def exchange(request):
    client, server = socket.socketpair()
    client.settimeout(3)

    def serve_once():
        with server:
            server.settimeout(2)
            handle_connection(ReadCap(server), read_budget=2)

    worker = threading.Thread(target=serve_once)
    worker.start()

    with client:
        for offset in range(0, len(request), 3):
            client.sendall(request[offset:offset + 3])
        client.shutdown(socket.SHUT_WR)

        chunks = []
        while True:
            chunk = client.recv(4096)
            if chunk == b"":
                break
            chunks.append(chunk)

    worker.join(timeout=2)
    assert not worker.is_alive()
    return b"".join(chunks)

def body_of(response):
    return response.split(b"\r\n\r\n", 1)[1]

echo = exchange(
    b"POST /echo HTTP/1.1\r\n"
    b"Host: atlas.example\r\n"
    b"Content-Length: 11\r\n"
    b"\r\n"
    b"hello world"
)
assert echo.startswith(b"HTTP/1.1 200 OK\r\n")
assert body_of(echo) == b"hello world"

missing_host = exchange(
    b"GET /health HTTP/1.1\r\n\r\n"
)
assert missing_host.startswith(b"HTTP/1.1 400 Bad Request\r\n")

ambiguous = exchange(
    b"POST /echo HTTP/1.1\r\n"
    b"Host: atlas.example\r\n"
    b"Content-Length: 4\r\n"
    b"Transfer-Encoding: chunked\r\n"
    b"\r\n"
)
assert ambiguous.startswith(b"HTTP/1.1 400 Bad Request\r\n")

too_large = exchange(
    b"POST /echo HTTP/1.1\r\n"
    b"Host: atlas.example\r\n"
    b"Content-Length: 1048577\r\n"
    b"\r\n"
)
assert too_large.startswith(
    b"HTTP/1.1 413 Content Too Large\r\n"
)

head = exchange(
    b"HEAD /health HTTP/1.1\r\n"
    b"Host: atlas.example\r\n"
    b"\r\n"
)
assert b"Content-Length: 8\r\n" in head
assert body_of(head) == b""

print("HTTP/1.1 fragmented-input tests passed")

Run:

python3 test_mini_http11.py

Expected output:

HTTP/1.1 fragmented-input tests passed

These cases check reassembly with a 3-byte read cap, missing Host, conflicting framing, the body limit, and a valid HEAD request. They neither exhaust every partition or RFC grammar rule nor directly prove TCP recovery, deadlines, resets, or concurrency safety. Those require additional tests.

8. From Teaching Server to Production Server

• persistent connections, pipelining, and complete remainder buffers;
• chunked transfer encoding and trailers;
• request-targets in absolute-form, authority-form, and asterisk-form;
• TLS, ALPN, and HTTP/2;
• concurrency, queuing, backpressure, and graceful shutdown;
• access logging, request ID, metrics, and tracing;
• routing, media type negotiation, authentication, and authorization;
• header timeouts, body deadlines, rate limiting, and global resource budgets;
• proxy trust boundaries, such as which forwarded fields are trustworthy;
• fuzzing, differential testing, and request smuggling regression suites.

These are not features that can be safely added with just "a few more if statements." Real-world services should prioritize using well-maintained HTTP libraries and servers. Hand-rolled implementations are best suited for learning protocol boundaries and conducting isolated experiments.

9. Common Errors

"Not receiving data means the request has ended." A blocking socket waits; a timeout does not imply message boundary has been reached.

"Content-Length is only checked for POST requests." Request framing is independent of HTTP method semantics.

"If two identical Content-Length values appear, just pick the first one." Parser disagreement creates a security boundary, inputs must be handled consistently or rejected entirely.

"Limiting body size prevents resource exhaustion." You must also budget for headers, connection count, read duration, response queue depth, and parser CPU usage.

"After the server returns a 400, the connection can still be reused." Following a framing error, the buffer boundary may no longer be trustworthy; this experiment always closes the connection.

"My own server works with curl, so it's ready for production." The normal curl path does not cover malformed inputs, slow clients, proxy differences, or concurrent lifecycle scenarios.

10. Exercises and Acceptance Criteria

1. Split each header field into 1 byte at a time and verify that the response remains unchanged.
2. Add tests for repeated Host headers, field names containing whitespace, and negative Content-Length values, all of which should result in a 400 error.
3. Add an allowlist for POST /echo using Content-Type; if unsupported, return a 415 error.
4. Design a persistent connection state machine, clearly identifying the remainder buffer, idle timeout conditions, and when the connection should be closed.
5. Read the chunked decoding algorithm from RFC 9112 and list the additional constraints required before implementing trailer support.

Acceptance criteria: You must be able to explain why the parser continues or stops reading at each step, identify all memory and time limits, and correctly distinguish between a single recv, EOF, or timeout as a message delimiter.

The split POST /echo still yields the same body through CRLF and Content-Length parsing. Conflicting framing receives an error followed by closure. Explaining those boundaries demonstrates more than observing a single 200 response.

The next chapter moves into HTTPS and TLS. HTTP now has semantics and boundaries; the next step is to verify who the server on the other end is, and to ensure that any observers along the way cannot read or alter the content.

References

• RFC 9110: HTTP Semantics
• RFC 9112: HTTP/1.1
• RFC 9205: Building Protocols with HTTP