4.2 HTTP/1.1 Framing and the Minimal Server
The Beacon Tower first opens its HTTP receiver, and the first messenger delivers only half a line of POST /echo HT, pausing before sending the rest of the header. The next messenger bundles two requests into the same byte segment. If the administrator still follows the rule that "one recv equals one message," the entrance will quickly become chaotic.
HTTP/1.1 defines message boundaries on a TCP byte stream. A parser must incrementally read the start-line, field section, and content, while strictly enforcing time and memory limits. If proxies or origin servers interpret these boundaries differently, they can create openings for request smuggling.
This lesson first establishes a clear contract for message parsing, then implements a minimal server that supports only HTTP/1.1, Content-Length, and one request per connection. The feature set is intentionally narrow, with each boundary explicitly defined and discussed.
1. A Single recv Has No Protocol Significance
The following approach is invalid:
request = connection.recv(65536)
method, target, version = request.split(b"\r\n", 1)[0].split()recv returns "how many bytes are currently available," not "the next HTTP message":
- The start-line or
\r\n\r\nmay be split across multiple recv calls; - The body may not have arrived in full;
- Headers and body may arrive in a single chunk;
- On persistent connections, the next request might already be in the buffer;
- The peer might send only a partial message and then pause;
- EOF could occur before the declared message length is reached.
Therefore, the parser must maintain a buffer that persists across recv calls, along with a clear read state.
2. The Boundary of an HTTP/1.1 Request
For a standard request, first locate:
request-line CRLF
field-line CRLF
...
CRLFThe final empty line marks the end of the header section. Whether a message body exists is not determined by heuristic assumptions such as "POST requests usually have a body," but rather by the framing field.
In this lesson, we accept:
Content-Length: 11We do not implement:
Transfer-Encoding: chunkedWhen a feature is unsupported, we must explicitly reject the request. We cannot fall back to "keep reading from the socket until no data is available." On persistent connections, this would simultaneously break message boundary detection and connection liveness.
3. Parser Sets Resource Limits
Before writing code, establish the contract:
| Item | Teaching Server Policy |
|---|---|
| HTTP version | Accepts only HTTP/1.1 |
| request-target | Accepts only origin-form, i.e., begins with / |
| header section | Maximum 16 KiB, including the trailing empty line |
| body | Maximum 1 MiB |
| Host header | Must appear exactly once and be non-empty |
| Content-Length | At most one occurrence; must be a non-negative decimal integer |
| Transfer-Encoding | Not supported; if present alongside Content-Length, returns 400 |
| timeout | Each accepted connection is assigned a read timeout |
| connection reuse | After response, sends Connection: close |
These numbers are configuration values for this experiment, not global constants of HTTP. In production servers, limits should be chosen based on business requirements, proxy chain behavior, and resource budgets, and should distinguish between oversized headers, oversized content, and read timeouts.
4. Why Flexible Framing Is Dangerous
Suppose the frontend proxy splits a request according to Content-Length, while the backend prioritizes splitting according to Transfer-Encoding. The same byte stream could be partitioned into different messages at each end, allowing an attacker to smuggle hidden requests into the backend.
RFC 9112 establishes strict rules for framing conflicts. This experiment adopts a narrower, more conservative approach:
- If both
Transfer-EncodingandContent-Lengthappear: return 400 and terminate; - If any
Transfer-Encodingis detected: return 501 and terminate; - If
Content-Lengthis repeated: return 400 and terminate; - If the header name, whitespace, or control characters are invalid: return 400 and terminate.
The teaching code is not part of the proxy chain, yet it still avoids lenient, speculative parsing. Every hop in the message path must agree on the same message boundaries.
5. A Runnable Minimum Server
Save the following as mini_http11.py, using Python 3.10+. It supports:
GET /health;HEAD /health;POST /echo;- 404, 405, and several parser errors;
- reassembly of header/body from arbitrary TCP segments.
import re
import socket
from dataclasses import dataclass
from urllib.parse import urlsplit
MAX_HEADER_SECTION = 16 * 1024
MAX_BODY = 1024 * 1024
TOKEN = re.compile(rb"^[!#$%&'*+\-.^_\x60|~0-9A-Za-z]+$")
class HTTPError(Exception):
def __init__(self, status, message):
super().__init__(message)
self.status = status
self.message = message
@dataclass(frozen=True)
class Request:
method: str
target: str
version: str
headers: dict[str, list[str]]
body: bytes
def receive_header_section(connection):
buffer = bytearray()
marker = b"\r\n\r\n"
while marker not in buffer:
if len(buffer) >= MAX_HEADER_SECTION:
raise HTTPError(431, "header section too large")
chunk = connection.recv(
min(4096, MAX_HEADER_SECTION - len(buffer))
)
if chunk == b"":
raise HTTPError(400, "EOF before header section")
buffer.extend(chunk)
marker_at = buffer.find(marker)
return bytes(buffer[:marker_at]), bytes(buffer[marker_at + 4:])
def receive_exact(connection, initial, length):
body = bytearray(initial[:length])
while len(body) < length:
chunk = connection.recv(min(4096, length - len(body)))
if chunk == b"":
raise HTTPError(400, "EOF before complete request content")
body.extend(chunk)
return bytes(body)
def parse_request(connection):
head, buffered_after_head = receive_header_section(connection)
lines = head.split(b"\r\n")
if not lines or not lines[0]:
raise HTTPError(400, "missing request line")
request_parts = lines[0].split(b" ")
if len(request_parts) != 3 or any(not part for part in request_parts):
raise HTTPError(400, "malformed request line")
method_bytes, target_bytes, version_bytes = request_parts
if not TOKEN.fullmatch(method_bytes):
raise HTTPError(400, "invalid method")
try:
method = method_bytes.decode("ascii")
target = target_bytes.decode("ascii")
version = version_bytes.decode("ascii")
except UnicodeDecodeError as error:
raise HTTPError(400, "non-ASCII request line") from error
if version != "HTTP/1.1":
raise HTTPError(400, "only HTTP/1.1 is supported")
if not target.startswith("/") or " " in target:
raise HTTPError(400, "only origin-form targets are supported")
headers: dict[str, list[str]] = {}
for raw_line in lines[1:]:
if not raw_line:
raise HTTPError(400, "unexpected empty field line")
if raw_line[:1] in (b" ", b"\t"):
raise HTTPError(400, "obsolete folded field")
if b":" not in raw_line:
raise HTTPError(400, "field without colon")
name_bytes, value_bytes = raw_line.split(b":", 1)
if not TOKEN.fullmatch(name_bytes):
raise HTTPError(400, "invalid field name")
value_bytes = value_bytes.strip(b" \t")
if any(
byte < 32 and byte != 9 or byte == 127
for byte in value_bytes
):
raise HTTPError(400, "control character in field value")
name = name_bytes.decode("ascii").lower()
value = value_bytes.decode("latin-1")
headers.setdefault(name, []).append(value)
hosts = headers.get("host", [])
if len(hosts) != 1 or not hosts[0]:
raise HTTPError(400, "Host must appear exactly once")
content_lengths = headers.get("content-length", [])
transfer_encodings = headers.get("transfer-encoding", [])
if transfer_encodings and content_lengths:
raise HTTPError(400, "ambiguous message framing")
if transfer_encodings:
raise HTTPError(501, "Transfer-Encoding is not supported")
if len(content_lengths) > 1:
raise HTTPError(400, "duplicate Content-Length")
body_length = 0
if content_lengths:
raw_length = content_lengths[0]
if not raw_length.isascii() or not raw_length.isdecimal():
raise HTTPError(400, "invalid Content-Length")
body_length = int(raw_length)
if body_length > MAX_BODY:
raise HTTPError(413, "request content too large")
body = receive_exact(
connection,
buffered_after_head,
body_length,
)
return Request(method, target, version, headers, body)
REASONS = {
200: "OK",
400: "Bad Request",
404: "Not Found",
405: "Method Not Allowed",
408: "Request Timeout",
413: "Content Too Large",
431: "Request Header Fields Too Large",
500: "Internal Server Error",
501: "Not Implemented",
}
def encode_response(status, body=b"", headers=None, head_only=False):
fields = {
"Content-Type": "text/plain; charset=utf-8",
"Connection": "close",
}
if headers:
fields.update(headers)
fields["Content-Length"] = str(len(body))
start = f"HTTP/1.1 {status} {REASONS[status]}\r\n"
field_lines = "".join(
f"{name}: {value}\r\n"
for name, value in fields.items()
)
head = (start + field_lines + "\r\n").encode("ascii")
return head if head_only else head + body
def application(request):
path = urlsplit(request.target).path
if path == "/health":
if request.method in {"GET", "HEAD"}:
return 200, b"healthy\n", {}
return 405, b"method not allowed\n", {
"Allow": "GET, HEAD",
}
if path == "/echo":
if request.method == "POST":
return 200, request.body, {
"Content-Type": "application/octet-stream",
}
return 405, b"method not allowed\n", {
"Allow": "POST",
}
return 404, b"not found\n", {}
def handle_connection(connection):
try:
request = parse_request(connection)
status, body, headers = application(request)
response = encode_response(
status,
body,
headers,
head_only=request.method == "HEAD",
)
except socket.timeout:
response = encode_response(408, b"request timeout\n")
except HTTPError as error:
response = encode_response(
error.status,
(error.message + "\n").encode("utf-8"),
)
except Exception:
response = encode_response(500, b"internal error\n")
connection.sendall(response)
def serve(host="127.0.0.1", port=8080):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener:
listener.setsockopt(
socket.SOL_SOCKET,
socket.SO_REUSEADDR,
1,
)
listener.bind((host, port))
listener.listen(128)
print(f"listening on http://{host}:{port}")
while True:
connection, peer = listener.accept()
with connection:
connection.settimeout(5)
handle_connection(connection)
print("served", peer)
if __name__ == "__main__":
try:
serve()
except KeyboardInterrupt:
print("\nstopped")To start:
python3 mini_http11.pyVerify in another terminal:
curl -i http://127.0.0.1:8080/health
curl -i -X POST --data-binary 'hello world' \
http://127.0.0.1:8080/echo
curl -i -X DELETE http://127.0.0.1:8080/healthExpected responses: 200, 200, and 405; the 405 response includes Allow: GET, HEAD.
6. Key Decisions in a Parser
Why header values are decoded using latin-1
Latin-1 is used to map bytes directly to code points, avoiding silent UTF-8 substitution that could alter the original octets. This does not imply that all field values have ISO-8859-1 semantics. Before using any field value, the parser must still follow the ABNF specification or the library API for that field to ensure correct interpretation.
Why duplicate Content-Length is rejected
RFC 9112 allows for normalization of duplicate Content-Length headers, but security-sensitive parsers can safely reject such cases. The teaching server does not serve as a compatibility proxy and prioritizes determinism over leniency.
Why extra bytes in the buffer are not processed
receive_header_section may encounter bytes that come after the body of a message. The current server processes only one request per connection and closes the connection after sending a response, so it does not treat these extra bytes as part of another message. To support persistent connections, the parser would need to return the remaining bytes and continue parsing in the next round, discarding them would be unsafe.
Why all responses use Connection: close
HTTP/1.1 supports persistent connections by default, but "support" means the server must correctly parse multiple requests in sequence, send responses in order, set idle timeouts, and decide safely whether to reuse the connection after an error. This experiment explicitly closes the connection to simplify the state machine and avoid misleading users into thinking persistent connections are supported when they are not.
7. Testing Boundary Conditions with Fragmented Input
The messenger in the story now delivers only three bytes at a time. The following test uses socketpair, which doesn't consume ports or rely on external networks. Save it as test_mini_http11.py, and place it in the same directory as the server file.
import socket
import threading
from mini_http11 import handle_connection
def exchange(request):
client, server = socket.socketpair()
def serve_once():
with server:
server.settimeout(2)
handle_connection(server)
worker = threading.Thread(target=serve_once)
worker.start()
with client:
for offset in range(0, len(request), 3):
client.sendall(request[offset:offset + 3])
client.shutdown(socket.SHUT_WR)
chunks = []
while True:
chunk = client.recv(4096)
if chunk == b"":
break
chunks.append(chunk)
worker.join(timeout=2)
assert not worker.is_alive()
return b"".join(chunks)
def body_of(response):
return response.split(b"\r\n\r\n", 1)[1]
echo = exchange(
b"POST /echo HTTP/1.1\r\n"
b"Host: atlas.example\r\n"
b"Content-Length: 11\r\n"
b"\r\n"
b"hello world"
)
assert echo.startswith(b"HTTP/1.1 200 OK\r\n")
assert body_of(echo) == b"hello world"
missing_host = exchange(
b"GET /health HTTP/1.1\r\n\r\n"
)
assert missing_host.startswith(b"HTTP/1.1 400 Bad Request\r\n")
ambiguous = exchange(
b"POST /echo HTTP/1.1\r\n"
b"Host: atlas.example\r\n"
b"Content-Length: 4\r\n"
b"Transfer-Encoding: chunked\r\n"
b"\r\n"
)
assert ambiguous.startswith(b"HTTP/1.1 400 Bad Request\r\n")
too_large = exchange(
b"POST /echo HTTP/1.1\r\n"
b"Host: atlas.example\r\n"
b"Content-Length: 1048577\r\n"
b"\r\n"
)
assert too_large.startswith(
b"HTTP/1.1 413 Content Too Large\r\n"
)
head = exchange(
b"HEAD /health HTTP/1.1\r\n"
b"Host: atlas.example\r\n"
b"\r\n"
)
assert b"Content-Length: 8\r\n" in head
assert body_of(head) == b""
print("HTTP/1.1 fragmented-input tests passed")Run:
python3 test_mini_http11.pyExpected output:
HTTP/1.1 fragmented-input tests passedThis test still does not prove that the parser covers all RFC syntax. It only verifies the subset of behavior promised in this document: arbitrary TCP splitting, Host header, explicit body length, conflicting framing, size limits, and HEAD requests.
8. From Teaching Server to Production Server
The beacon tower can now reliably parse a supported request, but it's still far from being ready for public service. A production-grade implementation must at least handle:
- persistent connections, pipelining, and complete remainder buffers;
- chunked transfer encoding and trailers;
- request-targets in absolute-form, authority-form, and asterisk-form;
- TLS, ALPN, and HTTP/2;
- concurrency, queuing, backpressure, and graceful shutdown;
- access logging, request ID, metrics, and tracing;
- routing, media type negotiation, authentication, and authorization;
- header timeouts, body deadlines, rate limiting, and global resource budgets;
- proxy trust boundaries, such as which forwarded fields are trustworthy;
- fuzzing, differential testing, and request smuggling regression suites.
These are not features that can be safely added with just "a few more if statements." Real-world services should prioritize using well-maintained HTTP libraries and servers. Hand-rolled implementations are best suited for learning protocol boundaries and conducting isolated experiments.
9. Common Errors
"Not receiving data means the request has ended." A blocking socket waits; a timeout does not imply message boundary has been reached.
"Content-Length is only checked for POST requests." Request framing is independent of HTTP method semantics.
"If two identical Content-Length values appear, just pick the first one." Parser disagreement creates a security boundary, inputs must be handled consistently or rejected entirely.
"Limiting body size prevents resource exhaustion." You must also budget for headers, connection count, read duration, response queue depth, and parser CPU usage.
"After the server returns a 400, the connection can still be reused." Following a framing error, the buffer boundary may no longer be trustworthy; this experiment always closes the connection.
"My own server works with curl, so it's ready for production." The normal curl path does not cover malformed inputs, slow clients, proxy differences, or concurrent lifecycle scenarios.
10. Exercises and Acceptance Criteria
- Split each header field into 1 byte at a time and verify that the response remains unchanged.
- Add tests for repeated Host headers, field names containing whitespace, and negative Content-Length values, all of which should result in a 400 error.
- Add an allowlist for
POST /echousingContent-Type; if unsupported, return a 415 error. - Design a persistent connection state machine, clearly identifying the remainder buffer, idle timeout conditions, and when the connection should be closed.
- Read the chunked decoding algorithm from RFC 9112 and list the additional constraints required before implementing trailer support.
Acceptance criteria: You must be able to explain why the parser continues or stops reading at each step, identify all memory and time limits, and correctly distinguish between a single recv, EOF, or timeout as a message delimiter.
11. The Beacon Tower Begins Operations
Fragmented testing splits a request into disjointed pieces, yet the server still reassembles it into a single POST /echo using CRLF and Content-Length. Framing conflicts are rejected at the entrance, preventing different receivers from independently guessing the structure.
The next chapter moves into HTTPS and TLS. HTTP now has semantics and boundaries; the next step is to verify who the beacon tower on the other end is, and to ensure that any observers along the way cannot read or alter the content.