5.1 TLS Handshake, Certificate Chains, and Security Boundaries
TLS establishes a secure channel that must answer three questions:
- Can an observer on the path read the content?
- Can the receiver detect modification?
- Is the peer the identity the client intended to reach?
TLS answers these with confidentiality, integrity, and peer authentication. HTTPS is typically HTTP over TLS. “SSL” is a historical name retained in many APIs; modern deployments should not enable SSLv2 or SSLv3.
1. TLS Position in the Protocol Stack
For HTTP/1.1 and HTTP/2, the typical flow is:
HTTP semantics
↓
HTTP/1.1 message framing or HTTP/2 frames
↓
TLS records
↓
TCP byte streamTLS isn't a fixed layer in the OSI model, practically speaking; it's best understood as a security layer between application protocols and the transport layer. HTTP/3 takes a different approach: QUIC integrates the TLS 1.3 handshake, and uses QUIC packet protection instead of the TLS record layer.
2. A TLS 1.3 Certificate Handshake
This is a full certificate-authenticated handshake without HelloRetryRequest, PSK resumption, or a client certificate. The ALPN selection appears in EncryptedExtensions; ServerHello does not contain every negotiated result.
Client Server
| |
| ClientHello |
| supported_versions, key_share, SNI, ALPN ------>|
| |
|<------ ServerHello, key_share |
| {EncryptedExtensions} |
| {Certificate} |
| {CertificateVerify} |
| {Finished} |
| |
| {Finished} ------------------------------------->|
| |
|<========== protected application data ==========>|The messages enclosed in braces are protected by the handshake traffic key. This is a key change in TLS 1.3: most of the handshake messages after ServerHello are no longer exposed in plaintext.
What’s in ClientHello
supported_versions: The TLS version the client actually supports;key_share: The ephemeral (EC)DHE public parameters;- SNI: The service name the client wishes to access, enabling a single IP address to select the correct certificate;
- ALPN: The application protocol the client wants to use (such as
h2orhttp/1.1) within the TLS session.
ServerHello selects the version, cipher suite, and key share. Both sides derive application keys from the ephemeral Diffie–Hellman shared secret and the handshake transcript; the ephemeral private key is not the same as the server’s certificate private key.
A small Diffie–Hellman calculation shows how both parties obtain the same shared value:
p, g = 23, 5
alice_private, bob_private = 6, 15
alice_public = pow(g, alice_private, p)
bob_public = pow(g, bob_private, p)
alice_shared = pow(bob_public, alice_private, p)
bob_shared = pow(alice_public, bob_private, p)
assert alice_shared == bob_shared == 2
print(alice_public, bob_public, alice_shared) # 8 19 2The modulus 23 is small enough to solve by enumeration and must not be used for encryption. Real TLS uses standardized groups and derives separate keys with HKDF and the transcript. This arithmetic alone also provides no authentication: an intermediary could substitute public parameters.
Identity Verification and Finished
Certificate presents the server’s certificate chain for this connection. CertificateVerify signs the handshake context using the certificate private key, proving that the peer actually possesses the corresponding private key. Finished performs a key-based verification of the transcript, allowing both parties to detect whether the handshake has been tampered with.
A complete TLS 1.3 handshake typically establishes application traffic keys in just one round-trip time (1 RTT). This does not mean “loading a webpage takes just 1 RTT”: DNS resolution, TCP or QUIC connection setup, HTTP redirects, and resource loading can all add additional round trips.
An unsuitable key share can trigger HelloRetryRequest and another round trip. A resumed handshake may omit Certificate and CertificateVerify and authenticate using a PSK. The diagram also does not require the server to wait for client Finished before sending any application data; each endpoint follows its own handshake state.
3. Certificate path validation is not "just signed by the right entity"
local trust anchor
|
intermediate CA certificate
|
leaf certificate for service identityServers typically send the leaf certificate and any required intermediate certificates. A root certificate is usually unnecessary in the transmitted list; sending one does not establish trust. The final trust anchor is determined by the client’s trust store and policy; it’s not automatically trusted just because it’s self-signed.
Certificate path validation is only one part of certificate verification. An HTTPS client must also verify:
- The certificate’s validity period includes the current time;
- The subjectAltName field matches the expected service identity;
- The key usage or extended key usage permits its intended purpose;
- The signature algorithm, key size, and policy are acceptable;
- The certificate has not been revoked by local policy.
That’s why certificate validation should never be implemented directly in business logic. Instead, rely on a mature TLS library with secure defaults, and explicitly configure the trust store and expected service identity.
For example, a DNS SAN of *.example.test covers api.example.test, but not example.test or a.b.example.test. An IP literal requires a matching IP SAN; an IP-looking string in a DNS SAN is not equivalent. RFC 9525 no longer accepts subject Common Name as a replacement for SAN.
4. What Does a Cipher Suite Actually Negotiate?
A TLS 1.2 cipher suite name may include components for key exchange, authentication, bulk encryption, and a hash function. For example, an IANA-named suite:
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
| | | |
key ex auth AEAD PRF hashOpenSSL often displays it as ECDHE-RSA-AES128-GCM-SHA256. The two string formats differ, but they do not indicate different protocols.
In TLS 1.3, the cipher suite is simplified:
TLS_AES_128_GCM_SHA256It specifies only the AEAD cipher and the hash function used with HKDF. The key exchange group and signature algorithm are negotiated separately in other extensions. Therefore, you cannot determine from the TLS 1.3 cipher suite name whether the server certificate uses RSA or ECDSA.
5. Forward secrecy protects the past
In traditional static RSA key exchange, if an attacker records encrypted traffic and later obtains the server's private key, they could potentially decrypt past session secrets. Ephemeral (EC)DHE generates temporary key material for each handshake; the long-term private key is used only for authentication and is never directly used as the application traffic key.
This is known as forward secrecy. It assumes that ephemeral secrets are promptly destroyed and that they do not magically eliminate the risk posed when endpoints were already compromised. TLS 1.3's certificate-based full handshake uses ephemeral (EC)DHE and no longer supports TLS 1.2's static RSA key exchange.
That conclusion applies to handshakes using ephemeral (EC)DHE. TLS 1.3 also defines a PSK-only mode, so the version number alone does not give every mode identical forward-secrecy properties.
6. 0-RTT Is Not Free Acceleration
TLS 1.3 enables clients to send early data during session resumption, but early data is susceptible to replay attacks, and its security properties differ from standard application data.
Therefore, do not unconditionally place non-idempotent operations (such as payments, order creation, or reward distribution) into 0-RTT. The server must explicitly decide whether to accept early data, and the application layer must implement replay protection.
Early data also lacks the same forward-secrecy guarantee as ordinary post-handshake data. A GET endpoint with side effects, or one whose replay causes unacceptable additional business effects, cannot be admitted solely because a method is labeled safe or idempotent. HTTP services can use 425 Too Early under RFC 8470 to request a retry after the handshake.
7. What TLS Does Not Solve
- Certificates typically verify service identity, not the business permissions of logged-in users;
- When an endpoint is compromised, plaintext data can still be exfiltrated before or after encryption;
- TLS does not protect against attacks like SQL injection, cross-site scripting (XSS), cross-site request forgery (CSRF), or privilege escalation;
- Metadata such as the remote endpoint IP, traffic timing, and volume remains observable; SNI hiding depends on ECH deployment and is not guaranteed by default;
- TLS does not ensure service availability.
8. Common Misconceptions
"A lock icon means the website is trustworthy." It only indicates that the browser successfully established a protected connection with the service's identity; it does not mean the site's business practices are trustworthy.
"The server's root certificate proves its identity." Trust originates from the client's preconfigured trust anchors and policies, not from the root certificate presented by the server.
"TLS 1.3's AES suite implies the certificate uses RSA." In TLS 1.3, the AEAD cipher suite and signature algorithm are negotiated separately, so the cipher suite does not indicate the key exchange or signature algorithm used.
"Turning off certificate verification still encrypts the content, so it can go live." Without authentication, encrypted content may be transmitted over a perfectly secure channel that is actually controlled by an attacker.
9. Exercises and Verification
- In order, explain what each of the following in TLS 1.3 (
Certificate,CertificateVerify, andFinished) demonstrates. - Why does leaking the server private key not necessarily allow an attacker to decrypt previously recorded TLS 1.3 full-handshake traffic?
- In your own words, distinguish between "certificate path valid" and "service identity matches."
- List two operations that should not be unconditionally included in 0-RTT, and explain the common reason behind both.
A real failure may return only CERTIFICATE_VERIFY_FAILED, leaving a missing intermediate, a name mismatch, or a validity-time problem to distinguish. Next, 5.2 TLS Secure Client and Fault Diagnosis separates those causes with evidence.