8.3 AOT, JIT, and Anti-Optimization: Betting on Runtime Information; And Knowing When to Revert

The final machine in the tower didn't compile all code into the highest optimization level at once. Cold code is first interpreted or rapidly compiled; only after hot-spot patterns accumulate does the system perform optimization. If assumptions about types or call targets prove invalid, the machine must revert to a safer, conservative execution version.

Learning Objectives

• Distinguish between AOT, JIT, tiered compilation, and PGO;
• Understand speculative optimization, guards, and deoptimization;
• Explain the engineering costs of OSR, safepoints, and code cache;
• Use differential, random, and IR-based verification to validate compiler correctness.

1. AOT and JIT Have Different Timing of Usage

AOT generates target code before deployment or runtime. Its advantages typically include:

• Predictable startup behavior;
• No need for a full compiler at runtime;
• Greater control over executable memory and deployment artifacts;
• Opportunity to perform extensive whole-program or link-time optimizations.

JIT compiles code during runtime, enabling observation of:

• Function and loop hotness;
• Actual receiver types;
• Branch probabilities;
• Target of calls and object shapes;
• Current hardware capabilities.

JIT incurs costs such as warmup time, compilation threads, profiling memory, code cache usage, executable memory safety, and performance fluctuations. It is incorrect to broadly claim that JIT outperforms AOT.

2. Layered Compilation Control Startup and Peak

A typical but not exclusive hierarchy:

Interpreter or bytecode execution
  ↓ Hotspot threshold
Fast baseline compilation
  ↓ More profiling data
Optimized compilation

Setting the threshold too low results in compiling code that runs only a few times, while a threshold that's too high misses the performance gains from hot code. The system must also decide when to reclaim invalidated code, whether multiple versions of a compiled unit can coexist, and how much CPU budget to allocate to compilation threads.

AOT can also leverage profile-guided optimization: collect profiling data from representative workloads first, then feed that back into the next build. The boundary between static and dynamic compilation is not an absolute divide based solely on whether profiling data is available.

3. Speculative Optimization Must Have Guards

If a call site has historically seen only the type Point, the JIT can temporarily specialize dynamic dispatch into a direct call to Point and inline it, but only after inserting a guard:

if receiver.shape != PointShape:
    deopt
fast_path_for_Point

A failed guard does not return an error result. Instead, it redirects execution to the generic version or triggers a recompilation. The more aggressive the speculation, the shorter the fast path becomes, but guard overhead, code bloat, and failure frequency may also increase.

Profiles are historical samples, not semantic proofs. A profile assumption without guards turns performance predictions into correctness bugs.

4. Deoptimization: Rebuilding Abstract State

Optimizing code may:

• Fold multiple source variables into a single register;
• Eliminate unused object allocations;
• Inline deeply nested function calls;
• Store values in constants or recomputed expressions.

During deoptimization, the interpreter or low-level runtime must reconstruct the stack frames, local variables, and objects required by the original abstract semantics. At each potential exit point, a state map must be preserved: indicating whether a given abstract value currently resides in a register, a stack slot, a constant, or a materialized virtual object.

This is precisely why optimization cannot produce only machine code, debugging, garbage collection, exception handling, and reoptimization all require a mapping between source code and runtime state.

5. OSR Lets a Running Loop Switch Versions

If a function enters a long-running loop, it makes no sense to apply an optimized version only after the function returns. On-Stack Replacement (OSR) enables safely migrating the current execution state into optimized code at a secure entry point within the loop.

The reverse can also happen: when optimization assumptions fail, the system can revert back to the generic version during execution. OSR mappings must handle live locals, stack values, exception states, and virtual objects. Errors typically manifest only under specific hot paths and at particular iteration points.

6. Safepoint Connection Between Compiler and Runtime

Garbage collection, thread suspension, and deoptimization can only be safely executed at safepoints, where the runtime has precise visibility into the stack and register state. The compiler generates at these points:

• Which locations contain object references;
• The inlined call stack;
• Recoverable local state;
• Return addresses and exception handling information.

Too few safepoints increase pause time due to waiting, while too many increase overhead from checks and metadata processing. Unbounded pure computation loops may require explicit polling to ensure the runtime eventually regains control.

7. Code Cache and W^X

JIT compilers must write new code and execute it. Secure implementations typically follow the write XOR execute principle: memory pages are not simultaneously writable and executable, with permissions switched after code generation, or using platform-specific mechanisms like dual mapping.

Additional concerns include:

• Synchronizing instruction cache updates;
• Relocating code addresses;
• Managing code cache capacity and eviction policies;
• Ensuring that concurrent threads are no longer executing the old code;
• Verifying the control flow integrity of generated code and maintaining sandbox boundaries.

JIT compilation expands the attack surface, especially when intermediate representations or bytecode originate from untrusted sources. Input validation, resource limits, and isolation of executable memory are essential components of runtime design.

8. Compiler Correctness Requires Multi-Layer Evidence

IR Verifier

After each pass, verify structural integrity, type consistency, SSA form, and control flow graph invariants.

Differential Testing

Run the same program through the interpreter, unoptimized compiler, and optimized compiler, then compare observable behaviors. Even reference implementations can contain bugs, so differences must be attributed, automatic failure detection cannot be trusted to identify errors in new versions.

Random and Mutation Testing

Generate small programs with well-defined behavior, or apply semantically preserving transformations to existing programs. If inputs contain undefined behavior, different outputs may all be valid, leading to false positives.

Translation Validation

For each specific transformation or entire function, attempt to prove equivalence between the input and output IRs, or that the transformation satisfies a refinement relation. It does not require proving that the optimizer is always correct, but rather validates the correctness of the current output.

Regression and Performance Testing

Individual gates must be enforced for correctness, compilation time, code size, peak performance, warm-up behavior, and memory usage. Counting assembly lines alone is insufficient to prove faster performance.

9. Top-Level Convergence

The compiler pipeline is now complete:

Characters
→ Tokens
→ Syntax Tree
→ Names and Types
→ IR / CFG / SSA
→ Optimization
→ Target Instructions and Registers
→ Executable Code

Real compilers iterate back and forth, retain multiple layers of IR, and collaborate with linkers, runtimes, debuggers, and garbage collectors. This pipeline is not a one-time translation but a sequence of semantic transformations preserving invariants.

Common Misconceptions

• JIT compiles only hot code, so cold code has no cost: Even cold code consumes resources, interpretation, profiling, and baseline execution all require system overhead.
• Profiled types can be assumed to be permanent: Any inference must be paired with guards and explicit exit paths to ensure correctness.
• Deoptimization simply means falling back to interpretation: In reality, the system must precisely reconstruct the abstract state that was previously optimized away.
• Output differing from a reference means it's wrong: Differences may stem from undefined behavior, non-determinism, or actual reference bugs, these must be ruled out before concluding failure.

Exercise

1. Design shape guard and deopt paths for monomorphic call inlining.
2. List the information a state map for an inlined two-layer function must store.
3. Design a benchmark that measures both warmup and steady-state throughput to avoid reporting only the best single run.
4. Design a verifier, differential, and random test suite for a constant folding pass.

Summary

The difference between AOT and JIT lies in when information about availability and cost is made available. JIT places bets on runtime behavior using guards, and pulls back when those assumptions fail via deoptimization; OSR, safepoints, and the code cache implement this strategy in actual runtime execution. Compiler correctness ultimately depends on clear semantics, local validation, and layered testing working together to maintain integrity.

When you step out of the spell tower, the head curator doesn’t say, “You now understand all compilers.” Instead, he hands you a new roadmap: the next hall in the data prophecy wing will deal with a different kind of input, not strictly valid programs, but data that is missing, skewed, noisy, and constantly changing.