8.3 AOT, JIT, and Deoptimization: Using Runtime Information and Recovering Safely

A tiered JIT usually does not compile every function at its highest optimization level immediately. Cold code runs in an interpreter or baseline tier; hot code is optimized after profiling. If assumptions about types or call targets fail, execution needs a version that still preserves the language semantics. Performance can be a bet; correctness cannot.

Learning Objectives

• Distinguish between AOT, JIT, tiered compilation, and PGO;
• Understand speculative optimization, guards, and deoptimization;
• Explain the engineering costs of OSR, safepoints, and code cache;
• Use differential, random, and IR-based verification to validate compiler correctness.

1. AOT and JIT Use Information at Different Times

AOT generates target code before deployment or runtime. Its advantages typically include:

• Predictable startup behavior;
• No need for a full compiler at runtime;
• Greater control over executable memory and deployment artifacts;
• Opportunity to perform extensive whole-program or link-time optimizations.

JIT compiles code during runtime, enabling observation of:

• Function and loop hotness;
• Actual receiver types;
• Branch probabilities;
• Target of calls and object shapes;
• Current hardware capabilities.

JIT incurs costs such as warmup time, compilation threads, profiling memory, code cache usage, executable memory safety, and performance fluctuations. It is incorrect to broadly claim that JIT outperforms AOT.

2. Tiered Compilation Balances Startup and Peak Performance

A typical but not exclusive hierarchy:

Interpreter or bytecode execution
  ↓ Hotspot threshold
Fast baseline compilation
  ↓ More profiling data
Optimized compilation

Setting the threshold too low results in compiling code that runs only a few times, while a threshold that's too high misses the performance gains from hot code. The system must also decide when to reclaim invalidated code, whether multiple versions of a compiled unit can coexist, and how much CPU budget to allocate to compilation threads.

AOT can also use profile-guided optimization: collect profiling data from representative workloads first, then feed that back into the next build. The boundary between static and dynamic compilation is not an absolute divide based solely on whether profiling data is available.

3. Speculative Optimization Needs Enforced Assumptions

Assume method definitions and prototype relationships remain valid, and null receivers have been handled according to the language. If a call site has only seen Point, the JIT may specialize and inline dispatch, checking the current input with a guard:

if receiver.shape != PointShape:
    deopt
fast_path_for_Point

A failed guard does not return an error result. Instead, it redirects execution to the generic version or triggers a recompilation. More aggressive speculation may reduce fast-path work while increasing guard overhead, code size, and invalidation cost. Its benefit needs measurement.

Profiles are historical samples, not semantic proofs. Protection need not be a guard at every call: a runtime can register dependencies and synchronously invalidate affected code before method, hierarchy, or prototype changes become visible. An object-shape check alone does not protect against every call-target change. An assumption without a guard, proof, or reliable invalidation mechanism is a correctness bug.

4. Deoptimization: Rebuilding Abstract State

Optimizing code may:

• Fold multiple source variables into a single register;
• Eliminate unused object allocations;
• Inline deeply nested function calls;
• Store values in constants or recomputed expressions.

During deoptimization, the interpreter or low-level runtime must reconstruct the stack frames, local variables, and objects required by the original abstract semantics. At each potential exit point, a state map must be preserved: indicating whether a given abstract value currently resides in a register, a stack slot, a constant, or a virtual object that still needs materialization.

Machine code alone is insufficient: debugging, GC, exceptions, and deoptimization need source and runtime-state mappings. Materializing virtual objects must preserve identity and sharing; it cannot arbitrarily rerun constructors with observable effects.

5. OSR Lets a Running Loop Switch Versions

If a function enters a long-running loop, it makes no sense to apply an optimized version only after the function returns. On-Stack Replacement (OSR) enables safely migrating the current execution state into optimized code at a safe entry point within the loop.

The reverse can also happen: when optimization assumptions fail, the system can revert back to the generic version during execution. OSR mappings must handle live locals, stack values, exception states, and virtual objects. Errors typically manifest only under specific hot paths and at particular iteration points.

6. Safepoint Connection Between Compiler and Runtime

Many managed runtimes provide the precise stack and register descriptions needed by GC, cooperative suspension, and deoptimization at safepoints. This is a runtime protocol: it does not mean the operating system can preempt threads only there, or that every GC performs all its work at safepoints. The compiler generates at these points:

• Which locations contain object references;
• The inlined call stack;
• Recoverable local state;
• Return addresses and exception handling information.

Too few safepoints increase pause time due to waiting, while too many increase overhead from checks and metadata processing. Unbounded pure computation loops may require explicit polling to ensure the runtime eventually regains control.

7. Code Cache and W^X

JIT compilers must write new code and execute it. Implementations commonly avoid making one virtual mapping writable and executable at the same time: generate through writable pages, then switch permissions and synchronize instruction caches before publishing executable code. Some platforms use RW/RX aliases of the same storage. That does not make executable physical content immutable; writable aliases, patching authority, and concurrent publication still need control.

Additional concerns include:

• Synchronizing instruction cache updates;
• Relocating code addresses;
• Managing code cache capacity and eviction policies;
• Ensuring that concurrent threads are no longer executing the old code;
• Verifying the control flow integrity of generated code and maintaining sandbox boundaries.

JIT compilation expands the attack surface, especially when intermediate representations or bytecode originate from untrusted sources. Input validation, resource limits, and isolation of executable memory are essential components of runtime design.

8. Compiler Correctness Requires Multi-Layer Evidence

IR Verifier

After each pass, verify structural integrity, type consistency, SSA form, and control flow graph invariants.

Differential Testing

Run the same program through the interpreter, unoptimized compiler, and optimized compiler, then compare observable behaviors. Even reference implementations can contain bugs, so a discrepancy needs investigation before blaming the new version.

Random and Mutation Testing

Generate small programs with well-defined behavior, or apply semantically preserving transformations to existing programs. If inputs contain undefined behavior, different outputs may all be valid, leading to false positives.

Translation Validation

For each specific transformation or entire function, attempt to prove equivalence between the input and output IRs, or that the transformation satisfies a refinement relation. It does not require proving that the optimizer is always correct, but rather validates the correctness of the current output.

Regression and Performance Testing

Individual gates must be enforced for correctness, compilation time, code size, peak performance, warm-up behavior, and memory usage. Counting assembly lines alone is insufficient to prove faster performance.

9. Connecting the Compiler Pipeline

The compiler pipeline is now complete:

Characters
→ Tokens
→ Syntax Tree
→ Names and Types
→ IR / CFG / SSA
→ Optimization
→ Target Instructions and Registers
→ Executable Code

Real compilers iterate back and forth, retain multiple layers of IR, and collaborate with linkers, runtimes, debuggers, and garbage collectors. This pipeline is not a one-time translation but a sequence of semantic transformations preserving invariants.

Common Misconceptions

• JIT compiles only hot code, so cold code has no cost: Even cold code consumes resources: interpretation, profiling, and baseline execution all require system overhead.
• Profiled types can be assumed to be permanent: Speculation needs guards or reliable dependency invalidation, plus safe exit paths.
• Deoptimization simply means falling back to interpretation: In reality, the system must precisely reconstruct the abstract state that was previously optimized away.
• Output differing from a reference means it's wrong: Differences may stem from undefined behavior, non-determinism, or actual reference bugs; these must be ruled out before concluding failure.

Exercise

1. Design shape guard and deopt paths for monomorphic call inlining.
2. List the information a state map for an inlined two-layer function must store.
3. Design a benchmark that measures both warmup and steady-state throughput to avoid reporting only the best single run.
4. Design a verifier, differential, and random test suite for a constant folding pass.

Summary

The difference between AOT and JIT lies in when information is available and costs are paid. JIT places bets on runtime behavior using guards, and pulls back when those assumptions fail via deoptimization; OSR, safepoints, and the code cache implement this strategy in actual runtime execution. Compiler correctness ultimately depends on clear semantics, local validation, and layered testing working together to maintain integrity.

References

• V8: speculative optimization and deoptimization
• OpenJDK: performance techniques and deoptimization