17.1 Assembly Basics and Calling Conventions

After C and C++ programs are compiled and linked, the CPU executes machine instructions. Function calls must follow a concrete protocol: where arguments go, who saves each register, and how return values move between caller and callee. The ABI and calling convention define that protocol.

The CPU doesn't recognize C's parameter names or return types. Two functions compiled separately can call each other because they follow the same Application Binary Interface (ABI): where parameters are placed, which registers must be saved, how the stack is aligned, how aggregate types are split, and how symbols are named.

This chapter uses the System V AMD64 LP64 ABI common on Linux, with 64-bit long and pointers; x32 ILP32 is outside its scope. Rules for Windows x64, AArch64, and RISC-V are different, and register conventions cannot be directly transferred across platforms.

1. First, understand the smallest function

int add(int left, int right) {
    return left + right;
}

add requires the mathematical sum to fit in int; seeing leal does not give signed C overflow defined wraparound semantics. On x86-64 SysV, the first two integer parameters are typically in %edi and %esi, and an int return value is in %eax. When GCC uses AT&T syntax, the optimized assembly may approach:

add:
    leal    (%rdi,%rsi), %eax
    ret

This isn't a one-to-one translation of C statements into assembly. The compiler determines that no stack slots, frame pointer, or register preservation is needed, so the entire function reduces to just operations and a return.

AT&T syntax is typically written as "source, destination," with registers carrying % and immediates carrying $; Intel syntax is often written as "destination, source." Always verify the syntax before examining disassembly to avoid misreading data direction.

2. Integer Parameters and Return Values

SysV AMD64 uses the first six parameters of the integer class in order:

1: rdi
2: rsi
3: rdx
4: rcx
5: r8
6: r9

int and other narrower values use corresponding subregisters, such as edi; pointers and common 64-bit integers use full registers. Integer return values typically originate from rax and may additionally use rdx if necessary.

Ordinary float/double arguments use xmm0–xmm7. AVX vectors can use the corresponding ymm/zmm registers; long double and other types follow additional classification rules. When a function contains both integer and floating-point parameters, the allocation rules for each group proceed separately, and the first six slots are not shared among all parameters.

Aggregate types must also be recursively classified according to the ABI's eight-byte classification. Remember that only six integer registers can handle common scalar functions and cannot replace formal ABI rules.

3. Caller-saved vs. callee-saved

Function calls divide registers into two categories:

Category	Common AMD64 Register in SysV	Responsibility
caller-saved	rax, rcx, rdx, rsi, rdi, r8–r11	If the caller needs the original value, it must save it before the call
callee-saved	rbx, rbp, r12–r15	If the callee modifies a register, it restores the original value before returning
stack pointer	rsp	restored to the position required by the calling convention

The caller preserves needed XMM data-register values. Ordinary arithmetic flags may change, but DF must be clear at function entry and return; floating-point control state has separate preservation rules.

“caller-saved registers being clobbered” doesn’t mean the callee must modify them all; it means the caller can’t rely on them remaining unchanged. The compiler decides whether to save them to other registers or the stack based on live ranges and register pressure.

Callee-saved registers aren't necessarily saved at the function entry point using push. Functions that only use r12 need only save it; the compiler might instead use mov to save and arrange restoration across multiple return paths.

4. Stack Alignment, Return Address, and Frame Pointer

x86-64's call pushes the address of the next instruction onto the stack before jumping to the target. Ordinary scalar calls require at least 16-byte stack alignment before executing call, so a regular function, upon entry, is offset by 8 bytes from a 16-byte boundary due to the 8-byte return address, as seen with rsp.

Stack-passed __m256, __m512, or more strongly aligned objects can require 32-byte, 64-byte, or stronger argument-area alignment. Sixteen bytes is not the sole rule for every signature.

A hypothetical frame that uses a frame pointer might look like:

higher addresses
  stack-passed arguments
  return address          <- call wrote this
  saved rbp               <- optional function prologue
  saved callee registers
  local spill slots
  outgoing call area / alignment
lower addresses           <- rsp after allocation

This isn't a fixed template available to every function. The optimizer often omits the rbp frame pointer, local variables may reside in registers, stack slots can be reused, and tail calls might jump directly to another function without creating a new return address.

Red zone

SysV AMD64 reserves 128 bytes below rsp as a red zone for data that need not survive a function call. Leaf functions can often use it without adjusting rsp, under the agreement that signal/interrupt handlers will not corrupt it. Kernel code and environments using different ABIs or compilation options cannot blindly rely on this region.

5. The Seventh Integer Argument and Beyond

When integer argument registers are insufficient, remaining parameters are placed in a stack parameter area prepared by the caller. The rule cannot simply be written as "push one by one from right to left":

• The caller must ensure overall stack alignment;
• Parameters might need their own alignment and categorization;
• The compiler first adjusts rsp once, then fills the slot with mov;
• Aggregated parameters may be split across register and memory rules;
• Returning a large object's hidden parameter occupies a parameter register;
• Variadic parameters also have additional ABI metadata.

For a regular function with eight long parameters, the first six use registers, while the seventh and eighth are read from above the return address at the callee entry point. The exact offsets must be determined by considering whether the prologue moves rsp/rbp, rather than by memorizing source code line numbers alone.

6. Small structs don't always take memory returns

The old rule "use a hidden pointer if larger than 8 bytes" is wrong. SysV AMD64 classifies aggregate returns in eightbytes. Common small non-vector structures can return in two register eightbytes; vector and special classifications require additional rules. For example:

typedef struct {
    long x;
    long y;
} Pair;

Two long typically belong to two INTEGER eightbyte values, so Pair can be returned via rax and rdx without writing to the caller's buffer.

Larger objects or objects classified as MEMORY typically use the sret protocol: the caller provides the return buffer address and passes a hidden pointer as an implicit parameter to the function. It modifies the register usage for subsequent explicit parameters, and the callee still returns the appropriate address as per the ABI specification.

Output parameters are not automatically faster than structure returns. Small aggregates can return efficiently in registers; even with hidden buffers, the compiler might eliminate unnecessary copies via return value optimization. Check the target ABI and generated code.

7. A Disassemblable Experiment

Save as abi-demo.c. It demonstrates eight long arguments, a 16-byte Pair return, and a 24-byte Triple return. The chosen inputs keep signed summation within range. noinline is a GCC/Clang extension used to maintain function boundaries for easy observation:

#include <stdio.h>
#include <stdlib.h>

#if defined(__GNUC__) || defined(__clang__)
#define NOINLINE __attribute__((noinline))
#else
#define NOINLINE
#endif

_Static_assert(sizeof(long) == 8 && sizeof(void *) == 8,
               "this experiment expects LP64");

typedef struct {
    long x;
    long y;
} pair_t;

typedef struct {
    long x;
    long y;
    long z;
} triple_t;

NOINLINE long sum_eight(
        long a, long b, long c, long d,
        long e, long f, long g, long h) {
    return a + b + c + d + e + f + g + h;
}

NOINLINE pair_t make_pair(long x, long y) {
    pair_t result = { x, y };
    return result;
}

NOINLINE triple_t make_triple(long x, long y, long z) {
    triple_t result = { x, y, z };
    return result;
}

int main(void) {
    volatile long input[8] = { 1, 2, 3, 4, 5, 6, 7, 8 };
    long total = sum_eight(
        input[0], input[1], input[2], input[3],
        input[4], input[5], input[6], input[7]
    );
    pair_t pair = make_pair(input[0], input[1]);
    triple_t triple = make_triple(
        input[0], input[1], input[2]
    );

    printf("%ld | %ld,%ld | %ld,%ld,%ld\n",
           total,
           pair.x, pair.y,
           triple.x, triple.y, triple.z);

    return total == 36
        && pair.x == 1 && pair.y == 2
        && triple.x == 1 && triple.y == 2 && triple.z == 3
        ? EXIT_SUCCESS : EXIT_FAILURE;
}

cc -std=c17 -O2 -g -Wall -Wextra -Werror -fno-omit-frame-pointer abi-demo.c -o abi-demo
./abi-demo

cc -std=c17 -O2 -S -fverbose-asm abi-demo.c -o abi-demo.s
objdump -drwC abi-demo

Observation point:

1. Where does sum_eight read its seventh and eighth arguments?
2. Does make_pair use rax/rdx?
3. Does make_triple receive a hidden result-buffer pointer, distinct from the return address pushed by call?
4. How does the prologue change when enabling and disabling -fno-omit-frame-pointer?
5. After removing volatile input, can the compiler perform further constant propagation of calls?

volatile preserves input accesses and noinline preserves calls, but neither disables all interprocedural optimization or cloning. These example functions have external linkage. To verify a separate-compilation boundary, put functions and callers in separate translation units and disable LTO. volatile still does not synchronize threads.

8. What does each tool answer

Let the compiler output assembly

cc -O2 -S source.c -o source.s
cc -O2 -S -masm=intel source.c -o source-intel.s

Assembly at the source code level makes it easy to see function boundaries and compiler comments, but doesn't have the final linked addresses.

Look at object file

cc -O2 -c source.c -o source.o
objdump -drwC source.o
readelf -s source.o
readelf -r source.o

objdump -dr shows disassembly beside relocations. readelf -s lists symbols, while readelf -r shows locations the linker will patch. Call offsets in an object may still be placeholders.

Look at the runtime stack

(gdb) break make_triple
(gdb) run
(gdb) info registers rdi rsi rdx rcx r8 r9 rsp rbp
(gdb) x/16gx $rsp
(gdb) info frame
(gdb) disassemble /m make_triple

Debugger results are affected by optimization. Variables may be eliminated, functions may be inlined, and tail calls may reduce the backtrace depth. -O0 enables step-by-step debugging but creates a large number of instructional stack slots; -O2 -g is closer to the actual code.

9. Why Variadic Arguments Are More Complex

printf and similar variadic functions do not know the type of unnamed parameters, so both caller and callee must follow additional ABI rules. SysV AMD64 uses %al to inform the variadic call site of an upper bound on how many vector register parameters are used; the callee must either establish or use a register save area so that va_arg can sequentially access register and stack parameters.

This is not a general rule that "preserves all floating-point parameter counts precisely," and it doesn't apply to non-variadic functions. If declarations and definitions disagree about whether parameters are variadic, their types, or their prototype, bits will be misinterpreted at the wrong location.

Default argument promotions in ... convert float to double and promote small integers to int or unsigned int according to the integer promotion rules. printf format mismatch with argument type is undefined behavior, not a safe dynamic cast.

10. The Real Risks of Inline Assembly

The format of GCC extended asm is:

asm qualifiers (
    template
    : output operands
    : input operands
    : clobbers
)

Constraints tell the optimizer where inputs go, which values are modified, and whether memory not listed as an operand is accessed. Even if the assembly machine instructions are correct, wrong constraints can still produce incorrect code, because the compiler might assign two separate operands to the same register or move memory accesses into or out of the assembly block.

asm volatile prevents certain deletions and loop-hoisting optimizations, but some movement, duplication, or merging may still occur, without automatically becoming a C thread memory barrier. "memory" clobber constraint prevents the compiler from reordering memory accesses, but does not issue the hardware fence required by the target CPU.

Atomic operations should prefer <stdatomic.h> or compiler-built-in atomic instructions; system calls should use existing wrappers. Inline assembly is suitable for implementing compiler/kernel boundaries or for instructions that truly lack intrinsics, and such cases require review by target architecture experts.

11. How ABI Mismatch Errors Occur

The linker primarily looks up addresses by symbol and does not perform complete runtime type checking on C functions. The following inconsistencies could all successfully link and silently corrupt state:

• The caller considers the parameter as int, while the callee reads it from the XMM register as double;
• Using different struct packing;
• Enabled an incompatible calling convention;
• Variadic format does not match the actual type;
• C differs from FFI conventions in other languages regarding strings, ownership, or exception handling;
• Incompatible function declarations found in a compilation unit.

Public headers, FFI generators, and ABI tests help establish a shared contract. extern "C" selects C language linkage; it does not convert data layouts, ownership conventions, or C++ exceptions. Across language boundaries, it's essential to clearly define who is responsible for memory allocation and deallocation.

12. Architecture Comparison

AArch64 commonly uses x0–x7 to pass the first eight integer parameters, while RISC-V commonly uses a0–a7. Their callee-saved register sets, floating-point registers, stack alignment, and aggregate classification differ accordingly.

More argument registers alone do not make an ABI better. Register count, encoding, call density, compiler strategies, and workloads all jointly determine performance. When switching architectures, refer to the corresponding psABI, rather than mechanically replacing register names.

13. Summary

A calling convention is a binary protocol that enables code shared among separately compiled units:

• Common integer parameters come first rdi, rsi, rdx, rcx, r8, r9;
• Caller-saved values are not protected across calls, while callee-saved values are restored by the callee that uses them;
• The stack parameter area is also affected by alignment, type classification, and hidden parameters;
• The frame pointer can be omitted, and the stack frame does not have a unique fixed structure;
• Two integer eight-byte small struct types can be returned in registers;
• Large or MEMORY class return values typically use a hidden buffer;
• Variadic functions, inline assembly, and cross-language FFI all require stricter ABI contracts.

The next chapter covers Performance Engineering: starting with measurement, bringing back the costs of assembly, cache, branches, locks, and I/O into real workloads.

Check the ABI and Compiler Contracts

• System V AMD64 psABI
• GCC extended asm
• GCC function attributes