10.2 SIMD Kernel, Runtime Dispatch, and Numerical Semantics

The vectorized executor has already batched data processing, and the performance engineer still wants a single CPU instruction to compute multiple elements at once.

SIMD allows one instruction to operate on multiple lanes. A truly usable database kernel must also handle alignment, tail, NULL mask, selection, overflow, floating-point reduction order, and different CPU features. Writing _mm256_add_* is just the beginning.

ISA and lanes

• x86 SSE2: 128-bit integer/floating vectors;
• AVX: 256-bit floating operations;
• AVX2: Extends 256-bit integer operations and gathers, etc.;
• AVX-512: 512-bit vector + richer mask/compress subsets;
• ARM NEON: Common 128-bit SIMD;
• ARM SVE/SVE2: vector-length-agnostic programming model.

CPU claims that AVX-512 does not guarantee support for all AVX-512 subsets or that it's faster than AVX2; results can vary based on frequency, ports, memory bandwidth, downclocking, and kernel mixing.

If binary execution lacks runtime dispatch and runs on CPUs that don't support ISA, it will result in an illegal instruction. Special caution is required during container/VM migration and in heterogeneous clusters.

Safe AVX2 Addition

#include <cstddef>
#include <immintrin.h>

void add_f32_avx2(
    const float* left,
    const float* right,
    float* output,
    std::size_t count) {

    std::size_t i = 0;
    const std::size_t vector_end = count - (count % 8);

    for (; i < vector_end; i += 8) {
        const __m256 a = _mm256_loadu_ps(left + i);
        const __m256 b = _mm256_loadu_ps(right + i);
        _mm256_storeu_ps(output + i, _mm256_add_ps(a, b));
    }

    for (; i < count; ++i) {
        output[i] = left[i] + right[i];
    }
}

The for (i=0; i<n; i+=8) in the source will cause out-of-bounds reads when n is not a multiple of 8, and the tail loop cannot fix this. The main loop must only process complete vectors.

loadu supports unaligned addresses. Aligned load intrinsics require the corresponding alignment; violating this results in undefined behavior or a possible fault. On modern CPUs, whether unaligned loads are slower depends on whether they cross cache lines or pages, so it's not accurate to say universally that "unaligned access is always slower."

Runtime dispatch

Available multi-version functions:

scalar baseline
SSE2 version
AVX2 version
AVX-512 version

At process startup or first call, detect OS+CPU support and select a function pointer. For x86, in addition to CPUID feature bits, AVX requires the OS to preserve extended state (via XGETBV or compiler builtins, which typically handle this). Don't just check a single CPUID bit on your own.

GCC/Clang's function multiversioning, target attributes, or platform dispatch libraries can reduce manual errors; build flags and the minimum CPU baseline must be documented.

Auto-vectorization

Compilers vectorize more easily:

• simple countable loop;
• contiguous non-aliasing pointers;
• known alignment/stride;
• no loop-carried dependency;
• operation semantics allows reordering;
• branch can be converted to mask/select.

obstacle:

• possible pointer alias;
• function calls/virtual dispatch;
• variable-length strings;
• unpredictable gather/scatter;
• exact floating order;
• overflow/trap semantics;
• early exits.

Use the vectorization report and disassembly to prove, don't claim SIMD has been generated just because you see -O3.

Alias and restrict

If the compiler worries about output overlapping with left/right, it can't arbitrarily reorder them. C restrict or C++ compiler-specific assumptions can help, but incorrect promises lead to undefined behavior.

Safer is to explicitly disallow overlap in the API and verify it with tests/sanitizers; don't add unverified alias annotations just for performance.

Comparison, Masking, and Selection

AVX2 comparing 8 int32 values:

__m256i values = _mm256_loadu_si256(
    reinterpret_cast<const __m256i*>(input + i));
__m256i threshold = _mm256_set1_epi32(50000);
__m256i lanes = _mm256_cmpgt_epi32(values, threshold);
int bitmask = _mm256_movemask_ps(_mm256_castsi256_ps(lanes));

bitmask Each bit corresponds to a lane. Next possible:

• bit scan generates selected indices;
• Keep the bitmask for downstream;
• Use table/permute to compact;
• Go down the dense path when selectivity is high.

Storing 32-bit comparison lanes directly into uint8_t mask[8] would cause buffer overflow or incorrect layout.

NULL mask

If validity is bit-packed, the SIMD predicate mask must be combined with the valid bits. For column > constant:

true_mask = comparison_mask & validity_mask

But compound SQL expressions require (true_mask, null_mask) or an equivalent representation. For example:

FALSE AND NULL = FALSE
TRUE  AND NULL = NULL
TRUE  OR  NULL = TRUE
FALSE OR  NULL = NULL

You can't handle all logic operators just by processing AND and validity.

Integer overflow

C/C++ signed overflow is undefined behavior; database integer arithmetic typically requires overflow detection and error reporting, or type promotion for aggregate accumulators. The compiler cannot optimize away SQL overflow checks under -fstrict-overflow assumptions.

Vector add can be implemented using widened lanes, compare/sign logic, or ISA overflow detection patterns; the result must be consistent with scalar database semantics.

Floating aggregation

scalar left fold:

(((a0 + a1) + a2) + a3) ...

SIMD uses multiple partial accumulators to change the addition order. Floating-point addition is non-associative, so low bits and rounding can differ; NaN, signed zero, and overflow also need to be defined.

Databases may allow non-deterministic parallel floating-point aggregates, or use more stable approaches like pairwise/Kahan/decimal. Regardless of the approach, it should be documented in the contract and tested. -ffast-math relaxes IEEE assumptions and should not be used in the SQL kernel without thorough review.

SIMD sum skeleton

#include <cstddef>
#include <immintrin.h>

float sum_f32_avx2(const float* values, std::size_t count) {
    __m256 vector_sum = _mm256_setzero_ps();
    std::size_t i = 0;
    const std::size_t vector_end = count - (count % 8);

    for (; i < vector_end; i += 8) {
        vector_sum = _mm256_add_ps(
            vector_sum,
            _mm256_loadu_ps(values + i));
    }

    alignas(32) float lanes[8];
    _mm256_store_ps(lanes, vector_sum);

    float total = 0.0F;
    for (float lane : lanes) {
        total += lane;
    }
    for (; i < count; ++i) {
        total += values[i];
    }
    return total;
}

This example teaches tail/alignment, doesn't handle NULL, and doesn't guarantee bit-identical results with scalar left fold. The production kernel should include ISA dispatch, tests, NaN policy, and better horizontal reduction.

Memory-bound vs. compute-bound

If scan performs only one add operation and memory bandwidth is saturated, the theoretical 8 lanes won't provide 8× speedup. You can intuitively understand this using the roofline model:

attainable performance <= min(compute peak,
                              memory bandwidth × arithmetic intensity)

Compression reduces bytes but increases decode compute; sometimes SIMD decode improves both. Hash probe/random gather is often limited by cache-miss latency.

AVX/SSE Transition and Frequency

Old x86 microarchitectures mixing legacy SSE and AVX register states might incur a transition penalty, and compilers often insert vzeroupper. Wide-vector frequency behavior also varies by microarchitecture and workload.

Don't specify fixed penalty cycles or claim AVX-512 will necessarily throttle, use target CPU hardware counters, frequency, and end-to-end workload measurements instead.

Benchmark

At least includes:

• aligned/unaligned, across cache line/page;
• size goes from L1-resident to memory-bandwidth bound;
• selectivity 0%, 1%, 50%, 99%, 100%;
• NULL density/pattern;
• tails 0–vector_width-1;
• skew/gather/string;
• scalar baseline, auto-vectorized, intrinsics;
• result bit correctness, overflow/NaN;
• runtime dispatch cold/warm;
• end-to-end query, rather than reporting microkernel alone.

Acceptance Checklist

• [ ] SIMD loop does not overflow and handles the tail;
• [ ] Unsupported CPU has scalar/baseline path;
• [ ] Mask layout matches the destination type;
• [ ] NULL preserves three-valued logic;
• [ ] Integer overflow and floating-point arithmetic have a contract;
• [ ] Use compiler report or disassembly to prove vectorization;
• [ ] Measure both cache-resident and memory-bound cases simultaneously;
• [ ] Verify microkernel revenue within the full query.

Chapter Summary

Vectorization is an execution architecture, and SIMD is a hardware capability within it. True performance gains come from appropriate data representations, batching, masking/selection, runtime dispatch, and strict numerical semantics, rather than simply replacing scalar loops with intrinsics. The next chapter returns to the CPU and explores transactions: how multiple operations form recoverable state changes through atomicity, consistency, isolation, and durability.