10.2 SIMD Kernel, Runtime Dispatch, and Numerical Semantics

The vectorized executor has already batched data processing, and the performance engineer still wants a single CPU instruction to compute multiple elements at once.

SIMD allows one instruction to operate on multiple lanes. A truly usable database kernel must also handle alignment, tail, NULL mask, selection, overflow, floating-point reduction order, and different CPU features. Writing _mm256_add_* is just the beginning.

ISA and lanes

• x86 SSE2: 128-bit integer/floating vectors;
• AVX: 256-bit floating operations;
• AVX2: Extends 256-bit integer operations and gathers, etc.;
• AVX-512: 512-bit vector + richer mask/compress subsets;
• ARM NEON: Common 128-bit SIMD;
• ARM SVE/SVE2: vector-length-agnostic programming model.

Support for one AVX-512 subset does not imply support for every subset or better performance than AVX2; results can vary based on frequency, ports, memory bandwidth, downclocking, and kernel mixing.

If binary execution lacks runtime dispatch and runs on CPUs that don't support ISA, it will result in an illegal instruction. Special caution is required during container/VM migration and in heterogeneous clusters.

Safe AVX2 Addition

These C++17 functions target x86-64 GCC/Clang. Each pointer must address at least count floats, and the output range must not overlap either input range. A zero count accesses no array elements. The target attribute enables AVX2 for this function only; calls still need the dispatch below. The floating-point addition itself needs only AVX; placing it in an AVX2 branch is a choice for this example.

#include <cstddef>
#include <immintrin.h>

__attribute__((target("avx2")))
void add_f32_avx2(
    const float* left,
    const float* right,
    float* output,
    std::size_t count) {

    std::size_t i = 0;
    const std::size_t vector_end = count - (count % 8);

    for (; i < vector_end; i += 8) {
        const __m256 a = _mm256_loadu_ps(left + i);
        const __m256 b = _mm256_loadu_ps(right + i);
        _mm256_storeu_ps(output + i, _mm256_add_ps(a, b));
    }

    for (; i < count; ++i) {
        output[i] = left[i] + right[i];
    }
}

A loop written as for (i=0; i<n; i+=8) would cause out-of-bounds reads when n is not a multiple of 8, and the tail loop cannot fix this. The main loop must only process complete vectors.

loadu supports unaligned addresses. Aligned load intrinsics require the corresponding alignment; violating this results in undefined behavior or a possible fault. On modern CPUs, whether unaligned loads are slower depends on whether they cross cache lines or pages, so it's not accurate to say universally that "unaligned access is always slower."

Runtime dispatch

Available multi-version functions:

scalar baseline
SSE2 version
AVX2 version
AVX-512 version

At process startup or first call, detect OS+CPU support and select a function pointer. For x86, in addition to CPUID feature bits, AVX requires the OS to preserve extended state (via XGETBV or compiler builtins, which typically handle this). Don't just check a single CPUID bit on your own.

Append this code after the addition function. On ordinary process call paths, GCC/Clang CPU builtins select a supported implementation; other CPUs use the scalar path:

void add_f32_scalar(const float* left, const float* right,
                    float* output, std::size_t count) {
    for (std::size_t i = 0; i < count; ++i) {
        output[i] = left[i] + right[i];
    }
}

void add_f32(const float* left, const float* right,
             float* output, std::size_t count) {
    using Add = void (*)(const float*, const float*, float*, std::size_t);
    static const Add implementation = __builtin_cpu_supports("avx2")
        ? add_f32_avx2 : add_f32_scalar;
    implementation(left, right, output, count);
}

Save the module as simd.cpp and compile with g++ -std=c++17 -O2 -c simd.cpp. Do not apply -mavx2 or -march=native to the whole module: the compiler could also introduce unsupported instructions into the baseline code. Test the scalar and dispatched entries separately; call the AVX2 entry directly only after the feature check passes. See GCC x86 CPU builtins.

GCC/Clang's function multiversioning, target attributes, or platform dispatch libraries can reduce manual errors; build flags and the minimum CPU baseline must be documented.

Auto-vectorization

Compilers vectorize more easily:

• simple countable loop;
• contiguous non-aliasing pointers;
• known alignment/stride;
• no loop-carried dependency;
• operation semantics allows reordering;
• branch can be converted to mask/select.

obstacle:

• possible pointer alias;
• function calls/virtual dispatch;
• variable-length strings;
• unpredictable gather/scatter;
• exact floating order;
• overflow/trap semantics;
• early exits.

Inspect vectorization reports and disassembly; -O3 alone does not establish that the compiler emitted SIMD.

Alias and restrict

If the compiler worries about output overlapping with left/right, it can't arbitrarily reorder them. C restrict or C++ compiler-specific assumptions can help, but incorrect promises lead to undefined behavior.

Safer is to explicitly disallow overlap in the API and verify it with tests/sanitizers; don't add unverified alias annotations just for performance.

Comparison, Masking, and Selection

Partial comparison fragment inside an AVX2 function: input addresses int32 data, and at least 8 readable elements remain at i. The containing function needs the same target attribute and dispatch:

__m256i values = _mm256_loadu_si256(
    reinterpret_cast<const __m256i*>(input + i));
__m256i threshold = _mm256_set1_epi32(50000);
__m256i lanes = _mm256_cmpgt_epi32(values, threshold);
int bitmask = _mm256_movemask_ps(_mm256_castsi256_ps(lanes));

Each bit in bitmask corresponds to a lane. The next stage can:

• bit scan generates selected indices;
• Keep the bitmask for downstream;
• Use table/permute to compact;
• Go down the dense path when selectivity is high.

Storing 32-bit comparison lanes directly into uint8_t mask[8] would cause buffer overflow or incorrect layout.

NULL mask

If validity is bit-packed, the SIMD predicate mask must be combined with the valid bits. For column > constant:

true_mask = comparison_mask & validity_mask

But compound SQL expressions require (true_mask, null_mask) or an equivalent representation. For example:

FALSE AND NULL = FALSE
TRUE  AND NULL = NULL
TRUE  OR  NULL = TRUE
FALSE OR  NULL = NULL

Implementing AND and validity propagation does not cover every Boolean operator.

Integer overflow

C/C++ signed overflow is undefined behavior; database integer arithmetic typically requires overflow detection and error reporting, or type promotion for aggregate accumulators. Check the range before addition or use checked arithmetic builtins. Performing overflowing signed addition and then checking the result has already invoked C++ undefined behavior, which can let the optimizer remove the intended check.

Vector add can be implemented using widened lanes, compare/sign logic, or ISA overflow detection patterns; the result must be consistent with scalar database semantics.

Floating aggregation

scalar left fold:

(((a0 + a1) + a2) + a3) ...

SIMD uses multiple partial accumulators to change the addition order. Floating-point addition is non-associative, so low bits and rounding can differ; NaN, signed zero, and overflow also need to be defined.

Databases may allow non-deterministic parallel floating-point aggregates, or use more stable approaches like pairwise/Kahan/decimal. Regardless of the approach, it should be documented in the contract and tested. -ffast-math relaxes IEEE assumptions and should not be used in the SQL kernel without thorough review.

SIMD sum skeleton

#include <cstddef>
#include <immintrin.h>

__attribute__((target("avx2")))
float sum_f32_avx2(const float* values, std::size_t count) {
    __m256 vector_sum = _mm256_setzero_ps();
    std::size_t i = 0;
    const std::size_t vector_end = count - (count % 8);

    for (; i < vector_end; i += 8) {
        vector_sum = _mm256_add_ps(
            vector_sum,
            _mm256_loadu_ps(values + i));
    }

    alignas(32) float lanes[8];
    _mm256_store_ps(lanes, vector_sum);

    float total = 0.0F;
    for (float lane : lanes) {
        total += lane;
    }
    for (; i < count; ++i) {
        total += values[i];
    }
    return total;
}

This function illustrates remainders and alignment; it does not handle NULL or promise bit-identical results with a scalar left fold. Call it only on a CPU that supports AVX2. Empty input returns the mathematical initial value 0; an outer state must supply SQL SUM semantics for empty or all-NULL input. Start with exactly representable small integer floats and lengths 0–33, then test NaN, infinity, and rounding separately. One absolute-error tolerance cannot define every numeric contract.

Memory-bound vs. compute-bound

If scan performs only one add operation and memory bandwidth is saturated, the theoretical 8 lanes won't provide 8× speedup. You can intuitively understand this using the roofline model:

attainable performance <= min(compute peak,
                              memory bandwidth × arithmetic intensity)

Compression reduces bytes but increases decode compute; sometimes SIMD decode improves both. Hash probe/random gather is often limited by cache-miss latency.

AVX/SSE Transition and Frequency

Old x86 microarchitectures mixing legacy SSE and AVX register states might incur a transition penalty, and compilers often insert vzeroupper. Wide-vector frequency behavior also varies by microarchitecture and workload.

Don't specify fixed penalty cycles or claim AVX-512 will necessarily throttle, use target CPU hardware counters, frequency, and end-to-end workload measurements instead.

Benchmark

At least includes:

• aligned/unaligned, across cache line/page;
• size goes from L1-resident to memory-bandwidth bound;
• selectivity 0%, 1%, 50%, 99%, 100%;
• NULL density/pattern;
• tails 0–vector_width-1;
• skew/gather/string;
• scalar baseline, auto-vectorized, intrinsics;
• result bit correctness, overflow/NaN;
• runtime dispatch cold/warm;
• end-to-end query, rather than reporting microkernel alone.

Acceptance Checklist

• [ ] SIMD loops stay within array bounds and handle the remainder;
• [ ] Unsupported CPU has scalar/baseline path;
• [ ] Mask layout matches the destination type;
• [ ] NULL preserves three-valued logic;
• [ ] Integer overflow and floating-point arithmetic have a contract;
• [ ] Use compiler report or disassembly to prove vectorization;
• [ ] Measure both cache-resident and memory-bound cases simultaneously;
• [ ] Verify microkernel gains within the full query.

Chapter Summary

Vectorization is an execution architecture, and SIMD is a hardware capability within it. True performance gains come from appropriate data representations, batching, masking/selection, runtime dispatch, and strict numerical semantics, rather than simply replacing scalar loops with intrinsics. The next chapter moves from CPU execution to transactions: how multiple operations form recoverable state changes through atomicity, consistency, isolation, and durability.