2.2 Floating-Point Representation and Numerical Errors

The temperature readings from the Earth Engine's thermostat show subtle discrepancies that are imperceptible to the naked eye. Ah Hua decides to trace the issue back to how numbers are represented at the machine level.

Integer Representation uses a fixed width to represent a contiguous range of integers. Floating-point format allocates a finite number of bits between significant digits and an exponent, enabling it to represent a much broader range of magnitudes through discrete, representable values.

Floating-point numbers are not "real numbers with decimal points"

After the controller powers back on, the sensor displays 0.1. This decimal string is first parsed into a nearby binary floating-point number, used in calculations, and then formatted back into text. The screen may still show 0.1, but the internal value does not necessarily equal the mathematical value of one-tenth.

Any fixed-width format can only represent a finite number of bit patterns, making it impossible to precisely express all real numbers. Floating-point design accepts this limitation and makes a trade-off between range and relative precision. It's more akin to binary scientific notation than an infinitely precise decimal container.

How are binary32 and binary64 bits allocated?

The two most commonly used binary formats of IEEE 754 are:

Format	Common Language Types	Sign Bit	Exponent Bit	Fraction Bit	Decimal Significant Digits
binary32	C/Java float	1	8	23	6–9
binary64	C/Java double	1	11	52	15–17

"6–9" doesn't mean every binary32 has nine decimal digits of precision. It expresses two distinct guarantees: approximately six decimal digits can be reliably converted back and forth between binary32, and uniquely identifying any binary32 value typically requires at most nine decimal digits.

For a normalized finite value, if the sign bit is s, the exponent field is E, and the fraction integer is F with p fraction bits, then the value is:

(-1)^s × (1 + F / 2^p) × 2^(E - bias)

binary32 has p = 23 and bias = 127; binary64 has p = 52 and bias = 1023. The leading 1 is not stored and is commonly called the hidden bit or implicit bit.

For example, the binary of 5.25 is 101.01, normalized to 1.0101 × 2^2. In binary32:

s = 0
Exponent field E = 2 + 127 = 129 = 10000001₂
fraction = 01010000000000000000000
bit pattern = 0 10000001 01010000000000000000000
Hexadecimal = 0x40A80000

Preserve the Exponent Field for Special Values

When the exponent field is all zeros or all ones, the standard normalized formula is not used:

Exponent Field	Fraction	Meaning
All 0s	All 0s	+0 or -0
All 0s	Non-zero	subnormal (denormal)
All 1s	All 0s	+∞ or -∞
All 1s	Non-zero	NaN

Denormal numbers lack the implied leading 1, and their value is:

(-1)^s × (F / 2^p) × 2^(1 - bias)

This allows values near zero to gradually lose precision rather than jumping abruptly from the smallest normalized value to zero. NaN is used to represent invalid results, such as 0.0 / 0.0; comparisons involving NaN with any value are false, even NaN == NaN is false, use isnan to check for NaN.

Positive and negative zero are considered equal in comparison, yet they can retain sign information in certain operations, such as 1.0 / +0.0 producing positive infinity and 1.0 / -0.0 producing negative infinity. Whether such results are permitted depends on the language, runtime, and floating-point exception settings.

Use memcpy to Inspect Object Representation

The following C17 program examines common bit patterns of binary32. It first validates the assumptions using sizeof and FLT_RADIX, then copies the object representation via memcpy, avoiding strict aliasing issues caused by incompatible pointer types.

#include <float.h>
#include <inttypes.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>

static void print_binary32(float value) {
    uint32_t bits = 0;
    memcpy(&bits, &value, sizeof bits);

    uint32_t sign = bits >> 31;
    uint32_t exponent = (bits >> 23) & UINT32_C(0xFF);
    uint32_t fraction = bits & UINT32_C(0x7FFFFF);

    printf("value=% .9g bits=0x%08" PRIX32
           " sign=%" PRIu32 " exponent=%" PRIu32
           " fraction=0x%06" PRIX32 "\n",
           value, bits, sign, exponent, fraction);
}

int main(void) {
    if (sizeof(float) != sizeof(uint32_t) || FLT_RADIX != 2
            || FLT_MANT_DIG != 24 || FLT_MAX_EXP != 128) {
        fputs("this example requires IEEE 754 binary32 float\n", stderr);
        return 1;
    }

    print_binary32(5.25f);
    print_binary32(0.1f);
    print_binary32(-0.0f);
    return 0;
}

On machines satisfying the assumptions, the first line's bits is 0x40A80000. The fraction portion of 0.1f is rounded at the end, and thus does not yield the infinite expansion of a fully precise result.

Why the Decimal 0.1 on Sensors Cannot Be Precisely Stored

The controller screen still shows 0.1, that’s just a formatted short text. When you expand the internal binary64 value, it lands near a mathematical tenth; the error stems from the representation grid, not from a careless arithmetic mistake during addition.

Finite binary fractions can only represent rational numbers whose reduced denominators are powers of two. Since decimal 0.1 = 1/10 has a factor of five in its denominator, its binary expansion is infinite and repeating:

0.0001100110011001100110011...₂

The parser can only round it to the nearest representable value. Both 0.1 and 0.2 are already approximations in binary64; when they are added and then rounded, the result typically becomes:

>>> format(0.1 + 0.2, ".17g")
'0.30000000000000004'
>>> 0.1 + 0.2 == 0.3
False

This isn’t a failure of all floating-point addition, each step strictly follows the format and rounding rules. Binary finite decimals like 0.5, 0.25, and others can be represented exactly. Many comparisons involving the same stored value also have well-defined and useful applications.

Also note that precision depends on magnitude. In binary32, the gap between adjacent values near 1 is approximately 2^-23, but by the time we reach 2^24, the spacing is so large that individual integers can no longer be represented. A wide range does not imply high precision.

Compare by First Defining "Close Enough"

Floating-point numbers can never be compared using ==, that statement is too absolute. In cases like comparing the same constant, exact zero results, cache keys, or special values defined by a protocol, exact comparison may indeed be required. However, results from approximate computations typically require an error model.

Using only a fixed absolute tolerance can be too strict for large numbers and may require a separate lower bound near zero. A common strategy combines both relative and absolute tolerances:

#include <math.h>
#include <stdbool.h>

static bool nearly_equal(double left, double right,
                         double relative_tolerance,
                         double absolute_tolerance) {
    if (relative_tolerance < 0.0 || absolute_tolerance < 0.0) {
        return false;
    }
    if (left == right) {
        return true; /* Also covers infinity of the same sign and positive and negative zero */
    }
    if (!isfinite(left) || !isfinite(right)) {
        return false;
    }

    double difference = fabs(left - right);
    double scale = fmax(fabs(left), fabs(right));
    return difference <= fmax(absolute_tolerance,
                              relative_tolerance * scale);
}

Tolerances should not be arbitrarily set to 1e-9. They must be derived from input error margins, algorithm stability, and domain-specific requirements. For instance, a temperature sensor might tolerate 0.01°C. Geometric calculations often incorporate relative scaling, while financial values typically use integer amounts at the smallest currency unit or follow explicit rounding rules with decimal types.

Floating-point addition order affects rounding

Floating-point addition satisfies the commutative law in common finite cases, but does not satisfy the associative law of real number addition:

(a + b) + c may not equal a + (b + c)

When a large number is added to a very small one, the small component may fall below the current floating-point interval and be lost. In long sequences of summation, the order of operations, grouping, and compensation algorithms all influence the accumulated error.

values = [1e16, 1.0, -1e16]

print(sum(values))
print(sum(reversed(values)))

Both of these orders can lose 1.0 in typical binary64 computations; switching to math.fsum(values) with a more accurate accumulation strategy yields 1.0. This does not mean one function is suitable for all scenarios: throughput, reproducibility, and error bounds must still be considered in light of the specific use case.

Fused multiply-add (FMA) rounds only once, and may be more precise than splitting into separate multiplication and addition operations. However, it can produce results that differ from those on platforms without FMA at the last bit. Parallel reduction changes the addition tree and may likewise alter the final bit. When bit-level reproducibility is required, the algorithm, compiler flags, hardware path, and rounding mode must all be carefully controlled.

Four Basic Rounding Directions

The basic rounding directions defined by IEEE 754 include:

• Round to the nearest value, with ties broken by rounding to the nearest even digit; this is the common default mode;
• Round toward zero;
• Round toward positive infinity;
• Round toward negative infinity.

Rounding modes can affect boundary results, but switching the runtime environment does not guarantee that all compile-time constant folding and optimizations automatically adhere to the new mode. C programs that rely on dynamic rounding environments must use <fenv.h>, ensure proper implementation support, and apply the appropriate compiler settings, and must validate behavior against the target toolchain.

Ask Yourself First: What Is the Use Case?

Scenario	Common Choice	Reason
Graphics, machine learning tensors	binary16/32/64	Strong hardware support; controlled approximation is acceptable
Scientific computing	start with binary64, paired with error analysis	Balanced range and precision; mature algorithms
Financial accounting	integer at smallest unit, or decimal/fixed-point	Decimal rounding rules enable clear auditability
Probabilities, measured values	floating-point with error semantics	Inputs are inherently approximate
Hash keys, identifiers	integers or normalized text	Approximate equality should not be used for identity comparison

Decimal types have limited precision and rounding, though they can represent common decimal fractions exactly. Fractional types can store rational numbers precisely, but numerators and denominators may grow rapidly. There is no unconstrained "universal exact type."

Executable Classification and Comparison Tests

import math

assert 0.5 + 0.25 == 0.75
assert 0.1 + 0.2 != 0.3
assert math.isclose(0.1 + 0.2, 0.3,
                    rel_tol=1e-12, abs_tol=0.0)

nan = float("nan")
assert nan != nan
assert math.isnan(nan)
assert math.isinf(float("inf"))
assert 0.0 == -0.0
assert math.copysign(1.0, -0.0) == -1.0

assert math.fsum([1e16, 1.0, -1e16]) == 1.0

These assertions cover exact binary fractions, decimal approximations, NaN, infinity, negative zero, and robust summation. Business tests should also include upper bounds on error under problem scales, rather than a few textbook constants alone.

Hands-on Breakdown of Errors

1. Manually compute the binary32 sign, exponent, and fraction components of -6.75f, and write out the corresponding hexadecimal bit pattern.
2. Using nextafter, find the nearest binary64 values adjacent to 1.0 and 1e20, and compare the gaps between them.
3. Construct a dataset where ordinary sum and math.fsum are clearly distinguishable, and explain where the loss of precision occurs.
4. For a latitude/longitude comparison interface, select between relative and absolute tolerance, clearly specifying the units and the rationale behind the choice.
5. Explain why NaN cannot serve as a conventional sentinel for "missing equals missing" in comparisons.

Representation Rules Enter Logical Circuits

Integers and floating-point values have been reduced to executable bit-level rules. The next chapter moves into Digital Logic: how gate circuits combine to form adders, registers, and state machines, and how a single operation in software ultimately lands on clock-driven hardware.