12.2 Floating-Point Numbers and Rounding: How Finite Bits Approximate Real Numbers

The Earth Engine once again sends a tiny numerical error, this time, the Mathematical Observatory explains it through the lens of finite representation and rounding rules.

In mathematics, real numbers are continuous and infinite. In computers, storage is finite. Floating-point numbers use a fixed number of bits to select a finite set of representable values; all other real numbers must be rounded to the nearest available value.

Learning Objectives

• Understand the role of symbols, significant digits, and exponents;
• Explain the result of 0.1 + 0.2 without referring to it as a language bug;
• Properly handle NaN, infinity, overflow, and underflow;
• Choose between tolerance-based comparisons, decimal fixed-point arithmetic, or higher precision based on the problem at hand.

1. Floating-Point Representation Is Like Scientific Notation

Binary floating-point numbers can be summarized as:

$$ (-1)^s \times significand \times 2^{exponent}. $$

The common IEEE 754 binary32 format uses 1 bit for the sign, 8 bits for the exponent field, and 23 bits for the mantissa (also called the significand); for normalized numbers, there is an implicit leading 1, giving a typical effective precision of 24 binary digits. Binary64 uses 1 bit for the sign, 11 bits for the exponent field, and 52 bits for the mantissa, yielding 53 bits of effective precision.

Exponent expansion increases the representable range, while the magnitude of the significand determines relative precision. The spacing of representable numbers along the real number line is not uniform: as the magnitude increases, the absolute gap between adjacent floating-point numbers generally grows larger.

Beyond normalized numbers, the standard includes:

• +0 and -0;
• subnormal numbers, which approach zero with reduced precision;
• +∞ and -∞;
• NaN (Not a Number), used to represent invalid or undefined results.

These special values participate in subsequent computations, and the system must decide whether to detect, propagate, or reject them.

2. Why 0.1 Cannot Be Represented Exactly

Finite binary fractions can only represent rational numbers whose denominators reduce to powers of 2. The decimal number 0.1=1/10 has a denominator containing a factor of 5, which results in an infinite repeating sequence when converted to binary, making it impossible to represent exactly. Instead, it must be rounded to the nearest floating-point value.

from decimal import Decimal

print(0.1 + 0.2)
print(0.1 + 0.2 == 0.3)
print(Decimal.from_float(0.1))
print(Decimal("0.1") + Decimal("0.2"))

Decimal.from_float(0.1) shows the exact decimal representation of a binary64 value; Decimal("0.1") demonstrates a value constructed directly from decimal text, without first undergoing binary floating-point rounding.

This behavior is an inherent limitation of finite binary representation and is not unique to Python. It also does not indicate an error in the IEEE 754 standard. Decimal systems face similar constraints, such as the inability to precisely represent 1/3 with a finite number of digits.

3. Every arithmetic operation may introduce rounding

The typical floating-point model operates by first computing the exact mathematical result, then mapping it to a representable value according to the current rounding rule. The default mode is "round to nearest, with ties broken by rounding to the nearest even digit," which helps prevent systematic bias over time.

Error types include:

• Representation error: the input value cannot be represented exactly;
• Rounding error: the result falls between two representable values;
• Overflow: the absolute value exceeds a finite range, potentially resulting in infinity;
• Underflow: the result is too small, entering the subnormal range or being rounded to zero.

Machine epsilon describes the relative spacing between representable values near 1, and is not a universal comparison threshold applicable across all magnitudes or algorithms. ULP (unit in the last place) varies with the magnitude of the value.

4. Equality comparisons must consider the context and meaning of the question

Floating-point numbers are not always impossible to compare. Exact comparison may be reasonable when comparing the same computed result, integer values within the exact range, or protocol-defined special constants. For approximated floating-point values derived through independent numerical paths, absolute and relative tolerances are typically used in combination: ==

$$ |a-b|\le \max(\text{abs_tol},\text{rel_tol}\cdot\max(|a|,|b|)). $$

import math

result = 0.1 + 0.2
print(math.isclose(result, 0.3, rel_tol=1e-12, abs_tol=1e-15))

Absolute tolerance handles values near zero, while relative tolerance handles different magnitudes. Thresholds should come from measurement precision, algorithmic errors, and business tolerance, never mechanically copy 1e-9.

NaN is unequal to any value, including itself. Use math.isnan to detect NaN; do not write x == float("nan").

5. Choices for Amounts, Counts, and Scientific Computations

• Counts: Prefer integers, with attention to range and overflow;
• Currency: Use integers representing the smallest monetary unit, or well-defined decimal fixed-point types;
• Scientific computing: binary64 is often a reasonable starting point, but precision requirements (especially error budgets) determine whether higher precision is needed;
• Machine learning: Lower precision can improve throughput and reduce GPU memory usage, though techniques like loss scaling, mixed precision, and numerically stable operators are typically required;
• Protocols and storage: Must explicitly define formats, rounding behavior, special values, and cross-language compatibility.

Decimal is not automatically "exactly precise." It still operates under contextual precision and rounding rules, and must approximate infinite decimal results; it addresses decimal semantics and controlled rounding, not the elimination of finite representation.

6. Operator Order Affects Results

Floating-point addition typically does not satisfy the associative property:

$$ (a+b)+c \ne a+(b+c). $$

a = 1e16
b = -1e16
c = 1.0

print((a + b) + c)  # 1.0
print(a + (b + c))  # 0.0 (Common binary64 Result)

In the second order, the small value of b + c may be lost during rounding. Parallel reduction changes the summation order, so floating-point programs with identical inputs can produce different results at the last bit due to different thread partitions. When bit-level reproducibility is required, the algorithm and execution environment must be explicitly constrained.

Common Misconceptions

• Double precision has 52 decimal digits of precision: 52 refers to the number of binary bits in the significand, not decimal digits.
• Epsilon is the smallest global positive number: Machine epsilon, the smallest normalized number, and the smallest subnormal number are distinct concepts.
• Decimal eliminates all rounding errors: Even decimal arithmetic is constrained by finite precision and rounding context.
• Floating-point exceptions always throw an exception: Many environments generate NaN or infinity and continue computation without interruption.

Exercise

1. Identify which simple fractions can be precisely represented as finite binary decimals and explain the pattern.
2. Use math.nextafter to examine the floating-point spacing near 1, 1e10, and 1e-10.
3. Compare two designs for an amount settlement module: "cents as integers" versus Decimal.
4. Construct a test that introduces a NaN into a sorting, aggregation, or serialization workflow and document the runtime behavior.

Summary

Floating-point numbers trade a vast dynamic range for a finite number of bits, at the cost of discrete representation, rounding, and special values. Don't fear decimals; instead, align your numeric types, comparison rules, and error tolerance with the semantics of the problem at hand.

The next lesson will continue asking: if both formulas use binary64, why does one remain stable while a mathematically equivalent one overflow or lose all significant digits?