Floating Point Numbers

Number Systems¶

Integers in Base 10¶

We typically write integers in base 10. A number $d_N d_{N-1} \dots d_1 d_0$ (where $d_i \in \{0, 1, \dots, 9\}$) represents:

Binary Representation¶

Computers use base 2 (binary), where digits $d_i \in \{0, 1\}$. A binary number $d_N d_{N-1} \dots d_1 d_0$ represents:

Signed Integers: Two’s Complement¶

Negative integers use two’s complement representation. For an $N$-bit signed integer $a = d_{N-1}d_{N-2}\dots d_1 d_0$:

Fixed Point Notation¶

To represent fractions, we allow digits after a radix point:

represents:

where $b$ is the base.

With finitely many digits, some numbers cannot be represented exactly (e.g., $\frac{1}{3}$ or $\pi$).

Floating Point Numbers¶

For scientific computing, we need to represent numbers of vastly different magnitudes—from Avogadro’s number ($6.02 \times 10^{23}$) to Planck’s constant ($6.63 \times 10^{-34}$).

Scientific notation allows the radix point to “float”:

In binary:

Note: In normalized binary scientific notation, the digit before the radix point is always 1, so we don’t need to store it!

IEEE 754 Standard¶

A floating point number consists of three parts:

• Sign bit $S$: 0 for positive, 1 for negative

• Exponent $E$: Shifted to allow negative exponents

• Mantissa/Fraction $m$: The significant digits

Single Precision (32-bit)¶

The value represented is:

where $d_i$ are the mantissa bits and $E$ is the stored exponent.

Double Precision (64-bit)¶

Integer Interpretation of Floating Point¶

The same 32 bits can be interpreted as either a float or an integer. Given the floating point representation $(S_x, E_x, m_x)$, the integer value is:

where $M_x = 2^{23} m_x$ is the integer value of the mantissa bits.

This dual interpretation is exploited in fast numerical algorithms like the fast inverse square root.

Rounding Error (Machine Epsilon)¶

Given a real number $x$, its floating point representation $\text{fl}(x)$ satisfies:

where $\mu$ is the machine epsilon (or unit roundoff):

with $t$ being the number of mantissa bits.

Application: The Finite Difference Trade-off¶

In the approximation theory chapter, we observed that finite difference errors increase for very small step sizes. Now we can explain why.

The Total Error¶

When computing the forward difference approximation:

we make two types of errors:

1. Truncation error from Taylor’s theorem: $\frac{h}{2}|f''(\xi)|$

2. Round-off error from floating-point arithmetic

For the round-off error: when $h$ is small, $f(x+h) \approx f(x)$, so we’re subtracting two nearly equal numbers. If both values have relative error $\mu$, the subtraction has absolute error roughly $2\mu|f(x)|$. Dividing by $h$ amplifies this to:

The total error is:

The Optimal Step Size¶

To minimize $E(h)$, differentiate and set to zero:

Solving (and assuming $|f(x)| \approx |f''(\xi)|$ for simplicity):

Connection to the Error Framework¶

This is a perfect illustration of our error analysis framework:

• Backward error (truncation): How much did we perturb the mathematical problem? We approximated $f'$ by a secant slope—error $O(h)$.

• Forward error (round-off amplification): How much did floating-point errors affect our answer? Error $O(\mu/h)$.

The condition number of the operation “subtract then divide by $h$” grows like $1/h$, which is why round-off errors get amplified for small $h$.