Condition Numbers

In the previous section, we saw that floating-point inputs carry small relative errors bounded by machine epsilon, and that subtracting nearly equal numbers amplifies those errors catastrophically. The condition number makes this precise — and applies to any computation, not just subtraction.

Condition of $f$ at $x$ ¶

When we evaluate $f(x)$ but only have an approximation $x^*$ to the input, the output $f(x^*)$ may be far from $f(x)$ . The condition number is the worst-case ratio of relative error in the output to relative error in the input.

Simplified Formula via Taylor’s Theorem¶

Proposition 1 (Condition Number for Function Evaluation)

Consider the problem of evaluating a differentiable function $f$ at a point $x$ . The condition number of this problem is:

\kappa = \left| \frac{x f'(x)}{f(x)} \right|

(2)

Since $x^*$ is close to $x$ , we can write $x^* = x + h$ and apply Taylor’s theorem:

f(x^*) = f(x + h) \approx f(x) + f'(x)(x - x^*)

(3)

This gives us:

f(x) - f(x^*) \approx f'(x)(x - x^*)

(4)

Substituting into the condition number definition:

\kappa = \sup_x \left| \frac{x f'(x)}{f(x)} \right|

(5)

Condition Number of a Differentiable Function¶

For a differentiable function $f$ , we can derive the condition number formula directly from the definition of relative error, without appealing to Taylor’s theorem.

Absolute Condition Number¶

The absolute condition number measures the ratio of absolute error in the output to absolute error in the input:

\hat{\kappa} = |f'(x)|.

(6)

If $|f'(x)|$ is large, a small absolute change in $x$ produces a large absolute change in $f(x)$ .

Derivation from Relative Errors¶

Starting from Definition 1, suppose $x$ is perturbed by a small amount $\Delta x$ . The relative change in $x$ is $\frac{|\Delta x|}{|x|}$ , while the relative change in the output is $\frac{|f(x + \Delta x) - f(x)|}{|f(x)|}$ . Taking the ratio:

\kappa = \frac{|f(x + \Delta x) - f(x)| / |f(x)|}{|\Delta x| / |x|} = \frac{|x|}{|f(x)|} \cdot \frac{|f(x + \Delta x) - f(x)|}{|\Delta x|}.

(7)

The last factor is a difference quotient. In the limit $\Delta x \to 0$ it becomes $|f'(x)|$ , recovering the formula from Proposition 1.

Remark 1

The formula $\kappa = |x f'(x) / f(x)|$ can be read as the ratio of the logarithmic derivative of $f$ to the logarithmic derivative of $x$ :

\kappa = \left| \frac{(\ln f)'}{(\ln x)'} \right|, \qquad \text{since } (\ln f)' = \frac{f'}{f} \text{ and } (\ln x)' = \frac{1}{x}.

(8)

This makes intuitive sense: the logarithmic derivative measures the infinitesimal rate of relative change, so the condition number is the ratio of the relative rate of change of the output to that of the input.

Note that if $f$ has a zero at $x$ , then $\kappa \to \infty$ . This is not necessarily because the computation is genuinely sensitive. The absolute error may be perfectly well-behaved, but relative error is undefined at a zero of $f$ .

Several Variables¶

For a differentiable map $f : \mathbb{R}^m \to \mathbb{R}^n$ , the relative condition number generalises to

\kappa = \frac{\|J(x)\|}{\|f(x)\| / \|x\|}

(9)

where $J(x)$ is the Jacobian matrix of $f$ at $x$ and $\|\cdot\|$ denotes the induced matrix norm. The one-variable formula is the special case $m = n = 1$ .

Examples¶

Example 1 (Square Root (Well-Conditioned))

Consider $f(x) = \sqrt{x}$ with $f'(x) = \frac{1}{2\sqrt{x}}$ .

Near $\bar{x} = 1$ , we have $\bar{y} = f(\bar{x}) = 1$ . Using a Taylor approximation:

y - \bar{y} = \sqrt{x} - 1 \approx \frac{1}{2}(x - 1) = \frac{1}{2}(x - \bar{x})

(10)

Variations in $y$ are always smaller than variations in $x$ . Computing the condition number:

\kappa = \left| \frac{x f'(x)}{f(x)} \right| = \left| \frac{x \cdot \frac{1}{2\sqrt{x}}}{\sqrt{x}} \right| = \frac{1}{2}

(11)

Result: $\kappa = 1/2$ — evaluating $\sqrt{x}$ is well-conditioned.

Example 2 (Tangent Near π/2 (Ill-Conditioned))

Consider $f(x) = \tan(x)$ near $x = \frac{\pi}{2}$ .

Take two points:

x_1 = \frac{\pi}{2} - 0.001, \quad x_2 = \frac{\pi}{2} - 0.002

(12)

Then $|x_1 - x_2| = 0.001$ , but $|f(x_1) - f(x_2)| \approx 500$ . A tiny input change causes a huge output change!

Computing the condition number:

\kappa = \left| \frac{x f'(x)}{f(x)} \right| = \left| \frac{x}{\cos(x)\sin(x)} \right| = |2x \csc(2x)| \to \infty \text{ as } x \to \pi/2

(13)

Result: $\kappa \to \infty$ — evaluating $\tan(x)$ near $\frac{\pi}{2}$ is ill-conditioned.

Example 3 (Logarithm Near 1 (Ill-Conditioned))

Consider $f(x) = \ln(x)$ near $x = 1$ .

\kappa = \left| \frac{x f'(x)}{f(x)} \right| = \left| \frac{x \cdot (1/x)}{\ln(x)} \right| = \frac{1}{|\ln(x)|}

(14)

As $x \to 1$ , we have $\ln(x) \to 0$ , so $\kappa \to \infty$ .

Result: $\kappa \to \infty$ — evaluating $\ln(x)$ near $x = 1$ is ill-conditioned.

But note why this happens: the condition number blows up because $f(x) = \ln(x) \to 0$ , so the relative error $|f(x) - f(x^*)|/|f(x)|$ has a denominator going to zero. The absolute error is perfectly well-behaved — $|f'(1)| = 1$ , so a small input perturbation produces a small absolute output perturbation. The “ill-conditioning” here is really that relative error is undefined at a zero of $f$ . This happens for any function near one of its roots, and is an artifact of measuring error in relative terms rather than a genuine sensitivity of the computation.

Condition Number of Subtraction¶

The condition number formula $\kappa = |xf'(x)/f(x)|$ works for functions of one variable. For a function of two variables $g(a, b) = a - b$ , we generalize: the condition number measures how the relative error in $g$ relates to the relative errors in $a$ and $b$ .

Example 4 (Subtraction is Ill-Conditioned When

a \approx b

)

Consider $g(a, b) = a - b$ with small perturbations $\tilde{a} = a(1 + \varepsilon_1)$ and $\tilde{b} = b(1 + \varepsilon_2)$ :

\tilde{g} - g = a\varepsilon_1 - b\varepsilon_2

(15)

The relative error in the result is:

\frac{|\tilde{g} - g|}{|g|} \leq \frac{|a| + |b|}{|a - b|} \cdot \max(|\varepsilon_1|, |\varepsilon_2|)

(16)

The condition number of subtraction is therefore:

\kappa = \frac{|a| + |b|}{|a - b|}

(17)

When $a$ and $b$ are well-separated: $\kappa \approx \mathcal{O}(1)$ — subtraction is well-conditioned.
When $a \approx b$ : $\kappa \to \infty$ — subtraction is ill-conditioned.

This is precisely the catastrophic cancellation we saw in the floating-point chapter, now explained through the lens of condition numbers.

Condition Number of the Finite Difference¶

We can now give a condition number explanation of the finite difference trade-off.

Example 5 (Finite Difference Condition Number)

The forward difference $\frac{f(x_0 + h) - f(x_0)}{h}$ requires computing $a - b$ where $a = f(x_0 + h)$ and $b = f(x_0)$ .

Applying the subtraction condition number with $a \approx b \approx f(x_0)$ :

\kappa \approx \frac{|f(x_0 + h)| + |f(x_0)|}{|f(x_0 + h) - f(x_0)|} \approx \frac{2|f(x_0)|}{h|f'(x_0)|}

(18)

As $h \to 0$ , this condition number grows like $1/h$ . Each input carries relative error $\mu$ (machine epsilon), so the round-off contribution to the finite difference is:

\text{round-off error} \approx \kappa \cdot \mu \cdot |f'(x_0)| \approx \frac{2\mu|f(x_0)|}{h}

(19)

This matches exactly the round-off term we derived from the floating-point analysis — but now we see it as a conditioning problem: the subtraction step is ill-conditioned for small $h$ .

Summary¶

The condition number $\kappa$ measures how much a problem amplifies input errors. It is a property of the mathematical problem, not the algorithm — if $\kappa$ is large, no algorithm can avoid accuracy loss. But what if $\kappa$ is small and we still get poor results? Then the problem isn’t to blame — the algorithm is. In the next section, we make this precise with the concept of stable and unstable algorithms.