11.1 Limits, Derivatives, and the Chain Rule: Describing Local Change

How much does a prediction change after a small parameter adjustment? What is the instantaneous velocity along a continuous position curve? Both questions begin with behavior near a point, followed by the limits of that local approximation.

Learning Objectives

• Use limits to understand derivatives, rather than just memorizing derivative rules;
• Distinguish between continuity, differentiability, and the appearance of smoothness in numerical data;
• Break down composite functions using the product and chain rules;
• Understand the boundaries between finite differences, symbolic differentiation, and automatic differentiation.

1. Limits Describe Behavior Near a Point

When $x$ approaches $a$, if $f(x)$ approaches $L$, we write:

$$\underset{x\to a}{lim}f(x)=L.$$

Limits focus on behavior near a point, without requiring that $f(a)$ exists or that $f(a)=L$. They allow us to rigorously discuss instantaneous velocity: we can't literally divide displacement by zero in a zero-time interval, but we can examine the limit of average velocity as the time interval shrinks toward zero.

For example, $f(x)=(x^2-1)/(x-1)$ is undefined at 1, but equals $x+1$ nearby and has limit 2. Precisely, a finite limit means that for every $\epsilon >0$ there is a $\delta >0$ such that $0<|x-a|<\delta$ implies $|f(x)-L|<\epsilon$. Here $\delta =\epsilon$ works; the definition deliberately excludes $a$ itself.

A function is continuous at $a$ if all three conditions are met:

1. $f(a)$ is defined;
2. $\underset{x\to a}{lim}f(x)$ exists;
3. These two values are equal.

Discrete computers can only sample a finite number of points. A smooth-looking graph does not imply mathematical continuity, sampling might miss sharp corners, discontinuities, or high-frequency oscillations.

2. The Derivative Is a Local Linear Approximation

The derivative is defined as the limit of a difference quotient:

$$f^{\prime }(x)=\underset{h\to 0}{lim}\frac{f(x+h)-f(x)}{h}.$$

Geometrically, it represents the slope of the tangent line; in modeling terms, it provides a first-order approximation to small perturbations:

$$f(x+\mathrm{\Delta }x)\approx f(x)+f^{\prime }(x)\mathrm{\Delta }x.$$

This requires differentiability at the point. More precisely, the remainder divided by $|\mathrm{\Delta }x|$ tends to zero; it does not guarantee small error for every finite step. For $f(x)=x^2$, expansion gives $x^2+2x\mathrm{\Delta }x+(\mathrm{\Delta }x{)}^2$, with $(\mathrm{\Delta }x{)}^2$ omitted by the first-order model.

Common rules include:

$$\frac{d}{dx}{x}^n=n{x}^{n-1},\frac{d}{dx}{e}^x=e^x,\frac{d}{dx}\sin x=\cos x.$$

More generally, the derivative of $c{e}^x$ is itself; it's not accurate to claim that $e^x$ is the only function with this property.

Apply the power rule where the function and right-hand expression are defined. Integer $n\ge 1$ works for all real inputs. The constant $x^0=1$ has derivative zero directly; do not evaluate $0x^{-1}$ at zero. Trigonometric derivative formulas use radians.

Differentiability implies continuity, but continuity does not imply differentiability. For example, $|x|$ is continuous at zero, yet its left and right derivatives are $-1$ and $1$, respectively. ReLU is also non-differentiable at zero; software frameworks adopt a subgradient convention at such points. This is an implementation choice, not a sudden restoration of differentiability in the classical sense.

3. Composite Systems Require the Chain Rule

If $g$ is differentiable at $x$, $f$ at $g(x)$, and $y=f(g(x))$, then:

$$\frac{dy}{dx}=f^{\prime }(g(x))\cdot g^{\prime }(x).$$

For example:

$$y=(3x+1{)}^2,$$

the derivative of the outer square is $2(3x+1)$, and the derivative of the inner linear function is $3$, so:

$$y^{\prime }=6(3x+1).$$

Software systems are often composed of multiple layered functions. In neural networks, a computation graph represents the operations, forward propagation evaluates its nodes, and backpropagation repeatedly applies the chain rule from output to input. Its key advantage isn't simply "computing one derivative"; it's the ability to reuse intermediate results, enabling efficient gradient computation of a scalar output with respect to a large number of parameters.

Writing the example as two intermediate nodes lets us inspect one backward pass. bar_v means the output derivative with respect to v; contributions through repeated uses of the same variable must be added:

x = 2.0
v = 3 * x + 1
y = v * v
bar_y = 1.0
bar_v = bar_y * (v + v)  # Both inputs of the product depend on v.
bar_x = bar_v * 3
print(y, bar_x)  # 49.0 42.0

This Python 3 code writes the local derivatives explicitly; it is not a general autodiff engine. A framework must also retain needed intermediate values, manage the graph, and handle unsupported operations.

You should also master the product rule:

$$(fg{)}^{\prime }=f^{\prime }g+f{g}^{\prime }.$$

Miswriting the product as $f^{\prime }{g}^{\prime }$ is one of the most common mechanical errors.

4. Multivariate Functions, Partial Derivatives, and Gradients

For a function $f(x_1,\dots ,x_n)$, a partial derivative changes only one variable while holding all others constant:

$$\frac{\partial f}{\partial x_i}.$$

The gradient is formed by collecting all partial derivatives:

$$\mathrm{\nabla }f(\mathbf{x})=\left[\begin{array}{c}\partial f/\partial x_1\\ ⋮\\ \partial f/\partial x_n\end{array}\right].$$

For differentiable functions, the directional derivative along a unit vector $\mathbf{u}$ is given by:

$$D_{\mathbf{u}}f=\mathrm{\nabla }{f}^{\mathsf{T}}\mathbf{u}.$$

The gradient points in the direction of steepest ascent; the negative gradient points in the direction of steepest descent. The term "steepest" depends on the Euclidean norm; altering the metric of the space changes what is considered the steepest direction.

When the gradient is zero, every first-order directional derivative is zero, so there is no unique steepest direction. Existence of partial derivatives alone does not imply differentiability: define $f(x,y)=xy/(x^2+y^2)$ away from the origin and 0 at it. Both coordinate partials at the origin are zero, but approaching along $y=x$ gives $1/2$, so the function is not even continuous there.

5. Finite differences are approximations, not automatic differentiation

The central difference formula is:

$$f^{\prime }(x)\approx \frac{f(x+h)-f(x-h)}{2h}.$$

from math import isfinite

def f(x: float) -> float:
    return x**3 - 2 * x

def central_difference(x: float, h: float) -> float:
    if not isfinite(h) or h <= 0:
        raise ValueError("h must be positive and finite")
    return (f(x + h) - f(x - h)) / (2 * h)

x = 1.5
exact = 3 * x**2 - 2
for h in [1e-1, 1e-3, 1e-5, 1e-8, 1e-12, 1e-20]:
    estimate = central_difference(x, h)
    print(f"{h:.0e} {estimate:.8f} {abs(estimate - exact):.3e}")

As $h$ decreases, truncation error initially drops; but when $h$ becomes too small, f(x+h) and f(x-h) become extremely close, and floating-point rounding and cancellation errors begin to dominate. Therefore, "make $h$ as small as possible" is not a reliable strategy.

For this cubic, exact-arithmetic central differences equal $f^{\prime }(x)+h^2$, giving truncation error $h^2$. The general second-order error bound needs sufficient smoothness, such as a bounded third derivative nearby. The true derivative here is 4.75. With typical binary64 Python floats, h=1e-20 makes x+h == x-h == x, so the estimate is zero and its error is 4.75. Last digits for other steps can vary by environment.

Three methods of differentiation must be distinguished:

Method	Result	Typical Use Case
Symbolic differentiation	Operates on expressions to derive a new formula	Algebraic derivation, simplification
Finite differences	Approximates derivatives using function values	Gradient verification, black-box functions
Automatic differentiation	Applies the chain rule to basic operations	Model training, differentiable programs

Automatic differentiation propagates derivatives of elementary operations through the chain rule without a finite-difference step. It still uses floating-point arithmetic and does not guarantee machine-precision final derivative error for every problem. Overflow, cancellation, ill conditioning, and nondifferentiable-point conventions affect results. See PyTorch’s autograd notes, including which branches and operations actually enter the computation.

Common Misconceptions

• The derivative is simply the function value divided by the input: The derivative is the limit of the ratio of local increments, not a direct division of output by input.
• Continuity implies a smooth slope: A function can be continuous at a point and still have a sharp corner, continuity does not guarantee differentiability.
• The gradient points to the global optimum: It only indicates the direction of first-order change at the current point; it does not guarantee a path to the global best solution.
• Smaller $h$ in numerical differentiation yields greater accuracy: There's a trade-off between truncation error and rounding error, using an extremely small $h$ can actually degrade precision.

Exercise

1. Use the definition of the derivative to derive the derivative of $f(x)=x^2$.
2. Apply the chain rule to differentiate $f(x)=\sin (x^2)$.
3. Compute the gradient of $f(x,y)=x^2+3xy+y^2$.
4. Run the central difference example to estimate the value of $h$ that minimizes error in the current environment.

Summary

Limits give a precise meaning to approaching a point, and derivatives provide a local linear model for functions. The chain rule breaks down complex compositions into products of local derivatives. Automatic differentiation systematizes these rules, but it doesn't decide whether a function is suitable for differentiation or how to handle points where the derivative doesn't exist.

The next lesson explores the other half of calculus: how local changes accumulate into totals, and how computers approximate analytical integrals.