11.3 Gradients and Optimization: From Descent Direction to Verifiable Solutions

You stand on a mountain with no view of the whole landscape, able to measure only the slope directly beneath your feet. Moving in the negative gradient direction is a natural starting point, but true optimization must answer deeper questions: how far to move, when to stop, whether constraints apply, and what exactly the final solution represents.

Learning Objectives

• Derive gradient descent from a first-order local approximation;
• Distinguish between convex problems, local minima, saddle points, and critical points;
• Understand the roles of learning rate, stochastic gradients, and momentum;
• Build a more reliable optimization checklist than simply "the loss decreased."

1. Getting a Descent Step from a Local Model

For a differentiable function $f$, near point $\mathbf{x}$, we have:

$$ f(\mathbf{x}+\Delta\mathbf{x}) \approx f(\mathbf{x})+\nabla f(\mathbf{x})^\mathsf{T}\Delta\mathbf{x}. $$

Under the Euclidean norm, the negative gradient points in the direction of steepest local descent, leading to the update:

$$ \mathbf{x}_{t+1} =\mathbf{x}_t-\eta_t\nabla f(\mathbf{x}_t), $$

where $\eta_t > 0$ is the step size or learning rate.

This derivation is local. If the step size is too large, the first-order approximation breaks down, potentially overshooting the minimum or diverging entirely. If it's too small, progress becomes sluggish, and the algorithm may appear to stall within numerical precision limits.

2. A Checkable Single-Variable Example

For the function $$ f(x)=x^2+3x+2, $$ the derivative is $f'(x)=2x+3$, and the analytical minimum occurs at $x=-1.5$.

def loss(x: float) -> float:
    return x * x + 3 * x + 2

def gradient(x: float) -> float:
    return 2 * x + 3

x = 8.0
learning_rate = 0.2

for step in range(100):
    grad = gradient(x)
    if abs(grad) < 1e-10:
        break
    x -= learning_rate * grad

print(step, x, loss(x), gradient(x))

This problem is a strictly convex quadratic function with a very clean structure. Extending its behavior directly to neural networks ignores non-convexity, stochastic gradient descent, pathological curvature, and parameter redundancy.

3. Critical Points Do Not Necessarily Mean Minima

A point where $\nabla f(\mathbf{x}) = 0$ is called a critical point, and it may represent:

• A local minimum;
• A local maximum;
• A saddle point;
• A higher-order flat point.

In one dimension, the second derivative can help determine the nature of the critical point. For multivariate functions, the Hessian matrix is composed of second-order partial derivatives:

$$ H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}. $$

Near a critical point, if the Hessian is positive definite, it typically indicates a strict local minimum; if negative definite, a strict local maximum; if indefinite, a saddle point. A semi-definite Hessian may not be sufficient to conclude the nature of the point based solely on second-order conditions.

If the objective function is convex, any local minimum is also a global minimum. If the function is strictly convex, there is at most one minimum. However, gradient descent alone does not guarantee convergence: additional constraints on smoothness, step size, and other factors are required. For general non-convex functions, it is not safe to claim that the algorithm will converge to a local optimum, instead, it might get stuck near saddle points, oscillate, or diverge entirely.

4. Learning Rate Control for the Trustworthy Range of Local Models

Common learning rate strategies:

• Fixed learning rate: simple, but requires manual tuning;
• Decaying learning rate: reduces oscillations in later stages;
• Line search: attempts to satisfy the sufficient decrease condition;
• Adaptive methods: adjust effective step sizes for different parameters based on historical gradients.

Adam, RMSprop, and momentum-based SGD all leverage historical information, but they differ in behavior, memory overhead, and convergence properties. It is incorrect to assume that all of them are equivalent across all problems simply because they ultimately subtract some update amount.

One form of momentum is:

$$ \mathbf{v}_{t+1} = \beta \mathbf{v}_t + \nabla f(\mathbf{x}t), \qquad \mathbf{x} = \mathbf{x}t - \eta \mathbf{v}. $$

This formulation accumulates velocity when gradients point in the same direction, helping to smooth out oscillations in narrow valleys. However, it also introduces new hyperparameters and risks of overshooting.

5. Full-Batch, Stochastic, and Mini-Batch Gradients

The objective in machine learning is often expressed as the average of sample losses:

$$ F(\mathbf{w})=\frac{1}{N}\sum_{i=1}^{N}\ell_i(\mathbf{w}). $$

• Full-batch gradient uses all samples per step, providing stable updates but at a high computational cost;
• Stochastic gradient uses only one sample per step, offering low cost but introducing significant noise;
• Mini-batch gradient strikes a balance between hardware parallelism, estimation noise, and throughput.

Randomness can sometimes help escape flat regions of the loss landscape, but it does not guarantee global optimality. The sampling strategy, batch size, and data ordering all influence the training trajectory.

6. Constrained Optimization Cannot Be Reduced to Simple Updates

If parameters must satisfy constraints (such as non-negative probabilities that sum to one) standard gradient descent may drive the solution outside the feasible region. Common approaches include:

• Projection gradient: After updating, project the parameters back into the feasible set;
• Reparameterization: Use structured functions like softmax to enforce constraints inherently;
• Lagrange multipliers: Analyze necessary conditions under equality constraints;
• Penalty or barrier methods: Transform constraint violations into the objective function.

Each method alters the geometric and numerical characteristics of the problem, rather than adding one formula and calling it a day alone.

7. Checklist for Verifying Optimization Results

Before stopping training, at least verify the following:

1. The objective value has decreased and is stabilizing;
2. The gradient norm or parameter update magnitude is sufficiently small;
3. No NaN, infinity, or obvious oscillations are present;
4. Constraints are being satisfied;
5. Validation set metrics align with the training objectives;
6. Results remain consistent when initial values or random seeds are changed;
7. Numerical gradient checks validate the automatic differentiation results.

Looking only at training loss hides issues like overfitting, data leakage, and incorrect objective functions. The optimizer only searches for what you've defined, it does not evaluate whether that objective reflects actual real-world needs.

Common Misconceptions

• Negative gradient points to global minimum: It only indicates the local descent direction at the current point.
• Gradient zero means training is successful: It could also signal a saddle point, a local maximum, or numerical underflow.
• Loss reduction means the model is correct: Incorrect labels, leaked features, or inappropriate objectives can still be optimized.
• Adaptive optimizers don’t need a learning rate: They still rely on global step size and other critical hyperparameters.

Exercise

1. Modify the learning rate in the single-variable example and observe the regions of stability, oscillation, and divergence.
2. Analyze the gradient and Hessian of $f(x,y)=x^2-y^2$ at the origin and explain why it is a saddle point.
3. Write the Lagrangian function for the minimization problem subject to the constraint $x+y=1$.
4. Design a stopping condition for a training loop to prevent reliance solely on a fixed number of iterations.

Summary

Gradient descent stems from a local linear approximation, and the learning rate determines how far we trust that approximation in one step. Convexity provides a global structure, while the Hessian describes local curvature. Stochastic gradients and momentum alter computational cost and trajectory. Truly reliable optimization results also require constraint checks, numerical stability checks, and validation against task-specific metrics.

The final chapter addresses two kinds of boundaries often hidden behind formulas: how information is actually measured, and why the same mathematical expression can yield completely different results when evaluated on finite-precision computers.