3.3 Policy Gradient, Actor-Critic, and Safety Evaluation: Optimizing Rewards Also Optimizes Reward Vulnerabilities

A finite Q-table cannot enumerate continuous steering angles; a parameterized action distribution offers another representation. Random trajectories produce noisy returns and gradients. If the reward can be claimed repeatedly near the goal, the policy may learn to collect it without finishing the task.

Policy optimization can handle stochastic and continuous actions, but it does not infer the designer's intent. Variance reduction, action constraints, and reward-hacking checks must be part of the algorithm and evaluation design.

This lesson's objectives

• Derive the log-derivative update for REINFORCE;
• Use baseline/advantage to reduce variance;
• Understand the bootstrap trade-off in actor-critic methods;
• Design independent assessment, offline inspection, and safety constraints.

1. Directly Optimize the Parameterized Strategy

Let the policy be ${\pi }_{\theta }(a\mid s)$, objective:

$$J(\theta )=E_{\tau \sim {\pi }_{\theta }}[G_0].$$

For a finite complete trajectory, let $G_t$ restart discounting at time t and assume environment transitions do not depend on policy parameters:

$${\mathrm{\nabla }}_{\theta }J(\theta )=E_{\tau \sim {\pi }_{\theta }}[\sum _{t=0}^{T-1}{\gamma }^t{G}_t{\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(A_t\mid S_t)].$$

REINFORCE weights trajectory gradients by sampled returns. The explicit ${\gamma }^t$ factor matches $J=E[G_0]$; an expression using discounted state visitation can absorb it, but silently dropping it changes the estimator. A positive-return term contributes toward increasing the selected action's log probability; contributions across a trajectory combine through shared parameters.

2. Confirm the Gradient Direction First

The log-probability gradient of the selected action under a Softmax linear policy is:

$${\mathrm{\nabla }}_W\log \pi (a\mid s)=s(onehot(a)-\pi (\cdot \mid s){)}^T.$$

This Python 3 + NumPy fragment (partial) uses finite discrete actions to demonstrate gradient direction; section 5 covers continuous actions. features is a fixed-length vector, and trajectory is a list of (features, action, reward) tuples generated by the current policy. It must end at a true terminal state; truncated fragments need a separate tail-return estimate.

import numpy as np

class LinearReinforce:
    def __init__(self, feature_dim, n_actions, gamma, lr, seed=0):
        self.w = np.zeros((feature_dim, n_actions))
        self.gamma = gamma
        self.lr = lr
        self.rng = np.random.default_rng(seed)

    def probabilities(self, features):
        logits = features @ self.w
        logits -= logits.max()
        exp = np.exp(logits)
        return exp / exp.sum()

    def act(self, features):
        p = self.probabilities(features)
        return int(self.rng.choice(len(p), p=p))

    def update(self, trajectory, baseline=0.0):
        if not trajectory:
            return
        returns = np.empty(len(trajectory))
        running = 0.0
        for t in reversed(range(len(trajectory))):
            _, _, reward = trajectory[t]
            running = reward + self.gamma * running
            returns[t] = running

        gradient = np.zeros_like(self.w)
        for t, ((features, action, _), g) in enumerate(zip(trajectory, returns)):
            p = self.probabilities(features)
            grad_logits = -p
            grad_logits[action] += 1.0
            gradient += self.gamma ** t * (g - baseline) * np.outer(
                features, grad_logits
            )
        self.w += self.lr * gradient

Compute every gradient at the parameters of the sampling policy, accumulate them, and update once. baseline is a constant fixed before sampling, defaulting to zero. The current trajectory's mean return is not an action-independent baseline: for a one-step trajectory, subtracting that mean removes the entire learning signal. The training log can keep printing anyway.

For a minimal check, use one feature, two actions, and zero initial weights. Reward 1 for action 0 should raise its probability above 0.5; reward −1 should lower it. See Spinning Up's policy-gradient derivation for baseline and sampling-policy requirements.

3. Baseline Condition That Doesn't Alter the Expected Gradient

Subtract a baseline that depends only on state and not on the chosen action from return:

$$(G_t-b(S_t))\mathrm{\nabla }\log \pi (A_t\mid S_t),$$

Under standard conditions, it can reduce variance without altering the expected gradient. If $b\approx V^{\pi }$, the difference can be viewed as an advantage estimate.

Return normalization and reward scaling alter optimization values; whether to maintain the target must be analyzed on a case-by-case basis and should not be conflated with legitimate baselines.

4. Actor-Critic

• actor: policy ${\pi }_{\theta }$;
• critic: Estimate $V_w$ or $Q_w$;
• TD error:

$${\delta }_t=R_{t+1}+\gamma V_w(S_{t+1})-V_w(S_t).$$

Using ${\delta }_t$ to approximate the advantage update for the actor enables online learning with lower variance, but the critic's bootstrap bias carries over into the policy update.

n-step/GAE balances bias and variance. Parameter selection must be made through multiple seeds and independent evaluations, not simply "the larger lambda is, the more accurate it is."

5. Continuous actions require a probability distribution and boundaries

A common strategy is to output the mean and standard deviation of a Gaussian, then sample the action. Note that:

• Standard deviation remains positive and constrains the value range;
• When actions are bounded, tanh squashing alters the log-probability and requires a Jacobian correction;
• Exploration noise must be within the equipment's safe operating range;
• Correlations across different dimensions may not be expressible using independent Gaussians;
• The deployment-time mean action and the training sampling strategy are not from the same distribution.

"Policy Gradient naturally handles continuous actions" only indicates ease of parameterization, not that constraints and training stability are automatically resolved.

6. Trust region and clipped objective

Large policy updates can cause too big a discrepancy between old and new trajectory distributions, leading to estimation distortion. Methods like TRPO/PPO limit policy changes. Clipping provides stability but does not guarantee monotonic performance improvement, nor can it correct errors in advantage or data realization.

Monitor approximate KL, clip fraction, entropy, value loss, explained variance, and gradient norm, and verify with true returns.

7. Out-of-distribution actions in offline RL

When training solely on historical logs, the policy can't safely try new actions. If the learned Q-values overestimate the rewards for rare actions in the data, the policy will exploit extrapolation errors.

Needed:

• Clearly define the behavior policy and action propensity (if available);
• Coverage/overlap diagnosis;
• Maintain conservative value or strategy constraints;
• Assumptions and confidence intervals for off-policy evaluation;
• Simulator/low-volume launch with security gate;
• Don't treat ordinary supervised learning validation scores as strategy values.

Without critical action coverage, data can't answer the consequences of new strategies, and algorithms can't generate counterfactual evidence.

8. Safety and constrained decisions

Real robots can't learn by falling down a thousand times. Instead use:

• simulator and domain randomization;
• action shield/rule constraints;
• constrained MDP and cost budget;
• safe set and fallback controller;
• Expand the state/action range in stages;
• Manually approve high-risk actions;
• kill switch and event replay

Setting an accident as a large negative reward isn't always sufficient: accidents can still occur during exploration, and function approximation might underestimate the probability of rare disasters.

9. Evaluation protocol

Fixed policy parameters, no learning; sample or choose deterministic actions as specified by the evaluation protocol
Multiple training seeds and independent environment seeds
Mean, quantiles, failure rate, and worst-case scenarios
In-distribution, out-of-distribution, and perturbation scenarios
Rewards composition and true task metrics are reported separately.
Constraint violations, interventions, and fallback counts
Compared to rules, randomness, and existing strategy baselines
Define stop and rollback conditions before going live

Online A/B testing is still an intervention and requires consideration of units, spillover effects, compliance, and safety exposure.

Common Misconceptions

• Policy Gradient Doesn't Require a Value Function: Pure REINFORCE doesn't need one, but actor-critic relies on the critic.
• Baseline Changes the Optimization Objective: A legitimate action-independent baseline primarily reduces variance.
• Reward increases lead to better task performance: The agent might exploit proxy vulnerabilities.
• Offline logs are large enough to evaluate any policy: Not identifiable when action coverage is lacking.

Practice

1. Derive the softmax log-policy gradient and check its direction with one-step positive and negative rewards.
2. Compare the variance of no baseline, constant baseline, and learned value baseline.
3. Correct the log-probability of the squashed distribution for bounded continuous actions.
4. Design cost constraints, shielding, fallback mechanisms, and launch gatekeeping for robots.

Summary

Policy gradient turns action probabilities into an optimizable quantity, while actor-critic reduces variance through value estimation. The more directly you optimize long-term returns, the more critical it is to review rewards, coverage, and safety boundaries, because the algorithm will exploit every flaw you leave behind.

Next chapter: Introduction to Supervised Learning, data is no longer generated continuously by an online policy, but how data is split, labeled, and evaluated still determines what the model learns.