3.3 Policy Gradient, Actor-Critic, and Safety Evaluation: Optimizing Rewards Also Optimizes Reward Vulnerabilities

Q-values are getting larger and larger, and the robot still needs to output continuous turning angles. You switch to directly learning a distribution of actions, but find that the same policy sometimes reaches the goal quickly and other times falls into traps, with drastic fluctuations in gradient direction. Even more problematic is that it learns to repeatedly trigger rewards near the end, rather than properly terminating the task.

Policy optimization can handle stochastic and continuous actions but doesn't automatically understand intent; estimating variance, action constraints, and reward pitfalls all must be manually incorporated into the algorithm design.

This lesson's objectives

• Derive the log-derivative update for REINFORCE;
• Use baseline/advantage to reduce variance;
• Understand the bootstrap trade-off in actor-critic methods;
• Design independent assessment, offline inspection, and safety constraints.

1. Directly Optimize the Parameterized Strategy

Let the policy be $\pi_\theta(a\mid s)$, objective:

$$ J(\theta)=E_{\tau\sim\pi_\theta}[G_0]. $$

The basic form of policy gradient:

$$ \nabla_\theta J(\theta)=E[ G_t\nabla_\theta\log\pi_\theta(A_t\mid S_t)]. $$

REINFORCE collects complete trajectories and uses sampled returns as weights, increasing the log probability for positive returns and decreasing it for negative ones.

2. Correct the gradient direction in the previous draft

The log-probability gradient of the selected action under a Softmax linear policy is:

$$ \nabla_W\log\pi(a\mid s) =s,(onehot(a)-\pi(\cdot\mid s))^T. $$

import numpy as np

class LinearReinforce:
    def __init__(self, feature_dim, n_actions, gamma, lr, seed=0):
        self.w = np.zeros((feature_dim, n_actions))
        self.gamma = gamma
        self.lr = lr
        self.rng = np.random.default_rng(seed)

    def probabilities(self, features):
        logits = features @ self.w
        logits -= logits.max()
        exp = np.exp(logits)
        return exp / exp.sum()

    def act(self, features):
        p = self.probabilities(features)
        return int(self.rng.choice(len(p), p=p))

    def update(self, trajectory):
        returns = np.empty(len(trajectory))
        running = 0.0
        for t in reversed(range(len(trajectory))):
            _, _, reward = trajectory[t]
            running = reward + self.gamma * running
            returns[t] = running

        baseline = returns.mean()
        for (features, action, _), g in zip(trajectory, returns):
            p = self.probabilities(features)
            grad_logits = -p
            grad_logits[action] += 1.0
            self.w += self.lr * (g - baseline) * np.outer(
                features, grad_logits
            )

The old code used probs - onehot but performed gradient ascent, which actually decreased the probability of selected actions; it also referenced an undefined self.gamma. Teaching code must clearly explain symbols and dimensions.

Here, the same-trajectory mean baseline is used only for illustrative purposes; in practice, baseline design must avoid improper action dependencies and consider bias/variance.

3. Baseline Condition That Doesn't Alter the Expected Gradient

Subtract a baseline that depends only on state and not on the chosen action from return:

$$ (G_t-b(S_t))\nabla\log\pi(A_t\mid S_t), $$

Under standard conditions, it can reduce variance without altering the expected gradient. If $b\approx V^\pi$, the difference can be viewed as an advantage estimate.

Return normalization and reward scaling alter optimization values; whether to maintain the target must be analyzed on a case-by-case basis and should not be conflated with legitimate baselines.

4. Actor-Critic

• actor: policy $\pi_\theta$;
• critic: Estimate $V_w$ or $Q_w$;
• TD error:

$$ \delta_t=R_{t+1}+\gamma V_w(S_{t+1})-V_w(S_t). $$

Using $\delta_t$ to approximate the advantage update for the actor enables online learning with lower variance, but the critic's bootstrap bias carries over into the policy update.

n-step/GAE balances bias and variance. Parameter selection must be made through multiple seeds and independent evaluations, not simply "the larger lambda is, the more accurate it is."

5. Continuous actions require a probability distribution and boundaries

A common strategy is to output the mean and standard deviation of a Gaussian, then sample the action. Note that:

• Standard deviation remains positive and constrains the value range;
• When actions are bounded, tanh squashing alters the log-probability and requires a Jacobian correction;
• Exploration noise must be within the equipment's safe operating range;
• Correlations across different dimensions may not be expressible using independent Gaussians;
• The deployment-time mean action and the training sampling strategy are not from the same distribution.

"Policy Gradient naturally handles continuous actions" only indicates ease of parameterization, not that constraints and training stability are automatically resolved.

6. Trust region and clipped objective

Large policy updates can cause too big a discrepancy between old and new trajectory distributions, leading to estimation distortion. Methods like TRPO/PPO limit policy changes. Clipping provides stability but does not guarantee monotonic performance improvement, nor can it correct errors in advantage or data realization.

Monitor approximate KL, clip fraction, entropy, value loss, explained variance, and gradient norm, and verify with true returns.

7. Distributional Out-of-Distribution Actions in Offline RL

When training solely on historical logs, the policy can't safely try new actions. If the learned Q-values overestimate the rewards for rare actions in the data, the policy will exploit extrapolation errors.

Needed:

• Clearly define the behavior policy and action propensity (if available);
• Coverage/overlap diagnosis;
• Maintain conservative value or strategy constraints;
• Assumptions and confidence intervals for off-policy evaluation;
• Simulator/low-volume launch with security gate;
• Don't treat ordinary supervised learning validation scores as strategy values.

Without critical action coverage, data can't answer the consequences of new strategies, and algorithms can't generate counterfactual evidence.

8. Security and Constraint Decision-making

Real robots can't learn by falling down a thousand times. Instead use:

• simulator and domain randomization;
• action shield/rule constraints;
• constrained MDP and cost budget;
• safe set and fallback controller;
• Expand the state/action range in stages;
• Manually approve high-risk actions;
• kill switch and event replay

Setting an accident as a large negative reward isn't always sufficient: accidents can still occur during exploration, and function approximation might underestimate the probability of rare disasters.

9. Evaluate the Agreement

Fixed policy snapshot, no evaluation during episode exploration/learning
Multiple training seeds and independent environment seeds
Mean, quantiles, failure rate, and worst-case scenarios
In-distribution, out-of-distribution, and perturbation scenarios
Rewards composition and true task metrics are reported separately.
Constraint violations, interventions, and fallback counts
Compared to rules, randomness, and existing strategy baselines
Define stop and rollback conditions before going live

Online A/B testing is still an intervention and requires consideration of units, spillover effects, compliance, and safety exposure.

Common Misconceptions

• Policy Gradient Doesn't Require a Value Function: Pure REINFORCE doesn't need one, but actor-critic relies on the critic.
• Baseline Changes the Optimization Objective: A legitimate action-independent baseline primarily reduces variance.
• Reward increases lead to better task performance: The agent might exploit proxy vulnerabilities.
• Offline logs are large enough to evaluate any policy: Not identifiable when action coverage is lacking.

Practice

1. Manually derive the softmax log-policy gradient to identify symbol errors in the old code.
2. Compare the variance of no baseline, constant baseline, and learned value baseline.
3. Correct the log-probability of the squashed distribution for bounded continuous actions.
4. Design cost constraints, shielding, fallback mechanisms, and launch gatekeeping for robots.

Summary

Policy gradient turns action probabilities into an optimizable quantity, while actor-critic reduces variance through value estimation. The more directly you optimize long-term returns, the more critical it is to review rewards, coverage, and safety boundaries, because the algorithm will exploit every flaw you leave behind.

Next chapter: Introduction to Supervised Learning, data is no longer generated continuously by an online policy, but how data is split, labeled, and evaluated still determines what the model learns.