6.3 Gradient Boosting and XGBoost: Each New Tree Fits the Direction of Current Loss Reduction

While random forests have multiple trees making independent judgments and then averaging their outputs, the Developer Workshop has received a new challenge: a small number of high-cost samples are still being systematically underestimated. The team wants to iteratively refine the model by addressing the unmet portions of the objective function in each round.

Gradient Boosting builds an additive model. Instead of broadly "focusing on mistakes," each new tree fits the negative gradient of the loss function with respect to the predicted value. Under squared loss, this direction precisely aligns with the residuals.

Learning Objectives

• Explain gradient boosting using function gradients;
• Understand that squared loss residuals represent only a specific case;
• Comprehend XGBoost’s second-order approximation, leaf weights, and regularization terms;
• Properly partition training, validation, and test sets and apply early stopping;
• Analyze the joint impact of learning rate, tree capacity, sampling, calibration, and engineering cost.

1. Addition Model

Start from a constant model $F_0(x)$, and in the $t$-th round add a tree:

$$ F_t(x)=F_{t-1}(x)+\eta f_t(x), $$

where $\eta$ is the learning rate or shrinkage. The training objective is to minimize empirical loss subject to complexity constraints, not to "make each tree individually accurate."

In gradient boosting within the function space, compute the pseudo-residual for each sample:

$$ r_{it}=-\left.\frac{\partial\ell(y_i,F(x_i))}{\partial F(x_i)}\right|{F=F{t-1}}. $$

The new tree fits $r_{it}$ or the corresponding Newton step, and then the update is applied to the current model with a step size.

2. Residual is a Special Case of Squared Loss

If:

$$ \ell(y,F)=\frac{1}{2}(y-F)^2, $$

then:

$$ -\frac{\partial\ell}{\partial F} = y - F, $$

which is exactly the ordinary residual.

However, binary log loss is typically optimized in the logit or raw-score space. Let $ p = \sigma(F) $, where $ \sigma $ is the sigmoid function. The gradient of the negative log-likelihood for a single sample is:

$$ g = \frac{\partial\ell}{\partial F} = p - y, $$

so the negative gradient is $ y - p $, not the "misclassified as 1, correct as 0" label. For ranking, Poisson, quantile, and other objectives, the pseudo-residual takes a different form. Thus, the idea that "a new tree fits the error of the previous tree" is only a rough heuristic.

3. XGBoost's Second-Order Local Objective

At round $t$, XGBoost considers the objective:

$$ \mathcal L^{(t)}=\sum_i\ell(y_i,\hat y_i^{(t-1)}+f_t(x_i))+\Omega(f_t). $$

It performs a second-order Taylor expansion of the current prediction, discarding constant terms unrelated to the new tree:

$$ \widetilde{\mathcal L}^{(t)} =\sum_i\left[g_if_t(x_i)+\frac12h_if_t(x_i)^2\right]+\Omega(f_t), $$

where:

$$ g_i=\partial_{\hat y}\ell(y_i,\hat y_i),\qquad h_i=\partial^2_{\hat y}\ell(y_i,\hat y_i). $$

For a tree with $T$ leaves and leaf weights $w_j$, a common regularization form is:

$$ \Omega(f)=\gamma T+\frac12\lambda\sum_{j=1}^Tw_j^2 $$

(Implementations may also support L1-type terms). For the set of samples falling into leaf $j$, denote $G_j=\sum_{i\in I_j}g_i$ and $H_j=\sum_{i\in I_j}h_i$. In the L2 case, the optimal local weight for leaf $j$ is:

$$ w_j^*=-\frac{G_j}{H_j+\lambda}. $$

The gain of a candidate split comes from the difference between the optimal objective value after splitting into two leaves and the value before splitting, minus the complexity cost of the new leaves. Second-order information is not abstractly "more accurate", it provides an estimate of the local curvature and enables Newton-style updates.

4. Capacity Parameters Are Not Independent Controls

Learning rate and rounds

Smaller $\eta$ typically requires more boosting rounds. While it may produce smoother paths, it does not automatically prevent overfitting; the number of rounds must still be selected based on validation performance.

Tree structure

max_depth, max_leaves/grow policy control interaction complexity; min_child_weight constrains Hessian weights within leaves, which is not equivalent to fixing sample size; gamma/minimum split loss requires sufficient splitting gain.

Leaf weight regularization

L2 reg_lambda and L1 reg_alpha shrink leaf values. The Hessian scale varies across objectives, so identical parameter values cannot be meaningfully compared without considering the objective function.

Row and column sampling

subsample and colsample_bytree introduce randomness, reducing correlation and cost, but setting them too low risks losing signal and increasing random noise.

These parameters interact with one another. There is no universal sequence (such as "first depth, then regularization, finally sampling") that applies across all cases. Instead, validation protocol and search budget should be fixed first, followed by joint evaluation of capacity, step size, and cost.

5. Histograms, Sparsity, and Categorical Support

Scanning all continuous thresholds precisely is computationally expensive. The histogram method maps values into bins and then evaluates splits based on cumulative gradient or Hessian statistics. The number of bins affects performance, memory usage, and approximation accuracy.

Implementations like XGBoost can learn default branch directions for missing values, but this does not mean the missingness mechanism has been correctly modeled. The meaning of NaN values during training must align with their interpretation at deployment, and missing rate drift should be monitored.

Native categorical support, categorical grouping, and one-hot encoding strategies vary across libraries and versions. Never treat arbitrary integer encodings as continuous values; instead, consult the current version documentation, pin dependencies, and test deployments with unseen categories.

6. Early stopping Must Use Validation

Data responsibilities should be separated:

• train: fit tree weights and leaf values per iteration;
• validation: determine the number of iterations, hyperparameters, and thresholds;
• test: evaluate once after all selections have been finalized.

Placing the test set inside eval_set to perform early stopping would cause the test data to participate in model selection. The final performance metric would then no longer represent an estimate on unseen data.

The current XGBoost sklearn interface training skeleton is shown below; in real projects, this structure should be fixed and dependency versions verified:

from xgboost import XGBClassifier
from sklearn.metrics import log_loss

model = XGBClassifier(
    objective="binary:logistic",
    eval_metric="logloss",
    tree_method="hist",
    n_estimators=2_000,
    learning_rate=0.05,
    max_depth=4,
    min_child_weight=5,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_lambda=1.0,
    early_stopping_rounds=50,
    random_state=42,
)

model.fit(
    X_train,
    y_train,
    eval_set=[(X_valid, y_valid)],
    verbose=False,
)

# sklearn interface Will be used during prediction early stopping The best round selected.
test_probability = model.predict_proba(X_test)[:, 1]
print("test log loss:", log_loss(y_test, test_probability))

Do not adjust max_depth or the learning rate based on test loss. Once you do, the original test set effectively becomes part of the validation set, requiring either new independent test data or a nested evaluation approach.

7. Pitfalls of Native API's Best Iteration Traps

When using xgboost.train, early stopping defaults to returning the last iteration of the model rather than truncating the booster at the best iteration. To explicitly specify the best iteration, you must configure it like this:

best_probability = booster.predict(
    dtest,
    iteration_range=(0, booster.best_iteration + 1),
)

Alternatively, use an early-stopping callback that supports save_best=True. Be aware that when multiple evals/metrics are specified, the decision of which dataset and metric triggers stopping is governed by the API's rules, this must be verified against logs and the current official documentation.

This behavior may change with API version updates. Therefore, code reviews should not simply confirm that "early_stopping_rounds" is set; they must also validate which iterations are actually used in the final prediction.

8. Differences Between Bagging and Boosting

Dimension	Random Forest / Bagging	Gradient Boosting
Training Dependency	Trees can train independently and in parallel	Round $t$ depends on results from rounds $1$ to $t-1$
Combination Strategy	Average probabilities or predictions	Additive accumulation of raw scores or predictions
Source of Diversity	Bootstrap sampling, random feature selection	Current gradient, sampling, and tree constraints
Role of Number of Rounds	Average gradually stabilizes; performance plateaus	Continuously modifies model capacity; requires selection of optimal round count
Common Risks	Deep tree size, correlated errors, probability miscalibration	Sensitive to noise and drift, coupled hyperparameter tuning, serial training

"The idea that bagging reduces variance and boosting reduces bias" serves as a useful introductory heuristic, but it is not a universal rule. Bagging can also affect bias, and boosting may either reduce or increase variance, outcomes depend on the base learner, loss function, regularization, and the data generation process.

9. Evaluation Cannot Stop at AUC

Boosted trees often exhibit strong ranking capabilities, but their sigmoid or softprob outputs do not guarantee calibration. Class weights, undersampling, misaligned target definitions, and shifts in data distribution can all alter the semantic meaning of predicted probabilities.

At a minimum, evaluate:

• log loss, Brier score, AUC, and precision-recall metrics on validation or test sets against business cost implications;
• reliability curves and calibration across different time periods or user groups;
• whether the best iteration remains consistent across folds;
• feature missing rates, raw scores, and shifts in predicted probability distributions;
• inference latency, model size, and peak memory usage.

If applying post-hoc calibration, the calibration data must be independent of the base learner’s training, or cross-validation must be used to ensure robustness.

10. Explanation and Extrapolation Boundaries

Importance metrics like gain, weight, and cover are defined differently and are influenced by correlated features and training paths. Tools such as SHAP provide a decomposition of a model's contribution relative to a background distribution, but they do not automatically offer causal interpretations, conclusions can vary depending on the choice of background data.

Tree ensembles typically maintain existing leaf values outside their training range and lack the natural extrapolation behavior of properly specified linear or mechanistic models. When dealing with long-term trends, price curves, or physical constraints, explicit out-of-range testing must be performed. Monotonic constraints, mechanism-based features, or alternative model types should be considered when necessary.

Common Misconceptions

• Boosting is simply fitting mislabeled samples repeatedly: Under typical objectives, the model fits the negative gradient or Newton direction, not randomly selected misclassified data.
• Second-order approximation is inherently more accurate: While it leverages local curvature, the final result still depends on the target function and the assumptions made in the approximation.
• A depth of 3–6 is a fixed rule: Tree growth strategies, data characteristics, and interaction terms vary significantly across use cases.
• Using early stopping eliminates data leakage: The validation set used to control stopping must be properly separated from the test set; otherwise, leakage can still occur.
• More rounds will always lead to better performance: The number of rounds is part of the model’s capacity, and increasing it does not guarantee improvement.

Exercise

1. Derive the gradient and Hessian of squared loss and binary log loss.
2. Plot training and validation loss for a fixed split, and mark the best iteration.
3. Compare whether the sklearn interface and the native API agree on the optimal number of rounds for prediction.
4. Jointly vary the learning rate and number of rounds, and compare performance, training time, and model size.
5. Generate inputs outside the training range and compare the extrapolation behavior of linear models, random forests, and boosting methods.

Summary

Gradient Boosting constructs an additive model sequentially, where each new tree corrects the prediction by moving in the direction of the current loss reduction. XGBoost transforms this process into a scalable tree optimization system by incorporating gradients, Hessians, structural regularization, and sampling. A truly reliable training pipeline still requires independent validation, accurate identification of the optimal round, proper probability calibration, and ongoing deployment monitoring.

The next chapter moves beyond labeled supervised learning: it first explores what unsupervised methods actually optimize, before examining the validation boundaries for clustering, dimensionality reduction, and anomaly detection.