7.2 PCA, SVD, and Visualization: Explaining Variance Does Not Equal Preserving Task Information

Dimensionality reduction can compress and visualize data, but each method preserves different structural properties. PCA retains variance along linear projections; t-SNE primarily preserves local similarity probabilities. A two-dimensional plot is not a lossless representation of the original space.

Learning Objectives

• Derive PCA scores and components from the SVD of a centered matrix;
• Explain the equivalence between maximizing variance and minimizing reconstruction error;
• Distinguish between centering, standardization, and whitening;
• Handle scenarios involving training/test splits, sparse inputs, and large-scale data;
• Correctly interpret PCA or t-SNE plots, without treating visualizations as evidence of clustering.

1. PCA Begins with Centering

Let $X\in {\mathbb{R}}^{n\times p}$, and let $X_c$ be the matrix obtained by subtracting the training mean from each column of $X$. Its singular value decomposition is:

$$X_c=U\mathrm{\Sigma }{V}^T.$$

• The columns of $V$ are the principal directions (or loadings);
• The first $q$ principal scores are $Z=X_c{V}_q=U_q{\mathrm{\Sigma }}_q$;
• The sample variance of the $j$th component corresponds to ${\sigma }_j^2/(n-1)$, under the standard convention of unbiased covariance.

In orthogonal linear projection, the first $q$ directions simultaneously:

1. Maximize the total variance of the projected data;
2. Minimize the squared reconstruction error of rank-$q$ approximation.

These two properties arise from the geometry of Euclidean squared-error minimization and do not imply that all information relevant to downstream tasks is preserved.

2. What the Explained Variance Ratio Answers

$${\text{EVR}}_j=\frac{{\sigma }_j^2}{\sum _r{\sigma }_r^2}.$$

The ratio requires nonzero total variance; with $n<2$, sample variance is undefined as well.

The sum of the first $q$ EVR values indicates how much total variance is retained in the selected linear subspace within the training snapshot. However, it does not tell you:

• Whether low-variance directions contain rare fault signals;
• Whether classes are separable;
• Whether causal variables are preserved;
• Whether the variance structure remains consistent across new time windows;
• Whether reconstruction errors are acceptable across business domains.

"Retaining 95% of variance" is merely one candidate rule for compression, it is not a universal quality benchmark.

This runnable Python 3 / NumPy example keeps only the highest-variance direction, without standardizing the columns:

import numpy as np

X = np.array([[-10., -1.], [-10., 1.], [10., -1.], [10., 1.]])
y = (X[:, 1] > 0).astype(int)
Xc = X - X.mean(axis=0)
_, singular, Vt = np.linalg.svd(Xc, full_matrices=False)
Z = Xc @ Vt[:1].T
print("retained variance:", round(singular[0] ** 2 / np.sum(singular ** 2), 4))
print("projected points:", Z.ravel())
print("labels:", y)

It retains about 0.9901 of total variance, yet each projected location has both labels. The discarded second coordinate predicts the labels perfectly. The sign of the projection may reverse across SVD implementations without changing this result.

3. Centering Does Not Mean Standardization

PCA must explicitly define a data baseline. Common PCA implementations center each column, but do not automatically scale the columns to unit variance.

• Covariance PCA: retains the original units and variance weights;
• Correlation PCA: standardizes first, effectively making the initial variance of each column equal;
• Robust scaling: uses robust center and scale estimates, but ordinary PCA still uses squared error and remains sensitive to outliers;
• Whitening: further scales the retained components to unit variance after projection, discarding relative scale information.

If the business meaning of "load 1 kilogram" differs from that of "delay 1 millisecond," standardization does not automatically yield correct results. The scaler parameters and units must be documented, and downstream objectives must be validated to ensure correctness.

4. A Leak-Resistant PCA Pipeline

This Python 3 / scikit-learn 1.9.0 fragment requires finite numeric X_train and X_valid matrices with matching columns, at least two training rows, and nonzero total variance:

from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

reducer = Pipeline([
    ("scale", StandardScaler()),
    ("pca", PCA(n_components=0.95, svd_solver="full")),
])

Z_train = reducer.fit_transform(X_train)
Z_valid = reducer.transform(X_valid)

pca = reducer.named_steps["pca"]
print("components:", pca.n_components_)
print("training EVR:", pca.explained_variance_ratio_.sum())

The training mean, scale, and components must be fitted only on the train split. Performing PCA on the entire dataset before cross-validation introduces the validation or test data's covariance structure into the feature representation, leading to data leakage.

If using PCA for dimensionality reduction in a supervised model, the entire pipeline (including the number of components) must be placed inside each CV fold. The component count should be selected via inner validation.

5. Reconstruction Requires Returning to Original Units for Verification

After retaining $q$ components:

$${\hat{X}}_c=Z{V}_q^T.$$

If prior standardization was applied, an inverse transformation should be performed to evaluate the reconstructed data in original units. The global mean MSE may mask critical issues such as:

• Excessive errors from a key sensor;
• Smoothing out of small groups or rare states;
• Failure to reconstruct extreme values or boundary conditions;
• Missing data imputation patterns being captured by the components.

Beyond evaluating EVR, compression applications should also measure storage footprint, encoding/decoding latency, and downstream performance.

6. Signs and Stability of Components

If $v$ is a principal direction, then $-v$ lies along the same direction. Therefore, a reversal of signs in PC1 across two runs does not indicate a change in the model.

When adjacent singular values are very close, a single component may rotate within the subspace, leading to unstable loadings rankings; the overall subspace, however, may remain stable. When comparing results, use principal angles, subspace overlap, or reconstruction quality rather than enforcing strict sign alignment column by column.

Extreme values can strongly influence means and covariances. It is essential to first determine whether outliers represent errors, rare true states, or actual signal of interest.

7. Sparse and Big Data Scenarios

Explicitly centering a sparse term-frequency matrix usually makes it dense. TruncatedSVD avoids centering and is useful for sparse inputs, but its decomposition has a different meaning from centered PCA. Some PCA solvers can handle centering implicitly; check the selected solver rather than assuming every sparse input must be densified.

For big data, consider:

• randomized SVD: to approximate the first few components;
• IncrementalPCA: updates in mini-batch fashion, reducing memory footprint per batch;
• distributed or streaming matrix factorization;
• applying feature hashing or selection first, followed by evaluation of approximation error.

Solver selection depends on $n,p,q$, sparsity, memory availability, and required precision. Fix dependency versions and validate the consistency of subspaces and reconstruction errors across the same dataset.

8. PCA Is Not Feature Selection

Each component typically combines multiple original features. It can reduce collinearity and compress noise, but at the cost of field-level interpretability and may complicate data governance.

If deployment must read only a few original columns, use feature selection or explicitly constrained feature groups. Sparse PCA and autoencoders learn representations and do not by themselves guarantee a smaller set of input fields.

9. t-SNE Preserves Local Similarity Probabilities

t-SNE converts high-dimensional neighborhood similarities into probability distributions and optimizes the Kullback-Leibler (KL) divergence between these and the corresponding low-dimensional neighborhood probabilities. The resulting objective function is non-convex, meaning that outcomes are highly sensitive to initialization, perplexity, learning rate, and random seed, each of which can significantly influence the final visualization.

When interpreting t-SNE plots, caution is essential:

• Local neighborhood structures are generally more reliable than global distances;
• A large gap between two "islands" does not imply the same distance in the original high-dimensional space;
• The size or density of an island does not reflect the true scale or variance of the original cluster;
• Color separation may be influenced by parameters, labeling choices, or preprocessing steps;
• It is common for multiple runs to produce different layouts.

scikit-learn's TSNE primarily provides fit_transform, rather than learning a standard linear transform like PCA does for arbitrary new samples. For stable deployment of embeddings, methods that support out-of-sample mapping should be selected, and their performance must be validated independently.

High-dimensional inputs are typically first reduced to a more manageable dimension using PCA (for dense data) or TruncatedSVD (for sparse data), before applying t-SNE. This helps reduce noise and computational cost. The target dimension is not a fixed truth and should be chosen based on the specific use case.

10. How to Validate Dimensionality Reduction

Choose evidence based on the goal:

• Compression: held-out reconstruction accuracy, per-feature error, storage footprint, and latency reduction;
• Denoising: performance and stability of downstream models on a strict split;
• Visualization: multiple parameter settings or random seeds, neighborhood fidelity, and representative sampling;
• Retrieval: recall of held-out neighbors and business-relevant relevance;
• Preprocessing for clustering: clustering stability and downstream action outcomes, never just the visual appeal of the resulting graph.

If labels are used solely for final evaluation, assess whether dimensionality reduction preserves task-relevant information. If labels are used to select a representation, then the process has become supervised model selection and must be acknowledged within a nested validation framework.

Common Misconceptions

• PCA must perform SVD on the covariance matrix: Performing SVD directly on centered $X$ is typically more numerically stable.
• The first two principal components are the most important original features: Components are linear combinations, not the original variables themselves.
• 95% explained variance ratio (EVR) means 95% of business information is retained: EVR only measures variance in the training data, not business or domain relevance.
• A 2D island implies a natural cluster: t-SNE layouts are highly sensitive to both target structure and hyperparameters.
• PCA has no labels, so it can be fitted on the entire dataset: Leakage into the evaluation pipeline still occurs, violating proper validation practices.

Exercise

1. Manually compute scores, loadings, explained variance ratios, and rank-$q$ reconstruction using SVD.
2. Compare covariance PCA with correlation PCA performed after standardization.
3. Introduce an extreme outlier and observe how the principal components and reconstruction change.
4. Run multiple t-SNE instances with different seeds or perplexity values on the same dataset, and compare the positioning of local neighborhoods versus isolated clusters.

Summary

PCA finds the highest-variance orthogonal linear subspace in centered data and provides minimum-squared reconstruction. This objective is well-suited for many compression and denoising tasks, but it does not guarantee preservation of label information, causal relationships, or rare event details. t-SNE is better suited for exploring local neighborhoods; its two-dimensional layout should not be taken as definitive clustering conclusions.

The next lesson addresses "a small number of different samples": first distinguishing outliers in training data from novel instances encountered at deployment, then determining how model scores can be transformed into actionable alerts.