10.1 Vectors and Inner Products: Turning Objects into Comparable Coordinates

Learning Objectives

After completing this lesson, you should be able to:

• Distinguish between the algebraic, geometric, and data-level meanings of vectors;
• Use norms, distance metrics, dot products, and cosine similarity;
• Determine whether two vectors can be directly compared;
• Explain why feature scaling affects the outcomes of distance-based models.

1. Coordinates Are Not the Object Itself

This lesson uses finite-dimensional real vectors, represented by ordered real coordinates:

$$\mathbf{x}=(x_1,x_2,\dots ,x_n{)}^{\mathsf{T}}.$$

This same structure can serve three distinct roles:

• A point or displacement in geometry;
• A system state, such as [CPU usage, memory usage, request latency];
• A feature representation, for example, a movie’s weighted scores across dimensions like "action," "comedy," and "mystery."

To interpret a vector, you need to know what its coordinates mean. [0.8, 0.1, 0.6] by itself conveys nothing; only when we know the order, units, and encoding scheme of its components does comparison become valid. If you put age, income, and click counts into one distance formula without scaling, the largest numerical range will often dominate. The formula will do the arithmetic; it cannot tell that dollars and years need different treatment.

2. Basic Operations in Vector Spaces

For vectors of the same dimension, component-wise addition is defined as:

$$\mathbf{x}+\mathbf{y}=(x_1+y_1,\dots ,x_n+y_n{)}^{\mathsf{T}}.$$

Scalar multiplication multiplies each component by the same scalar value:

$$c\mathbf{x}=(c{x}_1,\dots ,c{x}_n{)}^{\mathsf{T}}.$$

These two operations allow us to form linear combinations:

$$a_1{\mathbf{v}}_1+\cdots +a_k{\mathbf{v}}_k.$$

If a vector can be expressed as a linear combination of a set of vectors, we say it lies in the space spanned by that set. Later, when solving linear equations, the problem becomes: does the target vector lie within the space spanned by the column vectors of the matrix?

3. Norms and Distance

The 2-norm describes the Euclidean length of a vector:

$$\Vert \mathbf{x}{\Vert }_2=\sqrt{x_1^2+\cdots +x_n^2}.$$

The Euclidean distance between two points is the length of their difference:

$$d(\mathbf{x},\mathbf{y})=\Vert \mathbf{x}-\mathbf{y}{\Vert }_2.$$

Different norms emphasize different aspects:

Norm	Definition	Intuitive Meaning
$L_1$	$\sum _i|x_i|$	Sum of absolute deviations across dimensions
$L_2$	$\sqrt{\sum _i{x}_i^2}$	Straight-line distance
$L_{\mathrm{\infty }}$	$\underset{i}{max}|x_i|$	Maximum deviation in any single dimension

Choosing a norm is equivalent to choosing what counts as "close." In a delivery grid, $L_1$ distance might be more relevant; in error budgets, the maximum component deviation ($L_{\mathrm{\infty }}$) may matter; in general geometric spaces, $L_2$ is most commonly used.

4. Dot Product and Projection

The dot product maps two vectors of the same dimension to a scalar:

$${\mathbf{x}}^{\mathsf{T}}\mathbf{y}=\sum _{i=1}^n{x}_i{y}_i.$$

It also has a geometric form:

$${\mathbf{x}}^{\mathsf{T}}\mathbf{y}=\Vert \mathbf{x}{\Vert }_2\Vert \mathbf{y}{\Vert }_2\cos \theta .$$

Thus, the dot product is influenced by both direction and magnitude. If you're only interested in direction, use cosine similarity:

$$\cos \mathrm{\_}\mathrm{sim}(\mathbf{x},\mathbf{y})=\frac{{\mathbf{x}}^{\mathsf{T}}\mathbf{y}}{\Vert \mathbf{x}{\Vert }_2\Vert \mathbf{y}{\Vert }_2}.$$

The zero vector has no direction. It makes the denominator zero, so cosine similarity is undefined. Engineering code must explicitly define behavior: reject zero vectors, return a special value, or implement fallback logic at the business layer.

The projection of vector $\mathbf{x}$ onto the direction of a non-zero vector $\mathbf{u}$ is:

$${\mathrm{proj}}_{\mathbf{u}}\mathbf{x}=\frac{{\mathbf{x}}^{\mathsf{T}}\mathbf{u}}{{\mathbf{u}}^{\mathsf{T}}\mathbf{u}}\mathbf{u}.$$

This projection reappears in least squares: when an exact match to a target is impossible, the closest point within the column space is sought.

5. A Comparison in Recommendation Systems

Suppose the three dimensions represent action, comedy, and suspense. This complete example uses Python 3 and NumPy 2. In a virtual environment, install the dependency with python -m pip install "numpy>=2,<3", save the code as cosine_demo.py, and run python cosine_demo.py. Inputs are one-dimensional real arrays of equal size, with finite values of moderate magnitude:

import numpy as np

user = np.array([0.9, 0.2, 0.7], dtype=float)
movie_a = np.array([0.8, 0.1, 0.9], dtype=float)
movie_b = np.array([0.1, 0.9, 0.2], dtype=float)

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    denominator = np.linalg.norm(a) * np.linalg.norm(b)
    if denominator == 0:
        raise ValueError("Cosine similarity does not accept zero vectors")
    return float(a @ b / denominator)

print(f"{cosine_similarity(user, movie_a):.6f}")
print(f"{cosine_similarity(user, movie_b):.6f}")

The two outputs are approximately 0.979471 and 0.381929, so movie A is closer to the user vector in direction for these three features. That explains this ranking. A real recommender still needs evidence about how features are obtained, how cold starts are handled, and whether similar directions correspond to actual preferences.

6. Why Scaling Changes the Answer

Consider two features: age ranging from 18–70, and annual spending ranging from 0–100000. When computing Euclidean distance directly, spending dominates the result almost entirely. Common approaches include:

• Standardization: subtract the mean and divide by the standard deviation;
• Range scaling: map values into a fixed interval;
• Manually assigning weights based on business context;
• Using a distance metric suited to the scales, or a learned representation.

Scaling changes the geometry used by the model. Estimate means, standard deviations, and other parameters on the training set, then reuse them for validation and online data. Fitting on the full dataset leaks validation information into training. Fitting a separate scaler on validation data changes the coordinate system and evaluates a different processing pipeline. In cross-validation, fit the scaler only on the training portion of each fold.

7. Common Misconceptions

• Larger dot product means greater similarity: If the vectors differ significantly in magnitude, the dot product may simply reflect scale differences rather than semantic similarity.
• Cosine similarity equals distance: It measures direction, not magnitude; whether it's appropriate depends on the specific business context.
• Equal dimensions are enough for comparison: Semantic meaning, order, units, and encoding versions must also align for valid comparison.
• Cross product is a general vector similarity measure: Commonly used in 3D space, the cross product produces a vector output and solves a different class of geometric problems.

Exercise

1. Compute the dot product, L2 norm, and cosine similarity between x=(2,3) and y=(1,4).
2. Construct two 2D points with equal L1 distances from the origin but different L2 distances from it.
3. Add a "view count" dimension to the recommendation vector, and explain the potential bias that could arise if this dimension is not scaled.
4. Modify the code to implement a business-semantic handling strategy for the zero vector, and provide a justification for your approach.

Summary

Before comparing vectors, check the meaning, order, and units of their coordinates. Then choose the information to preserve: norms measure length, distance measures difference, and cosine similarity compares the directions of nonzero vectors. Representation and scaling directly affect the result.

The next lesson uses matrices to transform coordinates and connects linear equations to column spaces.