10.1 Convolution, Receptive Fields, and Residual Connections: Embedding Spatial Structure into Parameter Sharing

Increasing an image’s width and height increases the inputs to a flattened fully connected layer. Its weights also distinguish an edge in the top-left from an identical edge in the bottom-right. Convolution shares a local detector across positions, avoiding that repeated parameterization.

Convolution embeds two assumptions into its architecture: local neighborhoods are more relevant, and the same local pattern can appear in different locations. It doesn’t make the model "understand images" in a human sense; instead, it improves sample-to-parameter efficiency through local connectivity and weight sharing.

Learning Objectives

• Write out the input/weight/output shape and parameter count for Conv2d;
• Calculate how kernel size, stride, padding, and dilation affect output dimensions;
• Distinguish between translation equivariance and invariance;
• Recursively compute the theoretical receptive field;
• Understand the behavior of pooling, group/depthwise convolution, and residual connections.

1. PyTorch Conv2d Actual Computation: Cross-Correlation

The input typically has the following shape:

X: [batch, in_channels, height, width]
W: [out_channels, in_channels / groups, kernel_h, kernel_w]
Y: [batch, out_channels, out_height, out_width]

Each output channel consists of a set of local filters that operate across input channels, with the same filter weights shared across all spatial positions. Deep learning libraries generally do not flip the kernel, so the operation implemented is cross-correlation; during training, the weights are learned directly to achieve the desired behavior. In practice, this operation is still commonly referred to as convolution.

The number of parameters when using groups=1 is:

$$C_{out}{C}_{in}{K}_h{K}_w+{\mathbf{1}}_{bias}{C}_{out}.$$

This parameter count is independent of the input image dimensions $H,W$. In contrast, the number of parameters in a fully connected layer depends directly on the spatial size after flattening.

2. Output Dimensions Cannot Be Estimated by Eye

For a single spatial dimension:

$$L_{out}=\lfloor \frac{L_{in}+2p-d(k-1)-1}{s}+1\rfloor ,$$

where $k$ is the kernel size, $s$ is the stride, $p$ is the single-side padding, and $d$ is the dilation. The effective kernel size is:

$$k_{eff}=d(k-1)+1.$$

For example, with $L_{in}=28$, $k=3$, $p=1$, $s=2$, and $d=1$:

$$L_{out}=\lfloor \frac{28+2-2-1}{2}+1\rfloor =14.$$

Even kernels, asymmetric padding, and the framework's same rule can affect boundary behavior. Always validate output shapes using shape testing before deployment.

3. Channels Are Not Fixed "Edge Semantics"

The first-layer filter might capture directional or color patterns, while deeper channels represent distributed encodings under the training objective. You cannot preset that channel 17 must be a "cat-ear detector." The same functionality may be distributed across multiple channels, and different random seeds can lead to rotated or permuted representations.

Convolutional parameter sharing introduces approximate translation equivariance: when the input is shifted, the feature map shifts accordingly. However, padding, stride, pooling, boundary cropping, and discrete sampling all introduce deviations that break strict equivariance.

Classification tasks aim for some degree of translation invariance, typically achieved through pooling, downsampling, global aggregation, and data augmentation; convolutional operations themselves are closer to equivariance than invariance.

4. Stride, Padding, Dilation Change What?

• Padding: controls the boundaries and output dimensions, and introduces assumptions about boundary values (such as zero, reflection, or replication);
• Stride: performs skip sampling and downsampling, potentially introducing aliasing;
• Dilation: spreads out the kernel taps, expanding the receptive field without increasing model parameters;
• Kernel size: alters the local receptive field and affects the number of parameters and FLOPs.

Applying a low-pass filter before downsampling can reduce aliasing, but the specific anti-aliasing design must be tailored to the task at hand. padding=1 is not an unconditional default for every $3\times 3$ convolution; stride, dilation, and boundary semantics matter.

5. Receptive Field Must Track Jump

Let the spacing between adjacent feature positions at layer $l$ relative to the original input be $j_l$, and let the receptive field at that layer be $r_l$:

$$j_l=j_{l-1}\cdot s_l,$$$$r_l=r_{l-1}+(k_{eff,l}-1)\cdot j_{l-1},$$

with initial values $j_0=r_0=1$. Two consecutive stride-1 $3\times 3$ convolutions have a theoretical receptive field of $5\times 5$; this grows more rapidly when strides or pooling operations are introduced.

Theoretical receptive field only indicates which input elements a layer might depend on. The actual/effective receptive field is influenced by weights, nonlinearities, normalization, and training, leading to uneven contributions from the center positions.

For conv3/s1 → pool2/s2 → conv3/s1 → pool2/s2, the (r, j) pairs are (3,1) → (4,2) → (8,2) → (10,4). This recurrence describes local convolution/pooling paths; operations that aggregate across space, including training-mode spatial BatchNorm, introduce wider dependencies.

6. Pooling and Learnable Downsampling

• Max pooling retains the maximum value within a window, offering local invariance but at the cost of losing fine details;
• Average pooling computes the local average across a window;
• Strided convolution learns a downsampling filter through training, enabling adaptive spatial reduction;
• Adaptive or global average pooling aggregates features from any input spatial size into a fixed output dimension.

Tasks like segmentation and detection rely on dense prediction and must avoid premature loss of spatial information. Common strategies include skip connections, multi-scale feature pyramids, or encoder–decoder architectures that restore spatial resolution.

7. Grouped and Depthwise Convolution

groups groups channel connections. If groups=C_in and C_out=K C_in, each input channel uses its own spatial filters, resulting in depthwise convolution.

A common configuration of the depthwise separable block includes:

1. depthwise spatial convolution;
2. a $1\times 1$ pointwise convolution to mix channels.

This combination significantly reduces the number of parameters and FLOPs. However, actual latency depends on kernel implementation, memory bandwidth, tensor shape, and hardware architecture, fewer FLOPs do not guarantee faster performance in real-time scenarios.

8. Residual Connection Optimizes the Increment

Residual block:

$$y=x+F(x;\theta ).$$

When the dimensions are the same, the identity shortcut provides a direct path for both forward propagation and gradient flow, enabling the network to learn a residual relative to the identity function. This alleviates optimization degradation in deep networks, but it does not guarantee stability at arbitrary depths, nor does it mean that gradients are entirely unaffected by nonlinearity.

When the shapes differ, projection or downsampling via the shortcut is required. Before addition, spatial, channel, and data type dimensions must align; arbitrary broadcasting introduces bugs.

Pre-activation or post-activation, normalization, and residual scaling all influence training dynamics and should be implemented according to the target architecture.

9. A CNN Without Hardcoded Spatial Dimensions

This runnable Python 3 / PyTorch 2.14 CPU example checks two image sizes. Both spatial dimensions must be at least 4 for its two pooling operations; passing a shape check does not establish useful predictions.

import torch
from torch import nn

class SmallCNN(nn.Module):
    def __init__(self, in_channels=1, classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(in_channels, 32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
        )
        self.pool = nn.AdaptiveAvgPool2d((1, 1))
        self.classifier = nn.Linear(64, classes)

    def forward(self, x):
        x = self.features(x)
        x = self.pool(x)
        x = torch.flatten(x, start_dim=1)
        return self.classifier(x)

model = SmallCNN()
assert model(torch.randn(8, 1, 28, 28)).shape == (8, 10)
assert model(torch.randn(8, 1, 32, 40)).shape == (8, 10)

Adaptive pooling allows the classifier to avoid dependence on a fixed $7\times 7$ dimension, but semantic consistency, preprocessing pipelines, and training distributions across different resolutions still require validation.

10. Data Augmentation Writing Under the Invariance Assumption

Techniques like random crop, flip, color jitter, and mixup are not free data augmentation, they explicitly state that certain transformations should not alter labels or should follow defined rules for label mixing.

• Horizontal flipping is reasonable for some object classification tasks, but may introduce errors for text or medical imaging where orientation matters;
• Color jitter can compromise tasks that rely on color as a key discriminative feature;
• Resizing or cropping may remove subtle defects or fine details;
• Augmentation should be applied only during training; validation and test sets must use fixed, deterministic preprocessing.

Stress-test models under deployment-level corruption scenarios to avoid relying solely on self-verified robustness under a single augmentation strategy.

11. Deployment Boundaries of CNNs

Document the input channel order, color space, resize interpolation method, normalization mean and standard deviation, orientation, and dynamic shape. A well-structured model will fail if trained on RGB and deployed with BGR, or if integer division errors occur, these issues can render even the best architecture ineffective in production.

Evaluation should extend beyond accuracy to include:

• resolution, blur, compression, and lighting conditions;
• spatial subgrouping and device-specific behavior;
• calibration and abstention behavior;
• latency, peak GPU memory usage, and performance at batch size = 1;
• robustness to adversarial and natural distribution shifts.

Common Misconceptions

• Libraries perform strict mathematical convolution: Most deep learning libraries implement cross-correlation, not true convolution.
• CNNs are inherently translation-invariant: Convolutional networks are actually more equivariant; boundaries and downsampling can break this property.
• Two 3×3 layers equal one 5×5 layer: With stride and dilation 1 they share a receptive-field size, but nonlinearities and parameter factorization generally make them different functions.
• Residual connections completely resolve gradient vanishing: They improve gradient flow, but initialization, normalization, and optimization still play critical roles.
• Fewer FLOPs always mean faster performance: Hardware architecture and memory access patterns are often far more decisive than computational count.

Exercises

1. Compute output dimensions and parameter counts for different stride, padding, and dilation settings.
2. Use jump–RF recursion to calculate the receptive field of a CNN containing pooling layers.
3. Shift the input and measure the equivariance error before and after applying padding and stride.
4. Compare standard convolutions with depthwise-separable convolutions in terms of parameters, FLOPs, and actual latency.
5. For a medical or text-image task, evaluate which augmentations would alter the labels.

Summary

Convolution shares local weights across positions. Kernel, stride, dilation, and pooling set the spatial dependencies and sampling grid; residual shortcuts change the optimization path. Shape and receptive-field calculations are useful checks, while invariance, speed, and useful predictions still require measurement.

The next lesson shifts the sharing from spatial positions to time steps: RNNs use the same recurrent unit to update state, causing gradients to propagate across time by multiplying a sequence of state-dependent Jacobians built from shared weights.