How Convolution Works in CNNs

The Convolution Operation: From Kernel to Feature Map

A convolution slides a small grid of weights — called a kernel or filter — across an input image. At each position, the kernel computes a dot product (element-wise multiply and sum) with the input patch beneath it. The result is stored in a new spatial location, building up a feature map. Unlike a fully connected layer that treats every input pixel as an independent variable, convolution exploits the spatial structure of images by reusing the same kernel across all positions.

Mathematically, a 2D convolution is defined as:

y[i, j] = Σ Σ x[i+m, j+n] * w[m, n] + b

where x is the input, w is the kernel, and the sum iterates over the kernel dimensions. This operation is applied at different positions (i, j) across the input, creating the output feature map.

Dimensions and parameters

An input image has shape (height, width, channels). For example, a 28×28 grayscale image is (28, 28, 1). The kernel has shape (kernel_height, kernel_width, input_channels, output_channels). If you want to detect 32 different patterns in that image, you use 32 kernels of shape (3, 3, 1, 32) — each kernel is a 3×3 grid applied to all channels (though in this case only 1 channel exists).

The size of the output depends on three parameters:

• Kernel size (K): often 3×3 or 5×5. Larger kernels capture larger spatial patterns but require more parameters.
• Stride (S): how many pixels the kernel moves at each step. Stride=1 moves one pixel; stride=2 moves two pixels, reducing spatial dimensions more aggressively and cutting computation.
• Padding (P): zeros added around the input's border. Padding=1 wraps the input in one layer of zeros, letting the kernel reach the corners and preserving spatial dimensions. Padding is essential for maintaining information at image edges.

The output spatial dimensions are computed as:

output_height = floor((input_height - kernel_height + 2*padding) / stride) + 1
output_width = floor((input_width - kernel_width + 2*padding) / stride) + 1

For example, a 28×28 input, 3×3 kernel, stride=1, padding=1:

output_height = floor((28 - 3 + 2) / 1) + 1 = 28
output_width = floor((28 - 3 + 2) / 1) + 1 = 28

You keep the spatial size the same. If instead you use stride=2, the output becomes floor((28 - 3 + 2) / 2) + 1 = 14, shrinking the feature map by half. This stride-2 convolution is often used in modern architectures (like ResNet) instead of max pooling, as it is more learnable and efficient.

Why convolution is different from dense layers

To appreciate convolution's efficiency, compare parameter counts. A 28×28 grayscale image flattened has 784 pixels. A fully connected layer with 256 hidden units uses 784 × 256 = 200,704 weights. A convolutional layer with a 3×3 kernel and 32 filters uses only 3 × 3 × 1 × 32 = 288 parameters. This is a ~700x reduction while still capturing spatial patterns effectively.

Weight Sharing: Why Convolution Is Parameter-Efficient

A 28×28 grayscale image has 784 pixels. If you use a fully connected layer with 256 hidden units, you have 784 × 256 = 200,704 weights (plus biases). If you stack a second fully connected layer with 256 units, you add 256 × 256 = 65,536 more weights. Total: ~266,000 parameters for just two layers.

Now use a convolutional approach: a single 3×3 kernel applied to the entire image requires only 3 × 3 × 1 = 9 weights (ignoring the bias). If you use 32 output filters, that is 9 × 32 = 288 weights. Millions of times fewer. This is weight sharing: the same 9-weight kernel is applied everywhere, learning a single pattern (e.g., a vertical edge, a corner, a texture) and reusing it across the whole image.

Why does this work? Images have translation invariance: a vertical edge in the top-left corner looks the same as a vertical edge in the bottom-right. A single learned edge detector, applied everywhere, is far more efficient than learning different edge detectors for each region. Convolution enforces this inductive bias into the architecture.

Real-world impact: LeNet vs. fully connected

The original LeNet-5 (LeCun et al., 1998) achieved ~99.5% accuracy on MNIST using only ~60,000 parameters. A comparable fully connected network (two hidden layers of 256 units each) has ~200,000 parameters. For ImageNet-scale tasks, the savings are dramatic: ResNet-50 (convolution-based) has ~25.5 million parameters, while a fully connected network with comparable capacity would require billions.

| Architecture Type | MNIST Params | Accuracy | Practical Use | |---|---|---|---| | Fully connected (2 layers) | ~200k | ~97% | Baseline; very slow; memory-intensive | | LeNet-5 (convolution) | ~60k | ~99.5% | Efficient; fast; practical for embedded systems | | Modern CNN (ResNet-18) | ~11.7M | ~96% on CIFAR-10 | Strong baseline; widely used | | Very deep CNN (ResNet-152) | ~60M | ~98%+ on CIFAR-10 | High accuracy; requires significant compute |

Example: 2D Convolution from Scratch with NumPy

Implement a basic 2D convolution to see the mechanics clearly. This shows the dot-product computation at each position:

import numpy as np

def convolve2d(image, kernel):
    """
    Perform 2D convolution without padding or stride.

    Args:
        image: 2D numpy array (height, width)
        kernel: 2D numpy array (kernel_height, kernel_width)

    Returns:
        feature_map: (height - kernel_height + 1, width - kernel_width + 1)
    """
    h, w = image.shape
    kh, kw = kernel.shape

    out_h = h - kh + 1
    out_w = w - kw + 1

    feature_map = np.zeros((out_h, out_w))

    for i in range(out_h):
        for j in range(out_w):
            # Extract the patch under the kernel
            patch = image[i:i+kh, j:j+kw]
            # Compute dot product: element-wise multiply and sum
            feature_map[i, j] = np.sum(patch * kernel)

    return feature_map

# Example: 5x5 image with a 3x3 edge detector kernel
image = np.array([
    [1, 2, 3, 4, 5],
    [2, 3, 4, 5, 6],
    [3, 4, 5, 6, 7],
    [4, 5, 6, 7, 8],
    [5, 6, 7, 8, 9]
], dtype=float)

# Vertical edge detector: emphasizes differences left-to-right
kernel = np.array([
    [-1, 0, 1],
    [-1, 0, 1],
    [-1, 0, 1]
], dtype=float)

feature_map = convolve2d(image, kernel)
print("Feature map shape:", feature_map.shape)
print("Feature map:\n", feature_map)
# Output shows higher values where vertical edges (left-right intensity changes) occur

# Numerical walkthrough for position (0, 0):
# patch = [[1, 2, 3], [2, 3, 4], [3, 4, 5]]
# kernel = [[-1, 0, 1], [-1, 0, 1], [-1, 0, 1]]
# dot product = 1*(-1) + 2*0 + 3*1 + 2*(-1) + 3*0 + 4*1 + 3*(-1) + 4*0 + 5*1
#             = -1 + 3 - 2 + 4 - 3 + 5 = 6
# This detects a strong right-ward edge at position (0,0)

This is slow (three nested loops with O(h×w×k²) complexity), but it shows the idea: slide, multiply, sum. Real implementations use efficient matrix operations (im2col) or GPU kernels, reducing time to O(h×w×k²/s²) with stride s.

Now extend it to handle padding and stride, which is critical for controlling output dimensions:

def convolve2d_with_padding_stride(image, kernel, padding=0, stride=1):
    """
    Perform 2D convolution with padding and stride.
    """
    h, w = image.shape
    kh, kw = kernel.shape

    # Add padding
    if padding > 0:
        image = np.pad(image, ((padding, padding), (padding, padding)), mode='constant')
        h, w = image.shape

    out_h = (h - kh) // stride + 1
    out_w = (w - kw) // stride + 1

    feature_map = np.zeros((out_h, out_w))

    for i in range(out_h):
        for j in range(out_w):
            # Stride determines the step
            patch = image[i*stride:i*stride+kh, j*stride:j*stride+kw]
            feature_map[i, j] = np.sum(patch * kernel)

    return feature_map

# Same image, kernel, but with padding=1 and stride=2
feature_map_padded = convolve2d_with_padding_stride(
    image, kernel, padding=1, stride=2
)
print("Padded & strided output shape:", feature_map_padded.shape)

With padding=1, the image becomes 7×7; with stride=2, we sample every other position, giving output shape (7 - 3) // 2 + 1 = 3.

Using PyTorch's nn.Conv2d

In practice, use PyTorch (or TensorFlow) for efficiency:

import torch
import torch.nn as nn

# Define a Conv2d layer: 1 input channel, 32 output filters, 3x3 kernel
conv_layer = nn.Conv2d(
    in_channels=1,
    out_channels=32,
    kernel_size=3,
    stride=1,
    padding=1,
    bias=True
)

# Random 28x28 grayscale image (batch_size=1)
image_tensor = torch.randn(1, 1, 28, 28)

# Apply convolution
output = conv_layer(image_tensor)
print(f"Input shape: {image_tensor.shape}")
print(f"Output shape: {output.shape}")
# Output: torch.Size([1, 32, 28, 28])
# Batch size 1, 32 feature maps, 28x28 spatial dimensions (preserved by padding=1, stride=1)

# Inspect kernel shape
print(f"Kernel shape: {conv_layer.weight.shape}")
# torch.Size([32, 1, 3, 3]) = 32 output filters, 1 input channel, 3x3 spatial size

# Total parameters: 32 * (1 * 3 * 3 + 1 bias) = 32 * 10 = 320
print(f"Total parameters: {sum(p.numel() for p in conv_layer.parameters())}")

PyTorch handles padding, stride, and batching. The kernel weights are learned via backpropagation, just like dense layers.

Pooling: Reducing Spatial Dimensions and Gaining Robustness

After convolution, you often apply pooling — a simple operation that reduces spatial dimensions by taking a summary statistic over small patches.

Max pooling

Max pooling slides a window (e.g., 2×2) and takes the maximum value within each window. It is fast, reduces parameters in subsequent layers, and makes the network robust to small translations (if a feature moves one pixel left, max pooling still "catches" it within the window).

pool_layer = nn.MaxPool2d(kernel_size=2, stride=2)
# 2x2 windows, stride 2 (no overlap)

pooled_output = pool_layer(output)  # Take the previous Conv2d output
print(f"Pooled shape: {pooled_output.shape}")
# torch.Size([1, 32, 14, 14]) from [1, 32, 28, 28]
# Spatial dimensions halved in each direction

Average pooling

Average pooling computes the mean instead of the max. It is smoother but less commonly used for hidden layers (more often used as a final global pooling before classification).

avg_pool = nn.AvgPool2d(kernel_size=2, stride=2)
avg_pooled = avg_pool(output)
print(f"Avg pooled shape: {avg_pooled.shape}")

Building a Small CNN

Combine convolution and pooling:

class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        # First conv block: 1 -> 16 filters, then max pooling
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, padding=1)
        self.pool1 = nn.MaxPool2d(2, 2)

        # Second conv block: 16 -> 32 filters, then max pooling
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1)
        self.pool2 = nn.MaxPool2d(2, 2)

        # Flatten and classify
        self.fc1 = nn.Linear(32 * 7 * 7, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        # x: (batch, 1, 28, 28)
        x = torch.relu(self.conv1(x))  # (batch, 16, 28, 28)
        x = self.pool1(x)              # (batch, 16, 14, 14)
        x = torch.relu(self.conv2(x))  # (batch, 32, 14, 14)
        x = self.pool2(x)              # (batch, 32, 7, 7)
        x = x.view(x.size(0), -1)      # Flatten to (batch, 32*7*7=1568)
        x = torch.relu(self.fc1(x))    # (batch, 128)
        x = self.fc2(x)                # (batch, 10)
        return x

model = SimpleCNN()
test_input = torch.randn(4, 1, 28, 28)
test_output = model(test_input)
print(f"Model output shape: {test_output.shape}")
# torch.Size([4, 10]) = batch of 4 images, 10 class logits

# Count parameters
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total_params}")
# Much smaller than a fully connected network for the same input/output size

This tiny CNN has roughly 53,000 parameters and works well for MNIST or similar datasets. The weight sharing and spatial locality of convolution make this efficient.

Comparing Pooling Strategies and Trade-offs

Pooling is a critical design choice. Here is a detailed comparison:

| Pooling Type | Operation | Gradient Flow | Translation Robustness | Preserves Detail | Use Case | |---|---|---|---|---|---| | Max pooling | Max value in window | Sparse (only max position) | High: small shifts still activate | Moderate | Hidden layers; feature detection | | Average pooling | Mean value in window | Dense (all positions) | Lower: depends on average | High | Global pooling before classification; smoother features | | L2 pooling | √(sum of squares) | Dense | Moderate | Moderate | Rare; mostly historical interest | | No pooling (stride conv) | Stride > 1 in conv | Dense; fully learnable | Depends on kernel learning | High | Modern architectures (ResNet, EfficientNet) |

Trade-off analysis:

• Max pooling is aggressive: it discards information but provides robustness to small translations (if a feature shifts one pixel, max pooling may still catch it in the window).
• Average pooling is conservative: it retains more information but offers less translation robustness.
• Stride-based convolution (using stride > 1 in Conv2d) is fully learnable and modern architectures often prefer it to explicit pooling, as the network can learn the best downsampling strategy.

For typical image classification (e.g., MNIST, CIFAR-10, ImageNet), max pooling with stride=2 and 2×2 windows is standard and performs well. For tasks requiring fine detail (medical imaging, object detection), consider using stride-based convolution or smaller pooling windows (1×1 max pooling does nothing; 2×2 is minimal).

Receptive Field: Understanding What Each Neuron Sees

A neuron in layer L has a receptive field — the region of the input image it "sees" through the cascade of convolutions. This is critical for designing architectures.

The receptive field grows with each layer. For a 3×3 kernel with stride 1:

• Layer 1: receptive field = 3×3
• Layer 2: receptive field = 5×5 (each neuron in L2 sees a 3×3 patch in L1, which is a 5×5 patch in the input)
• Layer 3: receptive field = 7×7

For a network to learn global patterns (e.g., "is this a dog?"), neurons deep in the network need a receptive field covering the entire image or most of it. With k layers of 3×3 convolutions and stride 1, the receptive field is approximately (2k + 1) × (2k + 1). For a 224×224 image (ImageNet size), you need roughly 110 layers of 3×3 to cover the full image, which is why very deep networks (ResNet-152) are common.

Using stride > 1 or larger kernels (5×5, 7×7) rapidly increases receptive field:

• 7×7 kernel, stride 1: receptive field = 7×7 at layer 1
• 3×3 kernel with stride 2: receptive field doubles per layer, reaching 224×224 in ~8 layers

This is why modern architectures often use larger kernels early (7×7 in ResNet) or stride-2 convolutions to quickly expand receptive field and reduce computation.

Common mistake: forgetting to account for dimension changes

Convolution changes spatial dimensions. Many mistakes arise from forgetting the formula:

output_size = (input_size - kernel_size + 2*padding) / stride + 1

If you design a CNN and expect a particular feature map size for the final fully connected layer but use the wrong stride or padding, the linear layer will have mismatched input dimensions and crash. Always trace the dimensions through the network (as done in the SimpleCNN example above: 28 → 28 → 14 → 14 → 7 → 7 → flatten to 1568) before writing the model.

Another mistake: using stride or padding so aggressive that spatial dimensions become too small (e.g., reducing a 28×28 input to 1×1 too early), losing fine detail that could be useful. Generally, reserve large stride for the final pooling or when you explicitly want to downsample.

One more: confusing kernel size with stride. A 5×5 kernel with stride=1 moves one pixel and is expensive; the same kernel with stride=2 or stride=3 reduces computation but skips spatial detail.