Backpropagation: How Networks Learn

The Two Passes: Forward, Then Backward

Training a neural network is a cycle of two passes:

1. Forward pass: Feed input through the network to compute predictions and loss.
2. Backward pass: Compute gradients of the loss with respect to every weight by propagating derivatives back through the network using the chain rule.

The backward pass is called backpropagation. Without it, training deep networks would be intractable—you would have no way to know how to adjust the weights to reduce loss.

The diagram shows the forward pass (left to right) computing predictions and loss, then the backward pass (right to left) computing gradients. Gradients flow backward through each layer, informing weight updates.

The Chain Rule: Propagating Derivatives

Backpropagation relies on the chain rule from calculus. If loss L depends on weights w through intermediate values (predictions, neuron outputs), the chain rule tells us how to compute ∂L/∂w:

∂L/∂w = ∂L/∂y_pred * ∂y_pred/∂hidden * ∂hidden/∂w

Each fraction is a local gradient at one layer. By multiplying these together, we compute the full gradient from the loss all the way back to the weights.

General form (multi-layer): If we have a chain of functions:

L = loss(y_pred)
y_pred = f_n(h_{n-1})
h_{n-1} = f_{n-1}(h_{n-2})
...
h_1 = f_1(X, W_1)

Then the gradient of loss with respect to W_i is:

∂L/∂W_i = (∂L/∂y_pred) * (∂y_pred/∂h_{n-1}) * ... * (∂h_{i+1}/∂h_i) * (∂h_i/∂W_i)

This is a product of n terms. Backpropagation computes these products efficiently by working backward from the loss, storing intermediate results.

From Scratch: A Two-Layer Network with Backpropagation

Let's implement a tiny network with one hidden layer, compute the forward pass, loss, and backward pass manually.

import numpy as np

# Set random seed for reproducibility
np.random.seed(42)

# Sample data
X = np.array([[1.0, 0.5],
              [2.0, 1.5],
              [3.0, 2.0]])  # 3 samples, 2 features
y = np.array([[1.0],
              [1.0],
              [0.0]])  # 3 targets (binary)

# Network architecture: 2 inputs -> 3 hidden -> 1 output
W1 = np.random.randn(2, 3) * 0.1  # Input to hidden: 2x3
b1 = np.zeros((1, 3))  # Hidden bias: 1x3
W2 = np.random.randn(3, 1) * 0.1  # Hidden to output: 3x1
b2 = np.zeros((1, 1))  # Output bias: 1x1

def sigmoid(z):
    """Sigmoid activation."""
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))  # Clip to prevent overflow

def sigmoid_derivative(a):
    """Derivative of sigmoid with respect to its input."""
    return a * (1 - a)

learning_rate = 0.1

# Training loop
for epoch in range(100):
    # ========== FORWARD PASS ==========
    # Layer 1: hidden = sigmoid(X @ W1 + b1)
    Z1 = X @ W1 + b1
    A1 = sigmoid(Z1)

    # Layer 2: output = sigmoid(A1 @ W2 + b2)
    Z2 = A1 @ W2 + b2
    A2 = sigmoid(Z2)

    # Loss: binary cross-entropy
    epsilon = 1e-7
    loss = -np.mean(y * np.log(A2 + epsilon) + (1 - y) * np.log(1 - A2 + epsilon))

    # ========== BACKWARD PASS (BACKPROPAGATION) ==========
    # The chain rule: compute gradients layer by layer

    # Gradient of loss with respect to output activation
    dA2 = -(y / (A2 + epsilon) - (1 - y) / (1 - A2 + epsilon)) / y.shape[0]

    # Gradient of output with respect to Z2 (chain rule: dL/dZ = dL/dA * dA/dZ)
    dZ2 = dA2 * sigmoid_derivative(A2)

    # Gradients for W2 and b2
    dW2 = A1.T @ dZ2
    db2 = np.sum(dZ2, axis=0, keepdims=True)

    # Backprop to hidden layer
    dA1 = dZ2 @ W2.T

    # Gradient of hidden activation with respect to Z1
    dZ1 = dA1 * sigmoid_derivative(A1)

    # Gradients for W1 and b1
    dW1 = X.T @ dZ1
    db1 = np.sum(dZ1, axis=0, keepdims=True)

    # ========== WEIGHT UPDATE ==========
    W1 -= learning_rate * dW1
    b1 -= learning_rate * db1
    W2 -= learning_rate * dW2
    b2 -= learning_rate * db2

    if (epoch + 1) % 20 == 0:
        print(f"Epoch {epoch+1}: Loss={loss:.4f}")

# After training, check predictions
predictions = sigmoid(sigmoid(X @ W1 + b1) @ W2 + b2)
print(f"\nTraining complete!")
print(f"Final predictions: {predictions.flatten()}")
print(f"Targets: {y.flatten()}")

Key steps in the backward pass:

1. Compute dA2 (gradient of loss w.r.t. output activation).
2. Compute dZ2 = dA2 * sigmoid'(A2) (chain rule).
3. Compute dW2 and db2 (gradients for output layer weights).
4. Backprop to hidden layer: dA1 = dZ2 @ W2.T (chain rule).
5. Compute dZ1 = dA1 * sigmoid'(A1).
6. Compute dW1 and db1 (gradients for hidden layer weights).
7. Update all weights: w -= learning_rate * dw.

This is the essence of backpropagation: the chain rule applied systematically to every layer, backward from loss to input.

Understanding the Dimensions

The matrix dimensions in backpropagation can be confusing. Here is a mental model:

• dW has the same shape as W because the gradient tells us how much to adjust each weight.
• A @ dB means: each output gradient flows back through all relevant inputs.
• dZ @ W.T transposes: because we need to distribute the upstream gradient to each input.

Let's trace one example:

• X has shape (3, 2) — 3 samples, 2 features.
• W1 has shape (2, 3) — maps 2 inputs to 3 hidden units.
• A1 = sigmoid(X @ W1 + b1) has shape (3, 3) — 3 samples, 3 hidden units.
• dZ1 has shape (3, 3) — gradient w.r.t. hidden pre-activations.
• dW1 = X.T @ dZ1 has shape (2, 3) — same as W1, as expected.

The transpose operations ensure that gradients align correctly with weight shapes.

From Manual Backprop to PyTorch Autograd

Manually computing backprop for a two-layer network was already complex. For a real network with dozens of layers, doing this by hand is infeasible. This is where automatic differentiation comes in.

PyTorch's autograd module automatically computes gradients using backpropagation. You write the forward pass, and PyTorch builds a computational graph. Then loss.backward() automatically differentiates the entire graph:

import torch
import torch.nn as nn
import torch.optim as optim

# Same data as before, but as PyTorch tensors
X = torch.tensor([[1.0, 0.5],
                  [2.0, 1.5],
                  [3.0, 2.0]])
y = torch.tensor([[1.0], [1.0], [0.0]])

# Define the network
class TinyNet(nn.Module):
    def __init__(self):
        super(TinyNet, self).__init__()
        self.fc1 = nn.Linear(2, 3)
        self.fc2 = nn.Linear(3, 1)

    def forward(self, x):
        x = torch.sigmoid(self.fc1(x))
        x = torch.sigmoid(self.fc2(x))
        return x

model = TinyNet()
criterion = nn.BCELoss()  # Binary cross-entropy
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Training loop (much simpler!)
for epoch in range(100):
    # Forward pass
    predictions = model(X)
    loss = criterion(predictions, y)

    # Backward pass (PyTorch computes all gradients automatically)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (epoch + 1) % 20 == 0:
        print(f"Epoch {epoch+1}: Loss={loss.item():.4f}")

print("\nTraining complete!")
predictions = model(X)
print(f"Final predictions: {predictions.detach().flatten()}")
print(f"Targets: {y.flatten()}")

The PyTorch version is dramatically simpler: you define the forward pass, call loss.backward(), and the gradients are computed automatically. Under the hood, PyTorch is doing exactly the manual backpropagation we wrote above, but for thousands of layers if needed.

Comparison: Manual vs. Automatic Backprop

| Aspect | Manual NumPy | PyTorch Autograd | |--------|------|------| | Code length (for 2-layer net) | ~50 lines | ~10 lines | | Prone to errors? | Yes (manual ∂ calculations) | No (automatic) | | Scales to 100+ layers? | Impractical | Trivial | | Debugging difficulty | Hard (track shapes, gradients) | Medium (can inspect autograd graph) | | Performance | Good (optimized NumPy) | Excellent (fused operations) | | Learning value | Very high | Medium (black-box) |

For learning, implementing backprop manually in NumPy is invaluable; it builds intuition. For production, use PyTorch (or TensorFlow). The takeaway: you now understand what PyTorch does automatically.

The Computational Graph

PyTorch maintains a computational graph as you compute values. Each tensor that requires gradients (requires_grad=True) records which operations created it. When you call backward(), PyTorch traverses this graph in reverse, computing gradients at each node.

# Simple example of autograd
x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 3 * x + 1

# y = 2^2 + 3*2 + 1 = 4 + 6 + 1 = 11
print(f"Forward pass: x={x.item()}, y={y.item()}")

# Backward pass: compute dy/dx
y.backward()

# dy/dx = 2*x + 3 = 2*2 + 3 = 7
print(f"Gradient: dy/dx = {x.grad.item()}")  # Should be 7

PyTorch has automatically computed that dy/dx = 7 by applying the chain rule: dy/dx = (d/dx)(x^2) + (d/dx)(3x) + (d/dx)(1) = 2x + 3 = 7.

Why Backpropagation Made Deep Learning Possible

Before backpropagation was popularized in the mid-1980s, training deep networks was nearly impossible. For a 10-layer network with thousands of parameters, computing gradients by hand or by finite differences (small perturbations) was prohibitively slow.

Backpropagation solved this: it computes all gradients in a single backward pass, with computational cost similar to the forward pass. This made training deep networks practical. Without it, modern deep learning would not exist.

Layer-wise Gradient Flow: Visualizing Backprop

To understand backpropagation intuitively, let's visualize how gradients flow backward through layers.

# Simulate a 3-layer network and track gradient flow
class SimpleBackpropVisualizer:
    def __init__(self):
        self.layer1_W = np.random.randn(2, 3) * 0.1
        self.layer2_W = np.random.randn(3, 2) * 0.1
        self.layer3_W = np.random.randn(2, 1) * 0.1

    def forward(self, x):
        # Layer 1
        self.z1 = x @ self.layer1_W
        self.a1 = np.maximum(0, self.z1)  # ReLU

        # Layer 2
        self.z2 = self.a1 @ self.layer2_W
        self.a2 = np.maximum(0, self.z2)  # ReLU

        # Layer 3 (output)
        self.a3 = self.a2 @ self.layer3_W
        return self.a3

    def backward(self, x, y):
        batch_size = x.shape[0]

        # Loss: MSE
        loss = np.mean((self.a3 - y) ** 2)

        # Backprop output layer
        da3 = 2 * (self.a3 - y) / batch_size
        dz3 = da3  # No activation in output
        dW3 = self.a2.T @ dz3

        # Backprop layer 2
        da2 = dz3 @ self.layer3_W.T
        dz2 = da2 * (self.z2 > 0)  # ReLU derivative
        dW2 = self.a1.T @ dz2

        # Backprop layer 1
        da1 = dz2 @ self.layer2_W.T
        dz1 = da1 * (self.z1 > 0)  # ReLU derivative
        dW1 = x.T @ dz1

        return loss, dW1, dW2, dW3

# Test
viz = SimpleBackpropVisualizer()
x = np.random.randn(5, 2)
y = np.random.randn(5, 1)

output = viz.forward(x)
loss, dW1, dW2, dW3 = viz.backward(x, y)

print("Gradient norms by layer:")
print(f"Layer 1 gradient norm: {np.linalg.norm(dW1):.4f}")
print(f"Layer 2 gradient norm: {np.linalg.norm(dW2):.4f}")
print(f"Layer 3 gradient norm: {np.linalg.norm(dW3):.4f}")

Observe the gradient norms across layers. In healthy training, they should be of similar magnitude (tens of thousands of layers, this changes, leading to vanishing/exploding gradients).

Numerical Gradient Checking

When you implement backpropagation (or debug it), a useful sanity check is numerical gradient checking. Compute gradients numerically via finite differences and compare with your analytical gradients:

def numerical_gradient(func, x, epsilon=1e-5):
    """Compute gradient numerically via finite differences."""
    grad = np.zeros_like(x)
    for i in range(x.size):
        x_plus = x.copy()
        x_plus.flat[i] += epsilon
        x_minus = x.copy()
        x_minus.flat[i] -= epsilon
        grad.flat[i] = (func(x_plus) - func(x_minus)) / (2 * epsilon)
    return grad

# Example: check gradient of W2
def loss_fn(W2):
    Z2 = A1 @ W2 + b2
    A2 = sigmoid(Z2)
    return -np.mean(y * np.log(A2 + 1e-7) + (1 - y) * np.log(1 - A2 + 1e-7))

numerical_grad_W2 = numerical_gradient(loss_fn, W2, epsilon=1e-5)

# Compare with backprop gradient
print("Analytical gradient (from backprop):")
print(dW2)
print("\nNumerical gradient (finite differences):")
print(numerical_grad_W2)
print("\nDifference (should be very small):")
print(np.abs(dW2 - numerical_grad_W2))

If the difference is less than 1e-4, your backpropagation is likely correct. This check is slow (requires forward passes for every parameter), so use it only during development.

Vanishing and Exploding Gradients

A critical consequence of backpropagation is that gradients must travel through many layers. In deep networks, this can cause problems.

Vanishing gradients: In very deep networks (20+ layers), gradients shrink exponentially as they backprop. Each layer multiplies the upstream gradient by the activation's derivative (often < 1 for sigmoid/tanh). After 20 multiplications of 0.25, the gradient is ~10^-10—effectively zero. Early layers receive no meaningful gradient and do not learn.

Exploding gradients: Conversely, if weight matrices are initialized with large values or have large eigenvalues, gradients can grow exponentially, causing weights to diverge to NaN.

Mitigations:

1. Use ReLU instead of sigmoid/tanh (ReLU has gradient = 1 for positive inputs).
2. Batch normalization (see future lessons): Rescales activations to have mean 0, std 1, stabilizing backprop.
3. Gradient clipping: Cap the norm of gradients before applying updates.

   torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
   

4. Careful initialization (He/Xavier): Start weights in a range that prevents extreme gradients.

Example: Gradient clipping in a training loop

for epoch in range(100):
    optimizer.zero_grad()
    predictions = model(X)
    loss = criterion(predictions, y)
    loss.backward()

    # Clip gradients if norms exceed 1.0
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

    optimizer.step()

For networks with 5–10 layers and ReLU, vanishing/exploding gradients are rare. For very deep networks (>50 layers), batch normalization is almost essential.

Loss functions and their derivatives (for output layer backprop):

| Loss Function | Formula | Gradient (w.r.t. output) | When to Use | Output Range | |---|---|---|---|---| | MSE (L2) | (1/n) Σ(y_pred - y)² | (2/n)(y_pred - y) | Regression | (-∞, ∞) | | MAE (L1) | (1/n) Σ|y_pred - y| | sign(y_pred - y) / n | Regression (robust to outliers) | (-∞, ∞) | | Binary Cross-Entropy | -[y*log(p) + (1-y)*log(1-p)] | (p - y) / n | Binary classification | (0, 1) with sigmoid | | Cross-Entropy | -Σ(y * log(p)) | (p - y) / n | Multi-class classification | Probability dist. with softmax | | Huber | Smooth L1 (L2 for small errors) | Smooth blend | Regression with outliers | (-∞, ∞) |

The gradient formula determines how backprop flows. Simpler gradients (like MSE) lead to faster training; more complex ones may have numerical issues.

A Pedagogical Aside: Why Sigmoid Causes Vanishing Gradients

Let's quantify why sigmoid is problematic for deep networks:

# Simulate 10 layers of sigmoid backprop
gradient = 1.0  # Upstream gradient (from loss)
sigmoid_max_derivative = 0.25  # Max derivative of sigmoid

for layer in range(10):
    gradient *= sigmoid_max_derivative
    print(f"After layer {layer+1}: gradient = {gradient:.2e}")

After 10 layers: gradient ≈ 10^-7. The early-layer weights barely update. This is why ReLU (derivative = 1 for positive inputs) enabled training much deeper networks.

Gradient Magnitude and Training Dynamics

Not all gradients are equal. A weight with a large gradient update will change drastically; one with a tiny gradient barely changes.

# Print gradient magnitudes during training
model = TinyNet()
optimizer = optim.SGD(model.parameters(), lr=0.01)

for epoch in range(10):
    predictions = model(X)
    loss = criterion(predictions, y)
    optimizer.zero_grad()
    loss.backward()

    # Inspect gradients
    for name, param in model.named_parameters():
        if param.grad is not None:
            grad_norm = param.grad.norm().item()
            param_norm = param.data.norm().item()
            if epoch == 0 or epoch == 9:
                print(f"Epoch {epoch+1}, {name}: param_norm={param_norm:.4f}, grad_norm={grad_norm:.4f}")

    optimizer.step()

If gradients are consistently tiny, your learning rate is too low or you have vanishing gradients. If they are huge, you may have exploding gradients or a very steep loss landscape.

Common Mistake: Forgetting to Zero Gradients

In PyTorch, gradients accumulate by default:

# WRONG: Gradients accumulate
x = torch.tensor(2.0, requires_grad=True)
y1 = x ** 2
y1.backward()
print(f"First backward: x.grad = {x.grad}")  # 4.0

y2 = x ** 3
y2.backward()
print(f"Second backward: x.grad = {x.grad}")  # 4.0 + 6.0 = 10.0, not 6.0!

The second backward() adds to the existing gradient. In a training loop, you must zero gradients before each iteration:

# RIGHT: Zero gradients before backward
for epoch in range(100):
    optimizer.zero_grad()  # Clear old gradients
    predictions = model(X)
    loss = criterion(predictions, y)
    loss.backward()  # Compute new gradients
    optimizer.step()  # Update weights

Forgetting optimizer.zero_grad() is a common source of training bugs that silently corrupt the learning process.