Recurrent Networks and the Unrolling Trick

The Recurrent Loop: Processing Sequences One Step at a Time

A recurrent neural network (RNN) processes sequences by maintaining a hidden state that evolves as it reads each element. At each time step t, the RNN takes the current input x_t and the previous hidden state h_, computes a new hidden state h_t, and optionally produces an output y_t.

The key insight is weight sharing: the same weights are reused at every time step. This is different from a feedforward network, where each layer has its own weight matrix. In an RNN, you use the same weight matrix W_hh (hidden-to-hidden) and W_xh (input-to-hidden) across all time steps. For a sequence of length 100, a feedforward network would have 100 separate weight matrices; an RNN reuses a single set, reducing parameters by ~100x.

The recurrent cell equation

At time step t, the vanilla RNN cell (also called Elman RNN) computes:

h_t = tanh(W_xh · x_t + W_hh · h_{t-1} + b_h)
y_t = W_hy · h_t + b_y

• W_xh: maps input to hidden (shape: hidden_size × input_size)
• W_hh: maps hidden to hidden (shape: hidden_size × hidden_size) — this is the recurrent weight
• W_hy: maps hidden to output (shape: output_size × hidden_size)
• b_h, b_y: biases
• tanh: a common activation (though ReLU, sigmoid, or others are possible)

The hidden state h_t is the "memory" of the sequence so far. It accumulates information from all previous inputs through the recurrent connection. The initialization is critical: typically h_0 = 0 (zero hidden state) before processing begins.

RNN vs. feedforward: parameter efficiency

A feedforward sequence processor (separate weights per time step) on a sequence of length T has:

• T layers × (input_size × hidden_size + hidden_size × output_size) = T × (I×H + H×O) parameters

An RNN with weight sharing has:

• (input_size × hidden_size + hidden_size × hidden_size + hidden_size × output_size) = I×H + H² + H×O parameters

For T=100, I=50, H=100, O=10:

• Feedforward: 100 × (50×100 + 100×10) = 100 × 6000 = 600,000 parameters
• RNN: 50×100 + 100² + 100×10 = 5000 + 10000 + 1000 = 16,000 parameters (~37x fewer)

Unrolling: From Loop to Computation Graph

To train an RNN with backpropagation, you unroll the recurrent loop into a feedforward chain. Imagine a sequence of length T. Instead of thinking of a single cell that runs T times, imagine T copies of the cell stacked vertically, each one taking the previous cell's hidden state as input.

Time:      t=0    t=1    t=2    ...  t=T-1
Input:     x_0    x_1    x_2         x_{T-1}
           |      |      |           |
           v      v      v           v
Hidden:  h_0 --> h_1 --> h_2 --> ... --> h_T
Output:    y_0    y_1    y_2         y_{T-1}

Each arrow is a copy of the same weight matrix (W_xh, W_hh, W_hy). Now, to compute gradients, apply backpropagation through this unrolled graph. Errors flow backward through time: the gradient for h_t depends on the gradient from y_t and the gradient flowing back from h_.

This is called backpropagation through time (BPTT). It is not a separate algorithm — it is just backprop on the unrolled graph. But the unrolling reveals a critical problem.

The Vanishing Gradient Problem

In the unrolled view, the gradient of the loss with respect to h_0 (the hidden state many steps ago) must pass through multiple multiplications:

∂L/∂h_0 = ∂L/∂h_T · ∂h_T/∂h_{T-1} · ∂h_{T-1}/∂h_{T-2} · ... · ∂h_1/∂h_0

Expanding ∂h_t/∂h_ via the chain rule:

∂h_t/∂h_{t-1} = tanh'(z_t) · W_hh

where z_t = W_xh · x_t + W_hh · h_ + b_h. The derivative of tanh is bounded: tanh'(z) ≤ 1 for all z. The singular values of W_hh often satisfy σ_max(W_hh) < 1 (unless initialized carefully).

Multiplying T such terms (one per time step):

∂L/∂h_0 ≈ (tanh'_1 · W_hh) · (tanh'_2 · W_hh) · ... · (tanh'_T · W_hh)
        ≈ tanh'_1 · tanh'_2 · ... · tanh'_T · W_hh^T

If each tanh' ≤ 0.5 and σ_max(W_hh) ≤ 0.9, the product is roughly (0.5 × 0.9)^T ≈ 0.45^T. For T=50, this is ~10^, a vanishing gradient.

Concrete impact: On a 100-step sequence, a vanilla RNN cannot learn dependencies beyond ~20 steps (Hochreiter et al., 2001). Time-series or language modeling tasks that require understanding context from 50+ steps ago fail with vanilla RNNs.

This is vanishing gradient. An opposite problem (exploding gradient, where multiplying large matrices causes overflow) also exists and is addressed with gradient clipping (norm-based or value-based).

Exploding gradient is less common but can occur if W_hh has very large singular values. Gradient clipping keeps gradients bounded during training. Vanishing gradient is more fundamental and requires architectural changes.

Gating mechanisms like LSTM and GRU were designed to solve this by introducing additive paths (cell state) that bypass multiplicative interactions across time steps.

Example: Implementing a Basic RNN Cell from Scratch

Here is a minimal RNN cell to understand the mechanics:

import numpy as np

class SimpleRNNCell:
    def __init__(self, input_size, hidden_size):
        self.hidden_size = hidden_size

        # Initialize weights randomly (in practice, use proper initialization)
        self.W_xh = np.random.randn(hidden_size, input_size) * 0.01
        self.W_hh = np.random.randn(hidden_size, hidden_size) * 0.01
        self.W_hy = np.random.randn(input_size, hidden_size) * 0.01
        # Note: W_hy shape for simplicity (could be output_size, hidden_size)

        self.b_h = np.zeros((hidden_size, 1))
        self.b_y = np.zeros((input_size, 1))

    def forward_step(self, x_t, h_prev):
        """
        One forward step of the RNN cell.

        Args:
            x_t: input at time t, shape (input_size, 1)
            h_prev: hidden state from t-1, shape (hidden_size, 1)

        Returns:
            h_t: new hidden state, shape (hidden_size, 1)
            y_t: output at time t, shape (input_size, 1)
        """
        # Compute new hidden state
        h_t = np.tanh(
            np.dot(self.W_xh, x_t) +
            np.dot(self.W_hh, h_prev) +
            self.b_h
        )

        # Compute output (linear readout from hidden state)
        y_t = np.dot(self.W_hy, h_t) + self.b_y

        return h_t, y_t

    def forward_sequence(self, sequence):
        """
        Process an entire sequence.

        Args:
            sequence: list of (input_size, 1) vectors

        Returns:
            outputs: list of outputs
            hidden_states: list of hidden states at each step
        """
        h = np.zeros((self.hidden_size, 1))
        outputs = []
        hidden_states = [h]

        for x_t in sequence:
            h, y = self.forward_step(x_t, h)
            outputs.append(y)
            hidden_states.append(h)

        return outputs, hidden_states

# Example: process a sequence of 5 vectors, each of dimension 3
input_size, hidden_size = 3, 4
cell = SimpleRNNCell(input_size, hidden_size)

# Create a random sequence of 5 time steps
sequence = [np.random.randn(input_size, 1) for _ in range(5)]

outputs, hidden_states = cell.forward_sequence(sequence)
print(f"Sequence length: {len(sequence)}")
print(f"Output at t=0 shape: {outputs[0].shape}")  # (3, 1)
print(f"Hidden state at t=4 shape: {hidden_states[4].shape}")  # (4, 1)

This shows the core loop: maintain hidden state h, update it with each input, produce outputs. The unrolled graph for training would compute gradients backward through all these steps.

PyTorch's nn.RNN

In practice, use PyTorch's optimized implementations:

import torch
import torch.nn as nn

# Define an RNN layer: 3 input features, 4 hidden units, 1 layer
rnn_layer = nn.RNN(
    input_size=3,
    hidden_size=4,
    num_layers=1,
    batch_first=False  # Default: (seq_len, batch, features)
)

# Sequence of 5 time steps, batch size 2, input size 3
input_seq = torch.randn(5, 2, 3)
h_0 = torch.zeros(1, 2, 4)  # Initial hidden state

# Forward pass
output, h_final = rnn_layer(input_seq, h_0)
print(f"Input shape: {input_seq.shape}")    # (5, 2, 3)
print(f"Output shape: {output.shape}")      # (5, 2, 4)
print(f"Final hidden state shape: {h_final.shape}")  # (1, 2, 4)

# output contains all hidden states h_t for t=0..T-1
# h_final is the last hidden state (same as output[-1])

The output tensor contains the hidden state at each time step, useful if you want to apply a decoder at every time step (sequence-to-sequence). The h_final is the last hidden state, useful if you want to pass it to the next layer or decode once at the end.

LSTM and GRU: Gated Recurrent Cells

LSTM (Long Short-Term Memory) adds gates to control information flow. The key innovation is a separate cell state c_t that is updated additively (not multiplicatively), avoiding the gradient shrinkage problem.

LSTM equations:

i_t = sigmoid(W_xi · x_t + W_hi · h_{t-1} + b_i)    # Input gate
f_t = sigmoid(W_xf · x_t + W_hf · h_{t-1} + b_f)    # Forget gate
o_t = sigmoid(W_xo · x_t + W_ho · h_{t-1} + b_o)    # Output gate
c̃_t = tanh(W_xc · x_t + W_hc · h_{t-1} + b_c)       # Cell candidate

c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t                      # Cell state (additive update)
h_t = o_t ⊙ tanh(c_t)                                 # Hidden state

Where ⊙ is element-wise multiplication and sigmoid outputs values in [0, 1] (gate activations).

The critical difference: c_t is updated additively (addition and element-wise multiplication), not through matrix multiplication. The gradient for c_t backpropagates through addition (∂c_t/∂c_ = f_t, bounded by 1), not matrix multiplication. This allows gradients to flow across 100+ time steps without vanishing.

Benchmark: On Penn Treebank language modeling, vanilla RNNs plateau at ~150 perplexity; LSTMs achieve ~90 perplexity (lower is better). On sequences > 50 steps, LSTM outperforms RNN consistently.

| Architecture | Max Sequence Length | Vanishing Gradient | Parameters (per step) | Training Stability | |---|---|---|---|---| | Vanilla RNN | ~20 steps | Severe | H² (minimal) | Poor on long sequences | | LSTM | ~100+ steps | Minimal (additive c_t) | 4×H² (4 gates) | Excellent; widely used | | GRU | ~100+ steps | Minimal (additive update) | 3×H² (3 gates) | Excellent; faster than LSTM | | Bidirectional LSTM | ~100+ steps (both directions) | Minimal | 8×H² (4 gates × 2 directions) | Excellent for non-causal tasks |

GRU (Gated Recurrent Unit) is a simpler variant with only two gates (reset and update), combining cell state and hidden state into one vector:

z_t = sigmoid(W_xz · x_t + W_hz · h_{t-1} + b_z)    # Update gate
r_t = sigmoid(W_xr · x_t + W_hr · h_{t-1} + b_r)    # Reset gate
h̃_t = tanh(W_x · x_t + W_h · (r_t ⊙ h_{t-1}) + b)  # Candidate activation

h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ h̃_t              # Additive update

GRU has fewer parameters (3× vs. 4× for LSTM) and trains slightly faster, with comparable long-term dependency learning. GRU is preferred when compute is constrained or when gradient stability is not the primary concern.

# LSTM layer: same interface as RNN
lstm_layer = nn.LSTM(
    input_size=3,
    hidden_size=4,
    num_layers=1,
    batch_first=False
)

# For LSTM, initial state is (h_0, c_0): hidden and cell states
h_0 = torch.zeros(1, 2, 4)
c_0 = torch.zeros(1, 2, 4)

output, (h_final, c_final) = lstm_layer(input_seq, (h_0, c_0))
print(f"LSTM output shape: {output.shape}")  # (5, 2, 4)
print(f"Final hidden state shape: {h_final.shape}")  # (1, 2, 4)
print(f"Final cell state shape: {c_final.shape}")    # (1, 2, 4)

# Compare: RNN output is (5, 2, 4)
# LSTM output is also (5, 2, 4), same interface but with gating internally

LSTM cells are more expensive computationally (4 times more parameters per unit than simple RNN due to the 4 gate matrices), but they learn long-range dependencies much more reliably.

Practical RNN Design Patterns

Different tasks call for different RNN configurations. Here is a guide:

| Task Type | Recommended Architecture | Input Shape | Output Usage | Example | |---|---|---|---|---| | Classification | RNN → final hidden state → linear layer | (batch, seq_len, input_size) | h_final into softmax | Sentiment analysis, intent detection | | Sequence labeling | RNN, output at each step | (batch, seq_len, input_size) | All output[:, t, :] | Named entity recognition, POS tagging | | Sequence-to-sequence | Encoder RNN → Decoder RNN | Input: (batch, src_len, input_size); Output: (batch, tgt_len, input_size) | Encoder's h_final seeds decoder | Machine translation, summarization | | Time series forecasting | RNN with autoregressive prediction | (batch, history_len, input_size) | Final hidden state + projection | Stock price prediction, weather forecasting | | Bidirectional | Bi-LSTM (forward + backward RNNs) | (batch, seq_len, input_size) | Forward + backward hidden states concatenated | Sentiment analysis (full context), NER (full context) |

Example confusion: Using unidirectional RNN for a task that needs full context (past + future). For example, detecting whether a word is a noun requires looking at words before AND after it. Always use bidirectional RNNs for such tasks.

Common RNN Gotchas and How to Avoid Them

| Mistake | Symptom | Fix | |---|---|---| | Uninitialized hidden state across batches | Model ignores examples from previous batch | Initialize h_0 = torch.zeros() per batch, not reused | | Sequence length mismatch in batch | Shape error or wrong dimensions | Pad sequences to same length, use packing (pack_padded_sequence) | | Vanishing gradients on long sequences (vanilla RNN) | Model can't learn dependencies > 20 steps | Use LSTM or GRU instead | | Exploding gradients | Loss becomes NaN; training diverges | Use gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) | | Wrong activation function | Poor convergence; poor gradients | tanh (default) usually works; ReLU can cause exploding gradients; avoid sigmoid | | Forgetting to call .eval() at inference | Inflated performance; wrong predictions | Call model.eval() before evaluation; disables dropout/batchnorm | | Bidirectional RNN used for autoregressive task | Model sees future tokens it should predict | Use unidirectional RNN; don't look forward in causal/next-token tasks |

A Sequence-to-Sequence Example

Here is a toy example of using RNN/LSTM to process a sequence:

import torch
import torch.nn as nn
import torch.optim as optim

class SimpleSeqEncoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_size):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_size, batch_first=True)

    def forward(self, x):
        """
        Args:
            x: (batch, seq_len) of token indices

        Returns:
            context: (batch, hidden_size) - the final hidden state
        """
        embed = self.embedding(x)  # (batch, seq_len, embed_dim)
        _, (h_final, _) = self.lstm(embed)  # h_final: (1, batch, hidden_size)
        context = h_final.squeeze(0)  # (batch, hidden_size)
        return context

# Example: vocab of 100 tokens, embed dim 16, hidden size 32
encoder = SimpleSeqEncoder(vocab_size=100, embed_dim=16, hidden_size=32)
x = torch.randint(0, 100, (8, 10))  # Batch of 8 sequences, length 10
context = encoder(x)
print(f"Context shape: {context.shape}")  # (8, 32)

The encoder reads a sequence and produces a context vector (the final hidden state) that summarizes the sequence. This is used in tasks like sentiment analysis (classify the context), machine translation (pass context to a decoder), or any sequence-to-fixed-output task.

Common mistake: not thinking about sequence length when initializing

RNNs require all sequences in a batch to have the same length (or use padding and masking). If you forget to pad shorter sequences, PyTorch will either error or silently produce wrong results. Always check your data shape: (batch, seq_len, features).

Another mistake: using a simple RNN for very long sequences (T > 100 or so) and expecting it to work. The vanishing gradient problem kicks in hard. Always use LSTM or GRU for sequences longer than ~30 steps.

Also: don't forget to initialize the hidden state. If you pass None, PyTorch initializes it to zero, which is usually correct. But if you are chaining multiple RNN calls and want to maintain state across batches (unusual, but possible), you need to save and reuse the final hidden state.