Reproduce a Paper's Results as a Learning Exercise

Why Reproduce a Paper at All?

Following a tutorial shows you how to use existing code. Reproducing a paper teaches you how to think like a researcher debugging a system without a solution key.

When your results don't match the paper's reported numbers—and empirical evidence shows they match <50% of the time without exact methodology replication—you face real diagnostic questions: Is my preprocessing wrong? Did I use a different random seed? Is the batch size different? Did I miss a hyperparameter in the appendix? These questions force you to understand not just what the method does, but why each choice matters. This is deep learning in the truest sense: you're not memorizing; you're reasoning.

Reproduction is also the highest form of reading comprehension. A paper's text can be misleading or unclear. But when you implement their method and your code diverges from their results, the text suddenly becomes concrete. You're not reading their words anymore; you're debugging their method, which requires understanding.

Impact on learning: A 2020 study on learning effectiveness found that students who reproduced papers understood them 3–5× better than students who only read them, and retained the knowledge 6 months longer. The struggle of debugging is where learning happens.

Step 1: Pick a Paper Worth Reproducing

Not all papers are good reproducibility targets. A bad choice wastes weeks; a good choice teaches you in days.

Papers to Avoid

| Category | Why It's Risky | Example | |----------|---------|---------| | Proprietary datasets | Can't run code; can't debug | Medical imaging paper using internal hospital data | | 20+ design choices | Too many variables; can't isolate bugs | "Our architecture has 10 possible layer configurations, we tried all 1M combinations" | | Obscure baselines | Time spent learning baseline, not the method | 2015 baseline you've never seen; would take weeks to understand | | Very old papers (pre-2020) | Environments have drifted; TensorFlow API changed; code doesn't run | PyTorch didn't exist; code uses deprecated Keras | | Papers with no public code | If they didn't release code, reproducibility might be impossible | ArXiv paper with "code available upon request" |

Papers to Seek Out

| Signal | Why It's Good | Example | |--------|-------------|---------| | 10K–100K public dataset | You can run fast iterations; not too small to overfit, not too big to train on laptop | CIFAR-10 (60K images), MNIST (70K), small SQuAD subset | | Single clear contribution | You focus on one thing, not 10 tricks | "We add dropout to this architecture" or "We change activation from ReLU to GELU" | | Well-known baseline | You understand it already; focus is on the delta | ResNet (you know it), LSTM (you know it), standard BERT (you know it) | | Recent with code on Papers with Code | Shows paper is reproducible; reference implementation exists if you get stuck | 2023–2024 papers with >5 linked implementations | | Paper has published numbers on multiple datasets | Shows generalization, not cherry-picked results | "We test on ImageNet, COCO, and 5 domain-specific datasets" |

Papers with Code is your toolkit. Go to paperswithcode.com, filter by your domain, sort by "Implementation Count." Pick a paper with 5+ implementations (shows it's reproducible) and download the official or top-ranked implementation to reference.

Time and Complexity Estimation

Estimate your budget realistically:

# Reproducibility Complexity Guide (illustrative estimates)

**Tier 1: ~8–12 hours (simple, clear papers)**
- CIFAR-10 image classification (paper: "Should You Go Deeper?")
- Fashion MNIST baseline comparisons
- Simple optimization experiments (e.g., "Adam vs SGD on MNIST")
- Single dataset, standard metric, no special tricks

**Tier 2: ~1–2 days (moderate complexity)**
- ImageNet classification paper with new architecture
- Text classification fine-tuning (e.g., BERT on SST-2)
- Standard supervised learning on 2–3 datasets
- 3–5 ablation studies

**Tier 3: ~3–5 days (complex but scoped)**
- Language model fine-tuning with multiple techniques
- Multi-domain evaluation (3+ datasets with different preprocessing)
- 10+ hyperparameter combinations to explore
- Requires distributed training or GPU access

**Tier 4: >1 week (too complex for learning)**
- Massive-scale experiments (ImageNet full training from scratch)
- Reinforcement learning with 100+ hyperparameters
- Multi-modal models with custom preprocessing
- Papers with 20+ ablations

Rule of thumb: Budget 3–10 hours per paper for solid reproducibility. More than 20 hours and you're sinking too much time into one paper.

Step 2: Build or Find a Minimal Dataset

You don't need the exact same dataset the paper used. You need a representative subset small enough to iterate on in <10 minutes per experiment.

Dataset Selection Strategy

If they used a public dataset (ImageNet, CIFAR, WikiText, SQuAD):

Take a subset (10–20% of full data) for your first attempt:

• ImageNet: Use 10K of 1.3M training images (same 1000 classes, but fewer examples per class)
• CIFAR-10: Use full 60K images (already small enough)
• WikiText: Use first 10M tokens instead of full 103M
• SQuAD: Use SQuAD-small (10% of full) if available, or sample 10K QA pairs

This cuts your iteration time from hours to minutes. Once you match the paper's ratio (e.g., they reported 5% accuracy drop on full data; you see 5% drop on 10% subset), you've validated the method. You can scale up if needed.

Why this works: A well-designed method should show similar patterns on smaller data. If your subset results diverge sharply from the paper's, it might indicate:

• Your preprocessing differs
• Random seed differences dominate (smaller data = higher variance)
• Hyperparameters need retuning for smaller data

If they used proprietary data:

Find a public alternative in the same domain:

# Domain-Specific Dataset Alternatives

**Paper Domain**: Customer churn prediction (proprietary company data)
**Alternative**: Kaggle Churn Dataset (10K customers, public)

**Paper Domain**: Healthcare diagnosis (internal hospital records)
**Alternative**: MIMIC-IV (public, de-identified ICU records)

**Paper Domain**: Recommendation system (Netflix-like internal data)
**Alternative**: MovieLens-1M (public, 1M user-movie ratings)

**Paper Domain**: Fraud detection (internal financial data)
**Alternative**: IEEE-CIS Fraud Detection (Kaggle, 500K transactions)

**Paper Domain**: Time-series forecasting (proprietary sensor data)
**Alternative**: UCI Energy dataset (household power consumption, public)

It won't be identical, but it tests generalization. If the method fails on a public dataset, you know it's either:

• Brittle and overfit to the proprietary data, or
• You didn't implement it correctly

If no alternative exists:

Synthesize a toy dataset:

• Paper proposes "anomaly detection on industrial sensor data"?
• Generate synthetic time-series with injected anomalies: linear trend with seasonal noise, then spike anomalies at known indices
• Train your model to detect spikes
• Won't match their exact numbers, but validates the core logic

# Synthetic anomaly dataset
import numpy as np

def generate_synthetic_timeseries(n_samples=1000, anomaly_fraction=0.1):
    # Normal data: sine wave + noise
    t = np.linspace(0, 10, n_samples)
    normal = np.sin(t) + np.random.normal(0, 0.1, n_samples)

    # Inject anomalies (sudden spikes)
    n_anomalies = int(n_samples * anomaly_fraction)
    anomaly_indices = np.random.choice(n_samples, n_anomalies, replace=False)
    labels = np.zeros(n_samples)
    labels[anomaly_indices] = 1
    normal[anomaly_indices] += np.random.uniform(2, 5, n_anomalies)  # spike

    return normal, labels

# Use this to train and validate your anomaly detector
X, y = generate_synthetic_timeseries()

Iteration Speed Requirement

The goal: Train-test-evaluate cycles should take <10 minutes.

If a single training run is 2 hours, you can't run 10 experiments to debug why you're off by 2% accuracy. You'll get frustrated and quit.

| Dataset Size | Time per Epoch | Typical Total Time | ✓ or ✗ | |--------------|----------------|-------------------|--------| | CIFAR-10 full (60K images) | 30 sec | 5–10 min (50 epochs) | ✓ | | ImageNet 10% (130K images) | 3 min | 20–30 min | ✗ | | ImageNet 1% (13K images) | 20 sec | 2–3 min | ✓ | | MNIST (70K) | 10 sec | 1–2 min | ✓ | | WikiText 10% (10M tokens) | 2 min | 15–20 min | ✗ (borderline) | | SQuAD 10% (10K QA pairs) | 5 sec | 1–2 min | ✓ |

If your dataset is slow, go smaller. The iteration speed matters more than data realism for learning purposes.

Step 3: Implement the Core Method Only

This is where most people overscope and fail. They try to replicate every ablation, every regularization trick, every hyperparameter sensitivity study. You will have 10 competing project ideas. Don't fall into this trap.

Implement the novel part only.

Example: Paper is "Efficient Transformers via Sparse Attention" (imagine circa 2023):

• ✓ Implement: The sparse attention mechanism
• ✓ Use: Pre-trained BERT or GPT-2 from HuggingFace (don't reimplement from scratch)
• ✗ Skip: Ablation study on 5 different sparsity patterns
• ✗ Skip: Fine-tuning tricks on 10 different datasets
• ✗ Skip: Custom CUDA kernels for 40% speedup

Your implementation should be 200–500 lines of code. If it's 2000 lines, you're adding unnecessary complexity.

What to Implement vs Skip

# Reproduction Scope: Do This / Skip That

**Paper Type**: Vision – "ResNet-Hybrid: Adding Attention to ResNets"

**DO:**
✓ Core architecture: ResNet backbone + attention layers
✓ Standard training loop (SGD, data augmentation, learning rate schedule)
✓ Baseline comparison: Standard ResNet on same data
✓ Main results: Accuracy on ImageNet-10% and CIFAR-10

**SKIP:**
✗ All 10 ablations (block placement, attention head count, etc.)
✗ Multi-GPU distributed training optimization
✗ Knowledge distillation and ensemble tricks
✗ Testing on 20 different datasets
✗ Hyperparameter sensitivity analysis (grid search over 50+ configurations)

---

**Paper Type**: NLP – "Better Fine-tuning: LoRA for LLMs"

**DO:**
✓ Core method: LoRA (low-rank adapters)
✓ Training: Standard supervised fine-tuning loop with LoRA
✓ Baseline: Fine-tune without LoRA on same task
✓ Results: Task accuracy on 2–3 standard benchmarks

**SKIP:**
✗ All 5 ablations on rank selection, layer placement
✗ Testing on 20 downstream tasks
✗ Instruction tuning variations
✗ Hardware optimization and inference speed
✗ Merging LoRA weights back into the model (engineering, not learning)

Minimal Implementation Checklist

For your core implementation, you need these pieces:

# Minimum Viable Reproduction Implementation

[ ] Load data (use pre-made dataset or subset)
[ ] Define baseline model (use off-the-shelf; don't reimplement from scratch)
[ ] Implement novel component (the 1–2 ideas from the paper)
[ ] Loss function (usually cross-entropy or MSE; use PyTorch default)
[ ] Training loop (forward pass, backward pass, optimizer step)
[ ] Evaluation metric (accuracy, F1, BLEU, whatever the paper uses)
[ ] Logging (print loss per epoch, track validation metric)

Lines of code target: 300–400 (not including data loading boilerplate)

Example skeleton:
```python
import torch
from torch import nn, optim
from torchvision import datasets, transforms

# 1. Load data
train_loader = torch.utils.data.DataLoader(
    datasets.CIFAR10(root='./data', train=True, download=True,
                     transform=transforms.ToTensor()),
    batch_size=128, shuffle=True
)

# 2. Baseline model (use pretrained)
model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True)

# 3. Novel component: add sparse attention layer
class SparseAttention(nn.Module):
    def __init__(self, dim, num_heads=8, block_size=64):
        super().__init__()
        self.dim = dim
        self.num_heads = num_heads
        self.block_size = block_size
        # ... implement sparse attention logic here

    def forward(self, x):
        # Compute block-diagonal attention instead of full attention
        # ... implementation
        return output

# 4. Insert sparse attention into model (3–5 lines)
# Modify model's forward to use SparseAttention

# 5. Training loop (10–15 lines)
optimizer = optim.SGD(model.parameters(), lr=0.01)
for epoch in range(50):
    for images, labels in train_loader:
        logits = model(images)
        loss = nn.functional.cross_entropy(logits, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f"Epoch {epoch}, Loss {loss:.4f}")

# 6. Evaluate
# Standard validation loop (10 lines)

This is your template. The paper's novel idea should fit into the "Novel component" section. If it takes >100 lines, you're probably overcomplicating it.

Step 4: Compare Against Reported Numbers and Debug Methodically

Now you train and evaluate. You'll likely see a gap. That gap is where learning happens.

Three Scenarios and How to Handle Each

Scenario 1: You match the paper's numbers (±1–2%).

Congratulations. You've validated the method works. Document:

• Exact hyperparameters you used
• Dataset size and split
• Random seeds (and their results)
• Training time and hardware used
• Any deviations from the paper's description

You can now safely try modifications (different hyperparameters, different datasets, extending the idea) with confidence. The method is debugged.

Scenario 2: You're off by 5–10%.

This is the most instructive case. The method probably works, but details matter. Investigate methodically using this checklist:

# Debugging Checklist: Why Results Don't Match (5–10% gap)

## Step 1: Check Hyperparameters (30 minutes)
□ Batch size: Paper says 128? You used 64? (Can shift results by 2–5%)
□ Learning rate: Paper says 0.01? Compare to 0.001 and 0.1 (lr is very sensitive)
□ Number of epochs: Did you train long enough? (Under-training looks like underperformance)
□ Optimizer: SGD vs Adam? Momentum value? Weight decay?
□ Scheduler: Did they use learning rate decay? If so, at what schedule?

Hypothesis: If you change ONE hyperparameter and get closer, you've found the gap.

## Step 2: Check Preprocessing (20 minutes)
□ Data normalization: Per-channel mean/std? Global normalization? No normalization?
□ Data augmentation: Which augmentations? (RandAugment? AutoAugment? Just crops and flips?)
□ Train/test split: Standard 80/20? Different seeds? Custom split?
□ Class imbalance: Did they weight classes? Under/oversample?

Hypothesis: Preprocessing accounts for ~30–50% of unexplained variance. Often the paper doesn't describe it fully.

## Step 3: Check Randomness (20 minutes)
□ Random seed: Did you set torch.manual_seed and np.random.seed?
□ Multiple runs: Run 3 times with different seeds. What's the variance?
□ Initialization: How are weights initialized? Xavier? He initialization?

Hypothesis: Different seeds can shift results by 0.5–2% on small datasets. If you ran once, you might be unlucky.

## Step 4: Check Implementation Details (30 minutes)
□ Dropout rates: If paper uses dropout=0.1, you must use 0.1, not 0.5
□ Activation functions: ReLU or GELU? This changes output distributions
□ Layer normalization: Batch norm or layer norm? When applied?
□ Gradient clipping: Max norm? Applied before or after optimizer step?

Hypothesis: These details are often mentioned in passing in the paper, not in the main text. Check the appendix.

## Step 5: Compare Against Reference Implementation (20 minutes)
□ If Papers with Code has an official or popular implementation, run it
□ Does their code match the paper's claimed numbers?
□ If yes: your implementation has a bug (or hyperparameter mismatch)
□ If no: the paper's numbers might be non-reproducible (known issue)

Hypothesis: If reference code also doesn't match, the paper might have reproducibility issues. Not your fault.

Empirical finding from reproducibility research: ~70% of the time, a 5–10% gap comes down to hyperparameters (especially learning rate) or preprocessing details not clearly described in the paper.

Scenario 3: You're off by >15% or the method fails completely.

Something is fundamentally different. Investigate:

1. Reread the method section carefully. You might have misunderstood the core algorithm. Print out what you're computing at each step and compare to the paper's formulas.

2. Check for data leakage. Is test data somehow in your training set? Did you use future information in your features?

3. Try their code. If the official code is available on Papers with Code and you run it on your dataset, does it match their reported numbers? If yes, your implementation differs. If no, the paper might have reproducibility issues.

4. Increase data size. Your 10% subset might be too small. Try 20% or 50% of the full dataset. Sometimes methods need more data to work well.

5. Hyperparameter grid search. Do a 2D grid: learning rate × batch size × epochs. Try:

   • Learning rates: [1e-4, 1e-3, 1e-2, 1e-1]
   • Batch sizes: [32, 64, 128, 256]
   • One quick pass (10 epochs each) to find the sweet spot

6. Check for bugs. Write unit tests for your novel component:

   def test_sparse_attention():
       x = torch.randn(2, 10, 64)  # batch=2, seq_len=10, dim=64
       layer = SparseAttention(dim=64)
       output = layer(x)
       assert output.shape == x.shape, "Output shape mismatch"
       # Test that attention weights sum to 1
       weights = layer.get_attention_weights(x)
       assert torch.allclose(weights.sum(dim=-1), torch.ones(...))
   

7. Verify against the paper's ablations. If the paper shows "removing component X drops accuracy by 5%," implement their ablation on your code. If you see a different drop, your implementation diverges from theirs.

After 2–3 hours of debugging: If you still can't match within 10%, consider the gap acceptable and document it. Move on to learning (implementation, understanding) rather than perfect replication.

Step 5: Document What You Learned

Write a one-page reproducibility report. This becomes part of your learning portfolio and helps future you (or someone else) reproduce the work faster.

# Reproducibility Report: [Paper Title]

## Core Contribution (3 bullets)
- [What's novel about this paper?]
- [How is it different from prior work?]
- [What's the key insight?]

## Implementation
**Language/Framework**: PyTorch / TensorFlow
**Lines of code for novel component**: 250 lines
**Time to implement**: 4 hours

## Experimental Setup
- **Dataset**: CIFAR-10 (60K training images, 10K test)
- **Baseline**: ResNet-18 (standard PyTorch)
- **Our method**: ResNet-18 + SparseAttention layer
- **Hyperparameters**: LR=0.01, batch_size=128, epochs=50, optimizer=SGD
- **Hardware**: 1× GPU (NVIDIA RTX 3090)
- **Training time**: 15 minutes total

## Results
| Model | Accuracy | Notes |
|-------|----------|-------|
| Paper (ResNet-18 baseline) | 95.2% | Reported on full ImageNet-1K |
| Paper (their method) | 96.1% | 0.9% improvement on ImageNet-1K |
| Our baseline (ResNet-18) | 92.3% | On CIFAR-10 (smaller task, lower absolute numbers expected) |
| Our method (+ SparseAttention) | 93.8% | 1.5% improvement on CIFAR-10 |

**Analysis**: We matched the paper's *improvement ratio* (1.5% on CIFAR-10 vs 0.9% on ImageNet-1K is reasonable because CIFAR-10 is smaller/easier; more room for improvement). Absolute accuracy differs because CIFAR-10 is an easier task than ImageNet-1K.

## Key Hyperparameter Sensitivity
| Setting | Accuracy | Notes |
|---------|----------|-------|
| LR=0.001 | 91.2% | Too low; underfitting |
| LR=0.01 | **93.8%** | Best; paper also uses this |
| LR=0.1 | 92.1% | Too high; oscillation |
| Batch size=64 | 93.5% | Slightly worse |
| Batch size=128 | **93.8%** | Best; paper uses this |
| Batch size=256 | 93.2% | Worse (less stable gradient) |

**Insight**: Learning rate is the most sensitive parameter (±2% per 10× change). Batch size less sensitive (±0.3%).

## Deviations from Paper and Why
1. **Why absolute accuracy is lower**: Paper tests on ImageNet-1K (more complex); we test on CIFAR-10 (simpler). CIFAR-10 baseline accuracy for ResNet-18 is ~94–95%; our 92.3% is within normal variance.
2. **Random seed**: We report mean ± std over 3 runs. Our method: 93.8 ± 0.2%. Paper reports single run. Variance is normal.
3. **Dataset size**: Paper trained on 1.3M images; we used 60K. Method still shows consistent improvement.

## What I Learned
- **Core insight**: Sparse attention reduces parameters without losing much accuracy, especially on smaller/medium datasets. On tiny datasets (<10K images), sparse attention performs worse because the sparse pattern is too rigid.
- **Implementation challenge**: Getting sparse attention to work on GPU requires careful masking; I initially didn't mask correctly and got wrong results.
- **Generalization**: The 0.9–1.5% improvement holds across dataset sizes, but the method degrades on very long sequences (>4K tokens for vision, >8K for text). Paper didn't test these cases.

## Gaps and Future Work
- Paper doesn't test on sequences >4K tokens; does sparse attention scale?
- Hyperparameter sensitivity to block size (we tested fixed 64; what about 32 or 128)?
- Inference speed: we measured training only; what about latency at inference?

## Reproducibility: Easy or Hard?
- **Difficulty**: Medium (3–5 hours for someone familiar with PyTorch)
- **Code availability**: Official code not available; had to implement from scratch
- **Documentation**: Paper is clear on method, but missing some hyperparameter details (found them in appendix)
- **Result reproducibility**: Got within 1–2% of their numbers after 3–4 hyperparameter tweaks

## Would I Recommend Reproducing This Paper?
**Yes, for learning **. It taught me:
1. How sparse patterns can reduce computation
2. Where hyperparameter tuning matters (more than I expected)
3. How to debug ML code systematically

**Not for rapid results**. If I just needed the method, I'd use their code/PyTorch implementation online.

This report becomes:

• Your learning log entry (captures what you learned)
• Evidence for interviews ("I reproduced 3 papers, here's the report")
• Reference for future projects ("Oh, I did sparse attention before; here's how I tuned it")

Common Pitfalls and How to Avoid Them

Pitfall 1: Trying to match numbers exactly (to 0.1%).

Papers round, skip details, have typos, and use different random seeds. Matching within 1–2% is success. Obsessing over 0.1% is wasted time. You learned the method; the small difference doesn't matter.

Pitfall 2: Assuming hyperparameters from the paper are optimal for your data.

They're optimal for their data, size, and hardware. Learning rate 1e-4 might be right for their 1M-image dataset and completely wrong for your 10K-image subset. Use their numbers as a starting point, then do a 2D grid search:

• Try ±1 order of magnitude on learning rate (if they use 1e-3, try 1e-4 and 1e-2)
• Try 2–3 batch sizes around their value
• One quick pass (10 epochs each) to narrow down

Pitfall 3: Not fixing your random seed.

Different random initialization can shift results by 0.5–3% depending on the method and dataset. Fix this:

torch.manual_seed(42)
np.random.seed(42)
torch.cuda.manual_seed_all(42)

Run at least 3 seeds (42, 123, 456) and report mean ± std dev. This is how professional papers should report results.

Pitfall 4: Rebuilding the wheel.

If PyTorch or TensorFlow has the component you need (ResNet, LSTM, attention block), use it. You're learning the paper's novel contribution, not reinventing standard neural network basics. Using off-the-shelf baselines saves time and reduces bugs.

# Good: Use official model
model = torchvision.models.resnet18(pretrained=True)

# Bad: Reimplement ResNet from scratch
class ResNet(nn.Module):  # 500 lines of code; easy to introduce bugs
    ...

Pitfall 5: Stopping at implementation.

The learning only happens when you investigate why your numbers differ. Timeline:

• Day 1: Implementation done, results don't match → frustration
• Days 2–3: Systematic debugging (hyperparameters, preprocessing, seeds) → deeper understanding
• Day 4: You understand why the gap exists → real learning achieved

If you match the paper on day 1, great—move to modifying the method. If you don't match by day 4, document the gap and move on (don't sink 40 hours into perfect replication).

Pitfall 6: Not documenting preprocessing.

Preprocessing is often the hidden variable. If the paper doesn't describe it clearly, you'll spend hours debugging. Before you train, document:

# Exact data pipeline
transforms.Compose([
    transforms.RandomCrop(32, padding=4),        # Specific padding amount
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465),  # These exact values
                         (0.2023, 0.1994, 0.2010))
])

Save this to a file. When debugging, you'll know exactly what you're using.

Pitfall 7: Testing on full-size data from day 1.

Training on full ImageNet takes 8 hours per experiment. You'll do 2 experiments max before giving up. Start small (10% of data), verify the method works, then scale up. Example schedule:

• Day 1: CIFAR-10 subset (10K images) — train in 2 minutes, debug fast
• Day 2: CIFAR-10 full (60K images) — train in 15 minutes
• Day 3: ImageNet 10% (130K images) — train in 45 minutes
• Only Day 4+ do you touch full ImageNet if needed

This exponential scaling prevents wasted time.

A Worked Example: Reproducing LoRA (Low-Rank Adaptation) Fine-Tuning

Scenario: You find the LoRA paper and want to understand it by reproducing the core result: "Fine-tune a model with 1M trainable parameters instead of 7B, matching 99% of accuracy."

Plan:

• Base model: HuggingFace BERT (fine-tune on SST-2 sentiment classification)
• Dataset: Full SST-2 (67K training examples; usually takes 15 min to train)
• Baseline: Standard fine-tuning (update all 110M BERT parameters)
• Novel method: LoRA (update only 1M low-rank parameters)
• Time estimate: 8–10 hours total

Day 1: Setup Baseline (3 hours)

What you do:

# Download BERT and SST-2
python -c "
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModelForSequenceClassification.from_pretrained(
    'bert-base-uncased', num_labels=2
)
# Train on SST-2, save results
"

Results you get:

• Train baseline BERT (fine-tune, all parameters)
• Run for 3 epochs on SST-2
• Validation accuracy: 92.3% ± 0.2% (3 random seeds)
• Parameters trained: 110M
• Training time: 45 min
• Memory: 5.8 GB

Document:

Baseline:
  Model: BERT-base-uncased
  Task: SST-2 (sentiment, 2 classes)
  Accuracy: 92.3 ± 0.2% (mean ± std over 3 seeds)
  Parameters trained: 110M
  Training time: 45 minutes
  Hyperparameters:
    learning_rate: 2e-5
    batch_size: 32
    epochs: 3
    optimizer: AdamW

Day 2: Implement LoRA (4 hours)

What you do: Implement LoRA: replace weight matrices with low-rank factorization.

import torch
from torch import nn

class LoRALayer(nn.Module):
    def __init__(self, in_features, out_features, rank=8, alpha=16):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.rank = rank
        self.alpha = alpha

        # Low-rank matrices
        self.A = nn.Parameter(torch.randn(in_features, rank) * 0.01)
        self.B = nn.Parameter(torch.zeros(rank, out_features))

    def forward(self, x):
        # Compute LoRA update: x @ A @ B
        return (x @ self.A @ self.B) * (self.alpha / self.rank)

# Insert into BERT: replace linear layers
model.bert.encoder.layer[0].attention.self.query = LoRALayer(768, 768)
# ... do this for query, value layers in each attention head

Results you get:

• Parameters to train: ~1.2M (instead of 110M)
• Training time: 35 min (faster; smaller weight matrices)
• Validation accuracy: 91.8% ± 0.3% (after 3 epochs)
• Gap from baseline: −0.5%

Analysis:

• LoRA is 1.5% faster to train
• Accuracy is slightly lower (91.8% vs 92.3%)
• Is this because rank=8 is too low? Or do we need to tune the learning rate for LoRA specifically?

Day 2 evening: Debug (2 hours)

Hypothesis 1: Learning rate is wrong for LoRA

# Try 3 learning rates
for lr in [1e-5, 2e-5, 5e-5]:
    train_lora_model(lr=lr)
    # Results:
    # lr=1e-5: 91.1% (too low, underfitting)
    # lr=2e-5: 91.8% (same as before, not the issue)
    # lr=5e-5: 92.1% (better! still 0.2% below baseline)

Hypothesis 2: Rank is too low

# Try different ranks
for rank in [4, 8, 16, 32]:
    # Results:
    # rank=4:  90.5%
    # rank=8:  91.8%
    # rank=16: 92.2% (matches baseline!)
    # rank=32: 92.3% (matches baseline, but 2.5M parameters)

Finding: Rank=16 is sweet spot; matches baseline accuracy with 2.5M parameters (2% of 110M).

Day 3: Final Results and Documentation (2 hours)

Results table:

| Method | Accuracy | Parameters | Training Time | Memory | |--------|----------|-----------|----------------|--------| | Standard fine-tuning (baseline) | 92.3 ± 0.2% | 110M | 45 min | 5.8 GB | | LoRA (rank=8) | 91.8 ± 0.3% | 1.2M | 35 min | 3.2 GB | | LoRA (rank=16) | 92.2 ± 0.2% | 2.5M | 38 min | 3.8 GB | | LoRA (rank=32) | 92.3 ± 0.2% | 5.0M | 40 min | 4.2 GB | | Paper's claim | "99% of full" (91.4% if full is 92.3%) | 1–2% (1.1–2.2M) | ~40 min | ~3–4 GB | | Our result | Matched (92.2 vs 92.3) | Rank=16: 2.5M ✓ | Matched ✓ | Matched ✓ |

Key learnings:

1. Rank matters enormously — doubling rank from 8 to 16 adds 0.4% accuracy
2. Learning rate needs tuning — didn't need to change, but could have
3. Generalization: Method works as promised; "99% accuracy" means matching baseline, not 99 out of 100 times
4. Trade-offs: You can tune rank to your needs:
   • rank=8 for extreme memory constraints (90% of accuracy, 1% of parameters)
   • rank=16 for typical use (99% accuracy, 2% of parameters)
   • rank=32+ if you want full accuracy at higher parameter cost

Why Reproduction Teaches You More Than Code Review

Reading code teaches you syntax and patterns. Reproducing teaches you thinking.

When you reproduce, you make dozens of micro-decisions that force understanding:

| Question | Reproduction Path | Code Review Path | |----------|------------------|------------------| | Should I normalize before or after train/test split? | You try both, see one way breaks (data leakage), internalize the lesson | You read code and see the "correct" way, but don't feel the cost of the mistake | | What batch size is right? | You try 32, 64, 128; see that larger is faster but less stable; understand the tradeoff | You see they use 128; you copy it | | When to stop training? | You try early stopping; see how it prevents overfitting; tune threshold empirically | You see the early stopping condition in code; might not understand why it's set to 0.001 |

A code review shows you the answer. Reproduction forces you to ask the question first, struggle, fail, debug, then find the answer. The struggle is the learning.

After you've reproduced a paper, you understand:

• Why the paper's design choices matter (not just "they did it")
• Which choices are core to the method vs which are engineering details
• How sensitive the method is to hyperparameters (1% or 10%?)
• What would break if you changed something (will it still work on different data?)
• How to debug when things don't work (systematic approach, not random guessing)

This understanding doesn't come from reading or reviewing code. It comes from building it yourself, watching it fail, debugging with data in hand.

Common Mistakes

Mistake 1: Treating reproduction as pass/fail.

You think: "Either I match the paper exactly or I've failed." In reality, understanding why your numbers differ is often more valuable than matching exactly.

If you get 87% and the paper gets 91%, and you trace the 4% gap to:

• "Different random seed and their initialization was luckier" → you learned about variance
• "Their preprocessing clips outliers and I didn't" → you learned about data-specific engineering
• "They tuned hyperparameters and I used defaults" → you learned about hyperparameter sensitivity

The gap is data. You're debugging in real time. That's learning.

Mistake 2: Abandoning when you hit a mismatch.

The temptation: "This paper's results can't be right. Or my environment is broken." Most of the time, it's neither. It's a detail:

• Preprocessing step you missed in the appendix
• Hyperparameter mentioned in passing (learning rate decay schedule in Figure 2, not the text)
• Data split that's slightly different
• Random seed not fixed

These details take 30–90 minutes to track down. Invest that time. That's where learning happens.

Mistake 3: Reproducing only methods you're excited about.

Temptation: "Let me reproduce GPT-3 fine-tuning!" Reality: Complex, many moving parts, hard to debug.

Better: Reproduce simple, well-understood methods where you can iterate quickly. A simple method (Mixup, dropout, data augmentation) with a small dataset (CIFAR-10) teaches you more than a complex method with a big dataset because:

• You can iterate 5 experiments per hour, not 5 per day
• You'll hit more edge cases and debug more scenarios
• The learning is in the debugging process, not the method novelty

Simple paper reproduced deeply > Complex paper reproduced shallowly.

Mistake 4: Not documenting as you go.

You reproduce code, results don't match, you debug, you fix it. A week later, you don't remember what you tried or why it worked. Document:

• Hyperparameters you tried and results (even failed attempts)
• Bugs you found and how you fixed them
• Your hypothesis and findings (e.g., "Thought LR was wrong, wasn't. Preprocessing was the issue.")

This documentation is your artifact. It's proof you can debug and reason through problems.