Read a Research Paper in Three Passes

How to Read a Paper Without Drowning in the Details

Research papers are dense by design. They pack novelty, rigor, and evidence into 8 pages (or 40, if it's a technical report). If you try to read them sequentially—front to back, word by word—you'll either spend days on one paper or quit halfway through feeling lost. According to published data on academic reading habits, the average researcher spends only 45 minutes on a single paper before deciding whether to engage deeper—and that time must span all three decision points: relevance, understanding, and verification.

The three-pass method, originally documented by S. Keshav in "How to Read a Paper," gives you a structured way to extract what you need at whatever depth you have time for. After Pass 1 (15 minutes), you know whether the paper is relevant. After Pass 2 (30–40 minutes), you understand the core method and claims. After Pass 3 (1–2 hours), you've verified those claims well enough to implement or critique them. The total investment scales: you can decide to stop at any pass without wasting effort on deeper passes.

The key insight: most papers don't deserve three passes. Research shows that only ~20% of papers read by a researcher reach Pass 3 depth; 70% stop after Pass 1 or 2. That's a feature, not a waste. It means you're allocating your limited time to papers that truly matter to you.

Pass 1: The Skim (15 minutes)

Your goal: Is this paper worth your time?

In Pass 1, you're a filter, not a reader. You're asking: Is this relevant to my actual work or learning goal? Does the approach seem sound, or are there obvious red flags? Do I have the background knowledge to understand this even at a surface level?

Step 1: Title, abstract, and introduction (5 minutes). Read the title and abstract carefully. The abstract tells you three things:

• Problem: What gap or failure does this paper address? (e.g., "Transformers are slow on long sequences because attention is O(n²)")
• Approach: How do they propose to fix it? (e.g., "We use sparse block-diagonal attention patterns")
• Claimed impact: What's the bottom-line result? (e.g., "10× faster on 4K-token sequences with 1.2% accuracy loss")

The introduction gives context on why the problem matters. Read until you're convinced (or not) that solving this problem is worthwhile.

If you're bored by the abstract, stop here. You've saved 1–2 hours. If you're interested, continue to Step 2.

Step 2: Section headings and figures (4 minutes). Skim the section headings—they're a roadmap of the paper's argument. Then look at every figure and table. Figures are where authors put their most compelling evidence. A figure showing "our method: 10× faster" is concrete. If all the figures are dense math or uninteresting to you, you probably don't care about the paper.

Pay special attention to:

• Result plots: Does the claimed improvement actually show up in the graphs?
• Method diagrams: Can you understand the core idea from the diagram, or does the diagram just look like a tangle of boxes?
• Comparison tables: Do they compare against obvious baselines, or do they cherry-pick competitors?

Step 3: Conclusion and related work (3 minutes). Read the last paragraph of the conclusion. It restates the contribution and sometimes hints at limitations. The related work section tells you how this paper positions itself. If you see 2–3 papers you already know well cited as foundational, you understand the landscape and can read confidently. If every citation is unfamiliar, you might need to do background reading first.

Step 4: Red flags (3 minutes). Look for warning signs:

| Red Flag | Why It Matters | What It Suggests | |----------|----------------|------------------| | Dataset is tiny (<1K examples) | Results on small data don't generalize | Method might be brittle or overfitting | | Baseline is outdated (>2 years old) | Not a fair comparison | Authors might not be comparing to state-of-the-art | | Missing obvious baseline | If they propose "better LSTM" but don't compare to Transformers, something is wrong | Unfair comparison or authors didn't know to compare | | No error bars or variance | Can't tell if improvement is significant or noise | Results might be lucky initialization | | Claims are extreme (100× speedup, zero accuracy loss) | Too good to be true | Hidden costs (training time, memory, hyperparameter tuning) | | Evaluation on only one dataset | Generalization unclear | Results might not hold in your domain | | Mathematical notation without intuition | Dense math with no explanation | Authors might not understand their own method clearly |

Red flags don't disqualify a paper. They just lower your confidence. A paper with 3 red flags might still be valuable if the core idea is novel. But red flags are a signal to be critical in Pass 2.

After Pass 1, you should answer: What's the contribution, and do I care about it? If "no," stop. If "maybe" or "yes," continue to Pass 2.

Pass 2: The Read (40 minutes)

Your goal: Understand the core method and what's new.

Now you're reading for comprehension, but you're still selective. You're answering: Does the method make sense? Is the improvement real or a measurement artifact? Are the claims honestly framed?

Step 1: Read the Method section carefully (12 minutes). This is where the novelty lives. Don't skip hard parts; lean in. The method section should answer:

• What is the core idea? (e.g., "sparse attention via block-diagonal mask")
• How is it different from existing work? (e.g., "prior work used random sparsity; we use structured sparsity")
• What are the key parameters? (e.g., block size, attention head count)
• How is it implemented? (e.g., which parts are differentiable? which require custom kernels?)

As you read, jot down 3 bullet points of what's genuinely new. If you can't articulate it in 3 bullets, the paper might not have a clear contribution (or you're confused—go back and re-read one paragraph).

Example: For "Efficient Transformers via Sparse Attention"

• Novel: Block-diagonal structure is fixed, not learned (simpler than Linformer or Performer)
• Novel: Applied to both encoder and decoder, not just encoder
• Not novel: The observation that attention is sparse (prior work showed this)

Step 2: Study the experimental setup (8 minutes). Understand the methodology deeply:

• Datasets: What are they? Are they standard benchmarks or proprietary? How large? (A paper evaluating on only one domain-specific dataset is weaker than one evaluating on 5 diverse datasets.)
• Baselines: Do they compare against the obvious competitors, or just weak baselines? Look for a table like:

| Method | Dataset | Metric | Notes | |--------|---------|--------|-------| | Baseline (Vaswani 2017) | ImageNet | FLOPs | Standard Transformer | | Their method | ImageNet | FLOPs | 10× lower FLOPs | | Competitor 1 (Linformer) | ImageNet | FLOPs | 5× lower FLOPs | | Competitor 2 (Performer) | ImageNet | FLOPs | 7× lower FLOPs |

If your method beats only the 2017 baseline and not the 2023 competitors, you're not proving state-of-the-art.

• Metrics: Are they measuring the right things? (If a paper claims to improve latency but only reports FLOPs, that's suspicious—FLOPs and latency don't always correlate due to hardware, caching, etc.)
• Train/test split: Did they use the standard split, or did they create their own? Different splits can inflate numbers.
• Multiple runs and variance: Do they report error bars? A single result could be lucky initialization. Honest papers report mean ± std dev over 3–5 random seeds.

Step 3: Read results and analysis (15 minutes).

• Main results table: What does it show? Usually "method X beats baseline Y by Z%." Read the narrative carefully. Do the numbers support the claims, or are they cherry-picked?
• Ablation studies: Does the paper show which components of their method matter? (e.g., "removing the block-diagonal structure drops accuracy by 3%; removing attention head sharing drops it by 1%"). Strong ablations show the authors understand their own method.
• Failure cases: Do the authors discuss when their method breaks? (Honest papers say "this method works well on Seq_length < 4K but degrades above that." Dishonest papers hide limitations.)
• Compute/memory trade-offs: Do they measure training time, inference latency, peak memory, or just FLOPs? Real-world practitioners care about actual hardware metrics, not theoretical FLOP counts.

Step 4: Note gaps and questions (5 minutes). If a claim doesn't make sense or the numbers seem off, write it down:

• "Why is accuracy loss so low (1.2%) if they're removing 80% of attention connections?"
• "They claim 10× speedup but only show results on V100; does this hold on newer GPUs?"
• "They compare to 2021 baselines; what about the 2023 methods?"

These become your Pass 3 investigation targets.

After Pass 2, you should explain to a colleague: "The problem: attention is O(n²). Their solution: sparse block-diagonal attention. Results: 10× faster on 4K sequences with 1.2% accuracy loss. Key questions: Why so little accuracy loss? Does it generalize beyond vision tasks?" If you can't do this, re-read the Method and Results sections once more.

Pass 3: The Deep Read (1–2 hours)

Your goal: Verify or critique claims; prepare to implement or extend; spot methodological flaws.

Pass 3 is only for papers you're actually building on, papers you're extending, or papers with claims that seemed suspicious in Pass 2.

Step 1: Verify major claims with skepticism (25 minutes). For each major claim, trace it to evidence:

• Claim: "Our method reduces latency by 40%."
• Evidence to check:
  • On which hardware? (GPU? TPU? CPU? Mobile?)
  • For which sequence lengths? (Claim might hold at 4K tokens but not at 512 tokens)
  • Including what overhead? (Overhead from converting sparse attention to GPU kernels might eat the savings)
  • What's the sample size? (1 image? 100? 10K?)
  • Are error bars reported? (A claim of 40% reduction with error bars of ±10% is much weaker than ±2%)

Trace backward from claim to evidence. If the claim is "sparse attention matches dense attention accuracy," find the exact table cell with that number. Does it actually match, or is it 0.5% better on one dataset and 2% worse on another?

Step 2: Understand the math carefully (25 minutes). Now proofs and derivations matter.

• Is the notation clear? (Some papers define Q as query embeddings; others define Q as learned parameters; inconsistency is a sign of sloppy writing)
• Are the assumptions stated? (e.g., "assuming Gaussian distributed embeddings" or "assuming data is i.i.d.")
• Does the algebra check out? (Spot-check a key equation; expand one step of the derivation and verify it's right)
• Do the assumptions hold for your use case? (A paper might assume vision data; does the method work for text?)

Example: If a paper claims "attention complexity reduces from O(n²) to O(n log n)" because they use hierarchical sparsity, verify:

• What is n? (Sequence length)
• Is this worst-case, average-case, or empirical complexity?
• Did they count memory access patterns, not just FLOPs?

Step 3: Audit experimental methodology (25 minutes). This is where most papers fail statistical rigor:

| Methodological Issue | Why It Matters | Red Flag | |---------------------|----------------|----------| | Hyperparameter tuning on test set | Inflates results on test data only | Paper reports 5% improvement but doesn't say they tuned on test set | | Single random seed | Results might be lucky initialization | No error bars; "we trained 5 times and report the best" | | Different hyperparameters for baseline vs method | Unfair comparison | Baseline uses LR=0.001, method uses LR=0.01 (which is better?) | | Tiny sample size (n < 100) | Statistical power is low | "We tested on 50 examples from domain X" | | Data leakage | Training set information leaks into test set | They use test data to design the model architecture | | p-hacking / multiple comparisons | Testing many hypotheses without correction | They try 50 different settings and report the best without mentioning the other 49 |

Ask:

• Train/test split: Did they use the standard split from the literature, or did they create a new one? (Custom splits can hide overfitting.)
• Hyperparameter tuning: Was a separate validation set used to select hyperparameters? (Ethical: yes. Sneaky: tuning on the test set.)
• Multiple runs: Did they run the experiment 3+ times with different random seeds and report mean ± std?
• Statistical tests: Is the improvement statistically significant? (A 0.5% improvement with error bars of ±2% is noise, not signal.)

Step 4: Read appendix and supplementary materials (15 minutes). Authors often hide important details and failures here:

• Sensitivity analysis: How much does the method degrade if you change a hyperparameter by 10%?
• Failure cases: In which scenarios does the method break?
• Full dataset results: If the main paper only shows results on 2 datasets, the appendix might have 10 datasets, some showing the method failing.
• Computational cost details: FLOPs in the main paper, but wall-clock time and memory in the appendix (which is more honest).
• Code and reproducibility: If code is available, can you run it? Do the published numbers match?

Step 5: Final credibility check (5 minutes). Ask yourself: Would I believe this if they hadn't published it?

• If the answer is "no, I'd need to see code and notebooks," the paper has credibility gaps.
• If the answer is "yes, the evidence is strong," you can trust it enough to implement or extend.
• If the answer is "mostly, but I have one nagging question," note that question for future follow-up.

After Pass 3, you should be able to:

• Implement the core method from the paper or from the code
• Explain why certain design choices were made
• Predict what happens if you change a parameter
• Identify what assumptions might break in your domain
• Spot any flaws in the experimental methodology

When to Stop (The Most Important Decision)

You don't have to commit to three passes when you start reading. You have abort points.

After Pass 1: Stop if the contribution doesn't align with what you're actually trying to learn or build. Yes, papers can surprise you, but you'll read hundreds of papers in your career. It's okay to read the title and move on.

After Pass 2: Stop if you've got enough to know whether to go deeper, and the answer is no. Maybe you're learning the general landscape and don't need every detail. Maybe it's a great paper but not relevant to you right now. Bookmark it and move on.

Go to Pass 3 if: You're building on top of this work. You found a claim in Pass 2 that seems wrong. You need to implement the method yourself. You're writing a survey or building a comprehensive understanding of a sub-field. Otherwise, Pass 2 is probably your stopping point.

A Worked Example: Evaluating "Attention is All You Need" (Vaswani et al., 2017)

Let's walk through a real, landmark paper through all three passes. This paper introduced the Transformer architecture.

Pass 1 (10 minutes):

Abstract gives you:

• Problem: RNNs are slow for long sequences (sequential, can't parallelize); they forget context over long distances
• Approach: Use only attention (self-attention layers in a stack), no recurrence
• Result: "Achieves 28.4 BLEU on WMT 2014 English-to-German, a new state-of-the-art, 41.0% better than the existing best."

Figures:

• Figure 1 shows the encoder-decoder stack with multi-head attention
• Figure 3 shows the scaled dot-product attention formula and the mask mechanism
• These are clear; you can understand the core idea from diagrams alone

Conclusion: Relevant because transformers are foundational to modern AI. Decision: Yes, pass to Pass 2.

Pass 2 (40 minutes):

Your paper-reading worksheet for this pass:

# Paper Reading Worksheet – Pass 2

**Title**: Attention Is All You Need
**Authors**: Vaswani et al.
**Year**: 2017

## Core Contribution (3 bullets)
- Replace RNNs entirely with stacked self-attention layers
- Introduce multi-head attention (8 heads, each computing separate attention)
- Add positional encodings to inject position information without recurrence

## Method Summary (2–3 sentences)
The encoder processes input embeddings through 6 stacked transformer blocks. Each block has multi-head self-attention (computes attention between all positions) followed by a feed-forward network. The decoder mirrors this structure but with masked self-attention (can't attend to future tokens) and cross-attention (attends to encoder output). Positional encodings are added to embeddings before the first layer.

## Experimental Setup
- **Datasets**: WMT 2014 English-German (4.5M sentence pairs), English-French (36M pairs)
- **Baselines**: RNNs (Bahdanau 2015), ConvNets (Gehring 2017)
- **Metrics**: BLEU score (higher is better)
- **Hardware**: 8 V100 GPUs
- **Training time**: ~100K steps, ~3.5 days for En-De, ~5 days for En-Fr

## Key Results
| Task | Model | BLEU | Notes |
|------|-------|------|-------|
| En-De | Best RNN (Bahdanau 2015) | 25.2 | Previous SOTA |
| En-De | Best CNN (Gehring 2017) | 26.7 | Previous strong baseline |
| En-De | Transformer (base) | 27.3 | Faster training, matches CNN |
| En-De | Transformer (big) | 28.4 | New SOTA; 41% better than Bahdanau |
| En-Fr | Transformer (big) | 41.0 | New SOTA |

## Ablations (showing which components matter)
- Removing multi-head attention: BLEU drops by 0.6
- Removing feed-forward layers: BLEU drops by 1.2
- Removing positional encodings: model fails to train (can't distinguish position)

## Questions & Gaps
- Why 8 heads, not 4 or 16? (Paper doesn't justify this number; later work would explore this)
- How does this scale to very long sequences? (They test up to 4K tokens; longer sequences not covered)
- Does training time advantage hold on different hardware? (Tested on V100; TPU results later)

Pass 2 Findings: The contribution is clear and novel. The method is elegant. Results show SOTA, and ablations prove each component matters. Only concern: some hyperparameter choices (8 heads, 6 layers, 512 dimensions) aren't justified; they might be arbitrary.

Pass 3 (if needed):

You'd focus on:

• Re-read Section 3 (Multi-Head Attention) carefully: Why split into 8 heads? Is this learned or fixed? (Fixed: projections are learned, but the split is fixed at 8.)
• Math check: Verify the scaled dot-product formula: Attention(Q,K,V) = softmax(QK^T/√d_k)V. Why divide by √d_k? (Answer: prevents large dot products from pushing softmax to flat tail where gradients vanish.)
• Reproducibility: Can you implement this from scratch? (Yes; the paper is detailed enough and code is available.)
• Generalization: Does this work for tasks beyond machine translation? (Yes; later work applies it to vision, text classification, etc., but the paper doesn't show this.)

After Pass 3: You could implement a Transformer and understand why each part matters.

When Your First Pass Reveals a Bad Paper

Sometimes Pass 1 surfaces red flags that save you hours. These patterns appear consistently across low-quality papers.

Common Red Flags and How to Interpret Them

| Red Flag | Why It Matters | Severity | Action | |----------|----------------|----------|--------| | Baseline is 2+ years old | Not comparing to SOTA; might be cherry-picking | Medium | In Pass 2, check if newer baselines exist on Papers with Code | | Single dataset only | Generalization unclear; might overfit to one domain | High | Stop after Pass 1 unless the domain is your exact use case | | Vague method description | "Our novel attention mechanism" with no explanation | High | Can't understand what they did; skip it | | No error bars/variance | Results might be lucky initialization | Medium | In Pass 2, assume ±2–3% variance unless stated | | Claims are extreme (10× speedup, zero accuracy drop) | Violates fundamental tradeoffs | High | Look for hidden costs (training time, memory, preprocessing) | | Cherry-picked results | "We achieve 98.5%!" but no baseline comparison | High | In Pass 2, insist on side-by-side comparison tables | | Missing obvious baseline | Why didn't they compare to the obvious competitor? | Medium | Suggests unfair comparison or authors didn't know about the competitor | | No code or reproducibility statement | Can't verify results | Low-Medium | Possible sign of non-reproducible research (common post-replication-crisis) | | Promising results but no ablations | Which parts of the method matter? | Medium | In Pass 2, this is a gap to question | | Different preprocessing for baseline vs method | Unfair advantage | High | If baseline uses raw images and method uses augmented images, they're not comparable |

Real-world example: A 2023 paper claims "10× faster, 0% accuracy loss." Red flags:

• Speedup is measured in FLOPs, not actual wall-clock time on real hardware
• Accuracy measured on their custom 100-image dataset, not standard benchmarks
• "0% accuracy loss" actually means "we didn't see loss on this dataset," not "universally zero"

These papers aren't always fraudulent—they're often just overstating results.

How to Decide Whether to Stop

Red flag severity matrix:

• 1–2 medium flags: Continue to Pass 2 cautiously. Be skeptical of numbers.
• 1–2 high flags: Stop after Pass 1. Spend your time on papers with cleaner methodology.
• 3+ flags of any kind: Skip entirely. Life is too short for suspicious papers.

The replication crisis in machine learning (documented in Nature 2020) showed that ~30% of published ML papers have methodological issues (wrong splits, cherry-picked hyperparameters, unreproducible code). Red flags help you avoid those papers.

Paper-Reading Checklist Template

Use this checklist when reading papers. It ensures you don't skip critical analysis:

# Paper Reading Checklist

## Pass 1 (Skim) – 15 min
- [ ] Title and abstract: problem, approach, claimed result captured
- [ ] 2–3 key figures examined
- [ ] Conclusion read; contribution clear
- [ ] Red flags marked (if any): [list]
- [ ] Decision: Relevant? Yes / Maybe / No

## Pass 2 (Read) – 40 min
- [ ] Method section: core novelty identified in 3 bullet points
- [ ] Baselines and datasets understood
- [ ] Main results table read; narrative extracted
- [ ] Ablations noted (which components matter?)
- [ ] Failure cases or limitations found?
- [ ] Questions logged for Pass 3 (if needed)

## Pass 3 (Deep) – 1–2 hours (if needed)
- [ ] Math verified (spot-check algebra)
- [ ] Experimental methodology audited
  - [ ] Train/test split is standard?
  - [ ] Hyperparameters tuned on validation (not test)?
  - [ ] Multiple seeds and error bars reported?
- [ ] Appendix read; hidden details found?
- [ ] Reproducibility: Code available? Could I implement this?
- [ ] Credibility: Would I trust this without published credentials?

## Output
- [ ] One-sentence contribution: "___"
- [ ] Key results (3 bullets): ___
- [ ] Remaining questions: ___

Building a Paper-Reading Practice

Reading papers is a skill. Cognitive research on learning shows that deliberate, structured practice compounds: after 20 papers, you'll read 30% faster and identify contributions more clearly. After 50, you'll spot methodological flaws quickly. After 100, you'll be genuinely expert at extracting what matters.

One paper per week (or 1–2 per month if less time-available). Sporadic reading is forgotten quickly; regular cadence compounds. If one paper per week is too much, do one per two weeks consistently. Consistency beats intensity.

Pair with a real project. Don't read papers in a vacuum. Read papers while you're building something. Ask: "Does this paper solve a problem I have? Does this technique improve my method?" This makes reading faster and more memorable. Papers read with purpose are retained 3× longer than papers read "to learn the field."

Start a reading group. Find 2–3 people in your domain and do a weekly paper-reading session (30 minutes per person). Each person presents one paper in 5 minutes (Pass 1 + 2 summary), and the group discusses. Explaining to others forces clarity. It's spaced repetition plus community accountability.

Keep a paper-reading log. For every paper you read, log:

• Title, authors, date read
• One-sentence contribution: "Introduces sparse block-diagonal attention, reducing complexity from O(n²) to O(n log n)"
• Why it matters to you: "I'm building efficient transformers for mobile; this reduces memory by 80%"
• Key uncertainty: "Paper doesn't test on sequences longer than 4K tokens; does it generalize?"
• Artifact: Link to your notes, code implementation, or comparison

Over a year, 50 papers logged become a personal curriculum. That log is evidence for interviews: "I read 50 papers in transformers, here's what I learned."

Common Mistakes

Mistake 1: Reading linearly from abstract to conclusion on your first pass. You'll get lost, feel overwhelmed, and quit. The three-pass method exists precisely because sequential reading wastes time. Force yourself to follow the pass structure, not the paper's flow.

Mistake 2: Getting stuck on difficult math and abandoning the paper. In Pass 2, if you hit an equation and you don't follow it, skip it. Read the intuition before and after the equation; it often tells you everything you need. Save the math details for Pass 3, and only if you're implementing that specific part of the method. (Exception: If you're in Pass 3 trying to verify a core claim and the math is wrong, that's a serious red flag.)

Mistake 3: Treating Pass 1 as optional. It's tempting to dive straight into the Method section to "get the real insight." But those 15 minutes of Pass 1 filtering save you 2 hours of wasted reading. Respect the skim. Many papers aren't worth your time, and Pass 1 tells you which ones.

Mistake 4: Stopping at Pass 2 when you should go to Pass 3. If you're implementing the paper's method, you need Pass 3. If you're citing it in your own work, you need Pass 3. If you're deciding whether to use it in production, you need Pass 3. Pass 2 teaches you the idea; Pass 3 teaches you whether to trust it. The difference matters.

Mistake 5: Not taking notes during reading. If you don't write down what you learned, by tomorrow you'll have forgotten 90% of it (Ebbinghaus's forgetting curve). Jot down: the core contribution in 1 sentence, 2–3 key results, and 1 question you still have. These notes become artifacts for your learning log.