Use Spaced Repetition to Retain Technical Knowledge

The Forgetting Curve and Why Spacing Matters

You learn something on Monday. By Friday, you've forgotten ~60% of it. By next Monday, you've forgotten ~80%. This isn't laziness; it's how human memory works. It's Ebbinghaus's forgetting curve, mathematically modeled as R = e^(-t/S), where:

• R = retention percentage (what you remember)
• t = time since learning
• S = strength of memory (how well you learned it initially)

Ebbinghaus documented this in the 1880s. Modern research (2020 meta-analysis in Psychological Bulletin) confirms: unreviewed learning decays exponentially.

But here's the leverage: if you review on Thursday (day 4, before 80% loss), your retention curve resets. The curve becomes less steep; you forget more slowly. After the 2nd review, you forget even more slowly. After 3–4 reviews at optimal spacing, the material shifts from short-term to long-term memory. You retain 90%+ for months.

This is spaced repetition. It's not magic; it's a scheduling algorithm. Instead of cramming (memorize everything once, lose it after exams), you schedule reviews at increasing intervals:

• Review 1: Day 1 (immediately after learning)
• Review 2: Day 3 (just before you'd forget)
• Review 3: Day 10 (retention curve shallower now)
• Review 4: Day 30 (even shallower)
• Review 5+: Day 60+, 90+, etc.

| Timing | Retention Without Review | Retention With Spaced Review | Time Investment | |--------|--------------------------|------------------------------|-----------------| | Day 1 | ~100% (fresh) | ~100% (fresh) | 5 min to learn | | Day 3 | ~40% | ~95% (reviewed) | 2 min review | | Day 10 | ~10% | ~90% (reviewed) | 2 min review | | Day 30 | ~2% | ~85% (reviewed) | 2 min review | | Day 60 | <1% | ~80% (reviewed) | 2 min review |

Total time for durable retention: ~5 min learning + 8 min reviewing = 13 minutes. Total time without spacing: ~5 min learning + 120 min cramming + 30 min re-cramming = 155 minutes. Spaced repetition saves 90% of time while improving retention.

Most people encounter spaced repetition—"review things multiple times"—and apply it poorly. They cram before exams or re-read textbooks. Neither uses spacing effectively. Spaced repetition systems like Anki automate the scheduling so you review at optimal intervals: not too early (wasting time), not too late (you forgot it).

The catch: spaced repetition works best for material that's hard to learn and useful to retain. Vocabulary is classic. But it also works for technical concepts—if you write the right prompts.

Why Technical Knowledge is Different

Memorizing vocabulary is straightforward: Q: Spanish word for "house"? A: Casa. You either know it or you don't.

Technical knowledge is messier. If I ask you "What is gradient descent?" you could answer:

• "An optimization method that iteratively moves parameters toward lower loss."
• "It computes ∂L/∂θ and updates θ := θ − α∇L."
• "It's how neural networks learn, by following the error gradient downhill."
• "It's First-Order Taylor Approximation applied iteratively."

All are true, but they reveal different depths of understanding. A spaced repetition system that just checks "did you remember the definition" misses the point. You need prompts that force you to reason, not just recall.

How to Write Good Recall Prompts for Concepts

A bad prompt for "gradient descent":

Q: What is gradient descent? A: A method that updates parameters to reduce loss.

This invites surface-level memorization. You can parrot the definition without understanding when to use it or why it works. You'll forget it in a week.

A good prompt:

Q: You're training a model and the loss is stuck at a plateau for 10 epochs. Gradient descent is still running. Why might this happen, and what's one thing you'd check first? A: Learning rate too high (overshooting minima) or too low (no progress). Check the gradient norm and loss curve. If gradients are near zero, you're stuck in a flat region or bad initialization. If loss oscillates, learning rate is too high.

This prompt forces you to connect gradient descent to practice—diagnosis, not definition. You'll remember it because you've reasoned through it.

Quality Comparison: Bad vs. Good Prompts

| Metric | Bad Prompt | Good Prompt | |--------|-----------|------------| | Example | Q: What is attention? | Q: Why use multi-head attention? What breaks with single-head? | | Type | Definition recall | Scenario diagnosis | | Answer length | 1 sentence | 2–3 sentences | | Retention time | ~3 days | ~30 days (10× longer) | | Real-world applicable | No | Yes | | Thinking required | Minimal | Deep | | Failure rate week 2 | ~70% | ~20% |

"Failure rate" = % who can't answer in week 2. Good prompts have lower failure rates because you remember through reasoning, not rote.

Formula for Writing Good Prompts

1. Start with a concrete scenario. Not abstract.

   • Bad: "What is batch normalization?"
   • Good: "You're training a CNN. After batch norm, training is faster. Why? What could go wrong at inference?"

2. Ask for diagnosis or decision. Not definition.

   • Bad: "Define dropout."
   • Good: "Your model overfits. Dropout is one tool. Why does it help? When wouldn't you use it?"

3. Expect a multi-sentence answer. Longer = harder = better memory.

4. Ground it in your actual work. Your prompts should reflect what you're learning.

Concrete Prompt Examples with Answers

# Spaced Repetition Cards – Transformers & Attention

## Card 1: Scaling in Attention
Q: In transformer attention, why compute QK^T / √d_k instead of QK^T?

A: Large dot products push softmax into the flat tail (vanishing gradients).
Dividing by √d_k normalizes the scale. Example: d_k=64, divide by ~8,
keeps attention stable.

---

## Card 2: Multi-Head Attention Trade-off
Q: A transformer has 12 attention heads with d_model=768. Why not 1 big
head or 1000 tiny heads?

A: One big head: limited diversity, all parameters focused on one operation.
Many tiny heads: diverse patterns but low capacity per head. 12 heads is
empirical sweet spot. Different heads learn different patterns: one attends
to previous token, another attends globally.

---

## Card 3: Position Information in Transformers
Q: Transformers have no recurrence. How do they know token position?

A: Positional encodings are added to embeddings before layer 1. Sinusoidal
encodings use sin/cos at different frequencies per dimension. Alternative:
learnable embeddings. Without position, "dog bites man" and "man bites dog"
are indistinguishable.

---

## Card 4: Batch Norm Debugging
Q: You add batch norm to your CNN. Training is faster but validation
accuracy is lower. Why?

A: Batch norm uses batch statistics (mean/var) at train time, but running
statistics at test time. If they diverge, validation suffers. Solution:
increase batch size or disable batch norm momentum during evaluation.

Building a Card Library Over Time

Week 1: Add 5 cards about core concepts. Review time: 5 min/day.

Week 2: Add 5 more. Anki shows Week 1 cards (some "Hard" back in 1 day, some "Easy" in 10 days). Total: 10 cards, 10 min/day.

Week 4: 20 cards. You review 10/day (mix of old and new). Week 1 "Hard" cards reviewed 3–4 times; sticky now. Time: 15 min/day.

Month 3: 55 cards. Review 15–20/day. Sustainable; ~30 min/week. Total time invested: ~10 hours. Durable retention: 99.9% of 55 concepts for 6+ months.

Building a Weekly Review Routine

Spaced repetition only works if you actually review. Here's a sustainable routine:

Monday: Add new cards (15 minutes). After learning something from a course, paper, or project, convert it to 1–3 cards in your spaced repetition system (Anki is the standard). Write the prompt (scenario), write the answer, add links to the source.

Don't create a card for every detail. Pick 10% of what you learned—the things you're likely to forget but need to know. If you try to card everything, the system becomes a burden.

Wednesday: Review new cards (10 minutes). Anki shows you today's new cards plus anything from earlier days that's due. Rate your answers: "Easy" (I nailed it), "Good" (I got it but hesitated), "Hard" (I struggled), "Again" (I didn't know).

Friday: Review new cards (10 minutes). Spaced repetition systems increase the interval between reviews based on your ratings. "Easy" cards come back in 10 days; "Hard" cards come back in 1 day.

Sunday: Longer review session (20–30 minutes). Catch-up day for the week. Review anything overdue. The goal isn't speed; it's accuracy. If you rush and guess, you're training your brain to forget faster.

Total time: ~1 hour per week. Over a year, that's 52 hours of review, which locks ~50–100 major concepts into long-term memory with far less effort than rereading a textbook.

Anki Setup and Configuration

Create a new Anki deck with this base configuration:

# Anki Deck Settings (for technical ML concepts)

New Cards:
  steps: "1 10"              # Review after 1 min, then 10 min
  graduating interval: 1     # Card becomes "young" after 1 day if you rate "Good"
  easy interval: 4           # Card becomes "learning" after 4 days if you rate "Easy"
  starting ease: 250%        # 2.5x – controls how fast intervals grow

Reviews:
  easy bonus: 130%           # "Easy" cards grow intervals by 30% more
  interval modifier: 100%    # Keep this at 100% (default)
  hard penalty: 60%          # "Hard" cards shrink to 60% of interval
  maximum interval: 36500    # Max ~100 years (effectively "learned forever")

Lapses (cards you forget after learning):
  steps: "10"                # Show again after 10 min
  new interval: 25%          # Go back to 25% of original interval
  leash: 0.0                 # Reset completely on hard lapse

# This config is gentle: prioritizes retention over speed.
# Adjust "steps" if you want faster/slower cards.

Recommended deck contents for a ~50-card ML curriculum:

• 5 cards: Core gradient descent / backprop concepts
• 5 cards: Optimization (SGD, Adam, learning rates)
• 10 cards: Neural network architecture (activations, normalization, pooling)
• 15 cards: Transformers & attention (Q/K/V, positional encoding, multi-head)
• 10 cards: Training best practices (regularization, overfitting, validation)
• 5 cards: Domain-specific (vision-specific or NLP-specific, your choice)

Anki in Practice: A Three-Month Example

Let's see what a spaced repetition system looks like over 12 weeks.

Month 1:

• Week 1: Add 5 cards about gradient descent (chain rule, backprop, learning rates).
• Week 2: Add 5 cards about batch normalization. Review gradient descent cards (some rated "Hard," so they come back in 1 day).
• Week 3: Add 5 cards about dropout and regularization. Gradient descent cards come back in 3–10 day intervals now. Batch norm cards appear for the first time.
• Week 4: Add 5 cards about activation functions. First catch-up week; you're reviewing gradient descent and batch norm from earlier.

Month 2:

• Cards are on more spaced intervals (5–14 days). You're adding 4–5 new cards per week, reviewing 10–15 old cards per week.
• Total cards: 40 after month 2. Daily review time is 15–20 minutes (adding new + reviewing old).

Month 3:

• Cards are on very spaced intervals (10–30 days). You're adding 3–4 new cards per week (narrowing your focus), reviewing 15–20 old cards per week.
• Total cards: ~55 after month 3.
• The "Hard" cards (batch norm, learning rates) have been reviewed 6–8 times and are now locked into memory.
• The "Easy" cards (basic definition of gradient descent) are reviewed every 2–3 weeks and don't require much thought.

After three months, you've invested ~10 hours total (1 hour/week) and have 55 concepts durably memorized. This is efficient compared to highlighting a textbook or re-reading chapters.

Anki Settings That Matter

Default Anki settings work for most people, but a few tweaks help:

New cards per day: Start with 5, increase to 10 if you're motivated. More than 20 per day is usually unsustainable.

Ease factor: Controls how long intervals grow. Default is 2.5× (if you rate "Easy," next interval is 2.5× longer). For technical concepts, default is fine. Don't fiddle with this.

Review order: Set to "show new cards after reviews." This way, you review old material first, then add new cards. Psychologically, you end the study session with new, interesting content (not old review).

Tags: Use tags to organize (e.g., #transformers, #reinforcement-learning, #implementation). Later, you can review only cards in one domain or see which domains you're weakest in.

These settings are one-time setup. Don't obsess over optimization; the 80% solution (reasonable settings, consistent review) beats perfect settings used sporadically.

The Illusion of Understanding

Spaced repetition is powerful, but it has a trap: fluency doesn't equal understanding.

If you practice recalling "What does a GRU do?" over and over, you'll eventually rattle off "It's a gated recurrent unit with reset and update gates." You'll feel fluent. But fluency with a definition is not the same as understanding why gated units work or when they're better than LSTMs.

To avoid this trap:

Use spaced repetition for facts and concepts you've already struggled with. Don't use it as your primary learning method. You should first learn (by doing projects, reading, coding) and then use spaced repetition to retain what you've learned.

Write prompts that test application, not just recall. Every few weeks, look at your old cards and upgrade prompts that have become too easy. Change "What is X?" to "When would you use X instead of Y?" or "How would you debug a failure in X?"

Couple spaced repetition with active recall in your real work. If you're building a recommendation system, don't just review "collaborative filtering" cards. Actually implement collaborative filtering. The project makes the concepts stick in ways cards alone can't.

Worked Example: Spaced Repetition for Transformers

Imagine you're learning transformers and you want to solidify your understanding over 8 weeks.

Week 1: You learn attention and self-attention from a tutorial.

• Card 1: Q: In self-attention, what are Q, K, V, and why do we compute QK^T / √d_k? (Conceptual)
• Card 2: Q: If a transformer has 12 attention heads and d_model=768, what's d_head? Why split into multiple heads instead of one big head? (Design reasoning)
• Card 3: Q: You're encoding a sentence "The cat sat on the mat." Explain why the word "cat" might attend more strongly to "sat" than "mat" in an early layer. (Intuition)

Week 2: Anki shows you these cards again (new card interval).

• You rate Card 1 "Good" (you knew it but hesitated).
• You rate Card 3 "Hard" (you couldn't explain the intuition clearly).
• Anki reschedules Card 1 for 10 days, Card 3 for 2 days.

Week 3: You learn positional encoding.

• You add 2 new cards about positional encoding.
• Card 3 about attention comes up again (it's been 2 days since "Hard").
• You rate it "Good" this time—you've thought about it offline, and now you've got the intuition.

Week 4: You're building a transformer from scratch.

• The cards become anchors for your implementation. You know why you're computing certain things.
• A card about gradient flow in deep networks comes up; it's directly relevant to debugging your code.

Week 8: You've reviewed ~20 concepts over 8 weeks.

• The "Hard" cards have been reviewed 5–6 times; they're locked in.
• The "Easy" cards have been reviewed 1–2 times; they're in long-term memory.
• You can now explain transformers to someone else without referring to notes.

Common Mistake

The biggest mistake is creating too many cards. If you add 50 new cards a week, you'll spend 2+ hours daily reviewing, the system becomes a chore, and you'll quit in month 2.

Start with 5–10 new cards per week. This sounds like very little, but 10 cards per week × 52 weeks = 520 cards over a year, and you'll actually review them. 50 cards per week sounds ambitious but usually leads to burnout.

Another mistake: treating cards like flashcards. Flashcards are for fast recall; spaced repetition is for durable understanding. If a card doesn't teach you something you'll use, delete it. If a card doesn't force you to reason, rewrite it.

A third mistake: reviewing but not implementing. Spaced repetition alone won't make you a good ML engineer. You need projects. Cards are the glue that holds concepts in memory so they're available for projects. Without projects, your cards feel sterile and you'll quit.