Academy K

Grokking in Neural Networks

HardConceptualgrokkinggeneralizationtraining dynamics

Related module: Deep learning fundamentals

Problem statement

Grokking is the phenomenon where generalization occurs long after memorization.

  1. Describe the grokking phenomenon with the modular arithmetic example.
  2. What theories explain why grokking occurs?
  3. What implications does grokking have for training and safety?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.