Grokking in Neural Networks
HardConceptualgrokkinggeneralizationtraining dynamics
Related module: Deep learning fundamentals
Problem statement
Grokking is the phenomenon where generalization occurs long after memorization.
- Describe the grokking phenomenon with the modular arithmetic example.
- What theories explain why grokking occurs?
- What implications does grokking have for training and safety?