Academy K

Neural Scaling Laws

InterviewConceptualscalingcomputetraining

Related module: Training and scaling language models

Problem statement

Empirical scaling laws describe how loss decreases with compute, data, and parameters.

  1. State the Chinchilla scaling law relating optimal model size to training tokens.
  2. For a fixed compute budget, should you train a larger model on less data or a smaller model on more data?
  3. What happens to these laws as models get very large?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.