Neural Scaling Laws
InterviewConceptualscalingcomputetraining
Related module: Training and scaling language models
Problem statement
Empirical scaling laws describe how loss decreases with compute, data, and parameters.
- State the Chinchilla scaling law relating optimal model size to training tokens.
- For a fixed compute budget, should you train a larger model on less data or a smaller model on more data?
- What happens to these laws as models get very large?