Neural Scaling Laws
InterviewDerivationscalingpower lawscompute
Related module: Deep learning fundamentals
Problem statement
Neural network performance follows predictable scaling laws.
- State the Chinchilla scaling law relating optimal model size to compute budget.
- If you have more compute, how should you allocate it optimally?
- What are the limitations of scaling law extrapolations?