Compute-Optimal Training Budget
InterviewDerivationcomputescalingoptimization
Related module: Training and scaling language models
Problem statement
You have a compute budget of FLOPs. Using the approximation that training FLOPs where is parameters and is tokens:
- If you follow Chinchilla-optimal scaling (), what model size and token count should you use?
- Estimate the cost in GPU-hours assuming an A100 at 300 TFLOPs effective.
- If compute costs \2$/GPU-hour, what's the training cost?
Hints