Academy K

Compute-Optimal Training Budget

InterviewDerivationcomputescalingoptimization

Related module: Training and scaling language models

Problem statement

You have a compute budget of 102310^{23} FLOPs. Using the approximation that training FLOPs ≈6ND\approx 6ND where NN is parameters and DD is tokens:

  1. If you follow Chinchilla-optimal scaling (N≈D/20N \approx D/20), what model size and token count should you use?
  2. Estimate the cost in GPU-hours assuming an A100 at 300 TFLOPs effective.
  3. If compute costs \2$/GPU-hour, what's the training cost?

Hints

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.