Academy K

Energy Cost of Neural Network Inference

InterviewDerivationenergyinferenceefficiency

Related module: Connections to AI efficiency

Problem statement

Estimate the energy cost of running inference on a large language model.

For a model with NN parameters, context length LL, and generating TT output tokens:

  1. How many floating-point operations (FLOPs) are required?
  2. For a GPU at efficiency η\eta TFLOPs/Watt, what is the energy per token?
  3. Compare to the theoretical Landauer limit.

Hints

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The Thermodynamic Computing track covers this topic and more in depth.