Energy Cost of Neural Network Inference
InterviewDerivationenergyinferenceefficiency
Related module: Connections to AI efficiency
Problem statement
Estimate the energy cost of running inference on a large language model.
For a model with parameters, context length , and generating output tokens:
- How many floating-point operations (FLOPs) are required?
- For a GPU at efficiency TFLOPs/Watt, what is the energy per token?
- Compare to the theoretical Landauer limit.
Hints
Learn the foundations
The Thermodynamic Computing track covers this topic and more in depth.