Academy K

Output Token Probabilities and Calibration

Warm-upDerivationprobabilitycalibrationlanguage models

Related module: Transformers and autoregressive models

Problem statement

Language models output probability distributions over tokens.

  1. For logits z=[3.0,1.0,0.5]z = [3.0, 1.0, 0.5] over tokens [A, B, C], compute the probabilities.
  2. What does it mean for a model to be calibrated?
  3. How can you estimate the probability that a model's answer is correct using its token probabilities?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.