Academy K

Perplexity as Evaluation Metric

Warm-upDerivationevaluationperplexitylanguage models

Related module: Evaluation and interpretability

Problem statement

Perplexity is a standard metric for language models.

  1. Define perplexity mathematically.
  2. If a model achieves perplexity 10 on a test set, what does this mean intuitively?
  3. What are the limitations of perplexity as an evaluation metric?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.