Academy K

Speculative Decoding

HardConceptualinferencedecodingefficiency

Related module: Training and scaling language models

Problem statement

Speculative decoding uses a small "draft" model to accelerate inference from a large "target" model.

  1. How does speculative decoding work?
  2. Why does it produce identical outputs to standard decoding?
  3. What speedup can be achieved and when?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.