Speculative Decoding
HardConceptualinferencedecodingefficiency
Related module: Training and scaling language models
Problem statement
Speculative decoding uses a small "draft" model to accelerate inference from a large "target" model.
- How does speculative decoding work?
- Why does it produce identical outputs to standard decoding?
- What speedup can be achieved and when?