KV Cache for Efficient Inference
InterviewConceptualinferenceKV cacheefficiency
Related module: Training and scaling language models
Problem statement
During autoregressive generation, the KV cache stores past key-value pairs to avoid recomputation.
- Why is the KV cache necessary for efficient generation?
- What is the memory cost of the KV cache?
- How do techniques like Multi-Query Attention (MQA) reduce this cost?