Extending Context Length
HardConceptualcontext lengthattentionarchitecture
Related module: Sequence models and attention
Problem statement
Modern LLMs are extending context windows from 2K to 100K+ tokens.
- What are the main bottlenecks for long context?
- Describe three techniques for extending context length.
- How do you evaluate whether a model actually uses long context?