Academy K

Extending Context Length

HardConceptualcontext lengthattentionarchitecture

Related module: Sequence models and attention

Problem statement

Modern LLMs are extending context windows from 2K to 100K+ tokens.

  1. What are the main bottlenecks for long context?
  2. Describe three techniques for extending context length.
  3. How do you evaluate whether a model actually uses long context?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.