Attention Mechanism Complexity
InterviewDerivationattentioncomplexityefficiency
Related module: Transformers and autoregressive models
Problem statement
The standard self-attention mechanism has specific computational requirements.
- Derive the time and space complexity of self-attention for sequence length and dimension .
- Why is this problematic for long sequences?
- Name three approaches to reduce attention complexity and their trade-offs.