Self-Attention Computation
InterviewDerivationattentiontransformersmatrix operations
Related module: Sequence models and attention
Problem statement
In self-attention, queries , keys , and values are computed from input where is sequence length.
- What are the dimensions of each intermediate tensor?
- Why divide by ?
- What is the computational complexity of self-attention?