Academy K

Self-Attention Computation

InterviewDerivationattentiontransformersmatrix operations

Related module: Sequence models and attention

Problem statement

In self-attention, queries QQ, keys KK, and values VV are computed from input X∈Rn×dX \in \mathbb{R}^{n \times d} where nn is sequence length.

Q=XWQ,K=XWK,V=XWVQ = XW^Q, \quad K = XW^K, \quad V = XW^V
Attention(Q,K,V)=softmax(QKTdk)V\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
  1. What are the dimensions of each intermediate tensor?
  2. Why divide by dk\sqrt{d_k}?
  3. What is the computational complexity of self-attention?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.