Academy K

Sinusoidal Positional Encoding

InterviewConceptualpositional encodingtransformerssequences

Related module: Sequence models and attention

Problem statement

The original Transformer uses sinusoidal positional encodings:

PE(pos,2i)=sin⁡(pos/100002i/d)PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d})
PE(pos,2i+1)=cos⁡(pos/100002i/d)PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d})
  1. Why does attention need positional information?
  2. What property allows the model to learn relative positions?
  3. How do learned positional embeddings compare?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.