Academy K

Position Encodings in Transformers

Warm-upDerivationtransformersposition encodingattention

Related module: Transformers and autoregressive models

Problem statement

Transformers need position information since attention is permutation-invariant.

  1. Write the sinusoidal position encoding formula.
  2. Why are sinusoidal encodings designed with varying frequencies?
  3. Compare sinusoidal encodings to learned position embeddings.

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.