Position Encodings in Transformers
Warm-upDerivationtransformersposition encodingattention
Related module: Transformers and autoregressive models
Problem statement
Transformers need position information since attention is permutation-invariant.
- Write the sinusoidal position encoding formula.
- Why are sinusoidal encodings designed with varying frequencies?
- Compare sinusoidal encodings to learned position embeddings.