Sinusoidal Positional Encoding
InterviewConceptualpositional encodingtransformerssequences
Related module: Sequence models and attention
Problem statement
The original Transformer uses sinusoidal positional encodings:
- Why does attention need positional information?
- What property allows the model to learn relative positions?
- How do learned positional embeddings compare?