Layer Normalization vs Batch Normalization
InterviewConceptualnormalizationtransformerstraining
Related module: Sequence models and attention
Problem statement
Transformers use Layer Normalization rather than Batch Normalization.
- Define both normalizations mathematically.
- Why is LayerNorm preferred for transformers?
- What is the difference between Pre-LN and Post-LN transformers?