Academy K

Layer Normalization vs Batch Normalization

InterviewConceptualnormalizationtransformerstraining

Related module: Sequence models and attention

Problem statement

Transformers use Layer Normalization rather than Batch Normalization.

  1. Define both normalizations mathematically.
  2. Why is LayerNorm preferred for transformers?
  3. What is the difference between Pre-LN and Post-LN transformers?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.