Academy K

Implementing a Transformer Block

InterviewCodingtransformersimplementationarchitecture

Related module: Sequence models and attention

Problem statement

Implement a single Transformer decoder block in PyTorch with:
- Multi-head self-attention
- Feed-forward network
- Layer normalization (Pre-LN style)
- Residual connections

Include causal masking for autoregressive generation.

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.