Academy K

Vanishing Gradient Problem

InterviewConceptualgradientsdeep networksactivation functions

Related module: Learning from data and backpropagation

Problem statement

Deep networks with sigmoid activations often suffer from vanishing gradients.

  1. Why does the gradient vanish when using sigmoid activations in deep networks?
  2. How does ReLU help mitigate this problem?
  3. What other techniques address vanishing gradients?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.