Vanishing Gradient Problem
InterviewConceptualgradientsdeep networksactivation functions
Related module: Learning from data and backpropagation
Problem statement
Deep networks with sigmoid activations often suffer from vanishing gradients.
- Why does the gradient vanish when using sigmoid activations in deep networks?
- How does ReLU help mitigate this problem?
- What other techniques address vanishing gradients?