Academy K

Cross-Entropy Loss Derivation

Warm-upDerivationloss functionscross-entropyclassification

Related module: Learning from data and backpropagation

Problem statement

The cross-entropy loss for classification is:

L=−∑iyilog⁡(pi)L = -\sum_i y_i \log(p_i)

where yy is the one-hot target and pp is the predicted probability distribution.

  1. For a single correct class cc, simplify this expression.
  2. Show that minimizing cross-entropy is equivalent to maximizing log-likelihood.
  3. What is the gradient ∂L∂zc\frac{\partial L}{\partial z_c} where p=softmax(z)p = \text{softmax}(z)?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.