Academy K

Softmax Temperature Scaling

Warm-upDerivationsoftmaxtemperatureprobability

Related module: Learning from data and backpropagation

Problem statement

The softmax function with temperature TT is:

softmax(zi;T)=ezi/T∑jezj/T\text{softmax}(z_i; T) = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}
  1. What happens as T→0T \to 0?
  2. What happens as T→∞T \to \infty?
  3. For logits z=[2.0,1.0,0.1]z = [2.0, 1.0, 0.1], compute the softmax output for T=1T = 1 and T=0.5T = 0.5.

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.