Academy K

Variance Reduction in Policy Gradients

InterviewDerivationpolicy gradientvarianceRL

Related module: RLHF and reward learning

Problem statement

Policy gradient methods suffer from high variance.

  1. Write the REINFORCE gradient estimator.
  2. How does the baseline reduce variance without introducing bias?
  3. Derive why the baseline doesn't introduce bias.

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.