Academy K

Distributional Shift in RLHF

InterviewConceptualRLHFdistributional shiftreward modeling

Related module: RLHF and reward learning

Problem statement

RLHF involves training a reward model on human preferences, then optimizing a policy against it.

  1. Why does the policy distribution shift from the reward model's training distribution?
  2. What failure modes does this cause?
  3. How is the KL penalty used to mitigate this?

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.