Distributional Shift in RLHF
InterviewConceptualRLHFdistributional shiftreward modeling
Related module: RLHF and reward learning
Problem statement
RLHF involves training a reward model on human preferences, then optimizing a policy against it.
- Why does the policy distribution shift from the reward model's training distribution?
- What failure modes does this cause?
- How is the KL penalty used to mitigate this?