Direct Preference Optimization
HardDerivationDPOalignmentoptimization
Related module: Post-training and alignment methods
Problem statement
Direct Preference Optimization (DPO) bypasses reward model training.
Starting from the RLHF objective:
- Show that the optimal policy is .
- Derive the DPO loss function.
- What are the advantages of DPO over RLHF?
Hints