Academy K

Direct Preference Optimization

HardDerivationDPOalignmentoptimization

Related module: Post-training and alignment methods

Problem statement

Direct Preference Optimization (DPO) bypasses reward model training.

Starting from the RLHF objective:

max⁡πE[r(x,y)]−β⋅KL(π∣∣πref)\max_\pi \mathbb{E}[r(x,y)] - \beta \cdot \text{KL}(\pi || \pi_{\text{ref}})

  1. Show that the optimal policy is π∗(y∣x)∝πref(y∣x)exp⁡(r(x,y)/β)\pi^*(y|x) \propto \pi_{\text{ref}}(y|x) \exp(r(x,y)/\beta).
  2. Derive the DPO loss function.
  3. What are the advantages of DPO over RLHF?

Hints

Practise this problem

Verify you understand it—not just that you read the answer.

Learn the foundations

The AI / SI track covers this topic and more in depth.