Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization
Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update. We re-implement DNO-Prct…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
