همایش بین المللی هوش مصنوعی

صفحه اصلی / دومین همایش بین المللی هوش مصنوعی

From RLHF to DPO and Beyond: A Review of Preference Optimization Methods for Aligning Large Language Models

نویسندگان :

Poorya Saneei¹ Parham Rahimi² Behrouz Minaei-Bidgoli³

1- دانشگاه علم و صنعت ایران 2- دانشگاه علم و صنعت ایران 3- دانشگاه علم و صنعت ایران

کلمات کلیدی :

Large Language Models،LLM Alignment،Preference Optimization،Reinforcement Learning from Human Feedback (RLHF)،Direct Preference Optimization (DPO)،Reward Model،Reference-Free Algorithms

چکیده :

LLMs are not aligned with human values by default, and this challenge has fostered the development of various alignment methods to make models helpful, honest, and harmless. One of the first practical methods for solving this problem was Reinforcement Learning from Human Feedback (RLHF), but its complexity, instability, and high computational cost have prevented it from large-scale application. Since then, many alignment techniques have been suggested, but a consistent review of their evolution is missing. This paper traces the evolution of the alignment methods, focusing on how it evolved from RLHF to the family of algorithms called Direct Preference Optimization, or simply DPO. It elegantly reparametrizes the alignment problem into a supervised learning problem. Further, we consider how the new generation of algorithms overcame the limitations of DPO along three major axes: better computational efficiency by removing the reference model, flexibility of formats and objectives, and optimization for complex goals such as creativity. This paper provides an overview of how alignment research has moved from complexity to simple, efficient, and modular algorithms and shows what the future of alignment will be like.