RLHF

Source: https://github.com/opendilab/awesome-RLHF at 96acdd4571e7
License: Apache-2.0
Read the English original

This edition is an AI-assisted translation.

Великолепно RLHF (RL с человеческой обратной связью)

Это сборник исследовательских работ для ** Усиление обучения с человеческой обратной связью**RLHF). И хранилище будет постоянно обновляться, чтобы отслеживать границы RLHF.

Добро пожаловать и звезда!

Таблица содержимого

• Великолепно RLHF (RL с человеческой обратной связью)
  • Таблица содержимого
  • Обзор RLHF
    • Подробное объяснение
  • Документы
    • 2026
    • 2025
    • 2024
    • 2023
    • 2022
    • 2021
    • 2020 и ранее
  • Кодовые базы
  • Набор данных
  • Блоги
  • Книги
  • Поддержка других языков
  • Вклад
  • Лицензия

Обзор RLHF

Идея о RLHF Это использование методов обучения с подкреплением для прямой оптимизации языковой модели с обратной связью с человеком. RLHF Это позволило языковым моделям начать выравнивать модель, обученную на общем корпусе текстовых данных, с моделью сложных человеческих ценностей.

• RLHF Для большой языковой моделиLLM)

• RLHF Видеоигры (например). Atari)

Подробное объяснение

** (Следующий раздел был создан автоматически) ChatGPT)**

RLHF Это, как правило, относится к «укреплению обучения с человеческой обратной связью». Reinforcement Learning (RL) - это тип машинного обучения, который включает в себя обучение агента принимать решения на основе обратной связи из окружающей среды. в RLHFАгент также получает обратную связь от людей в виде оценок или оценок его действий, которые могут помочь ему быстрее и точнее учиться.

RLHF Это активная область исследований в области искусственного интеллекта с приложениями в таких областях, как робототехника, игры и персонализированные системы рекомендаций. Она направлена на решение проблем RL в сценариях, где агент имеет ограниченный доступ к обратной связи от окружающей среды и требует участия человека для повышения своей производительности.

Усиление обучения с обратной связью с человеком (RLHF) является быстро развивающейся областью исследований в области искусственного интеллекта, и существует несколько передовых методов, которые были разработаны для повышения производительности. RLHF систем. Вот несколько примеров:

• Inverse Reinforcement Learning (IRL): IRL Это метод, который позволяет агенту изучать функцию вознаграждения из обратной связи человека, а не полагаться на заранее определенные функции вознаграждения. Это позволяет агенту учиться на более сложных сигналах обратной связи, таких как демонстрация желаемого поведения.

• Apprenticeship LearningОбучение ученичеству - это техника, которая объединяет IRL с контролируемым обучением, чтобы агент мог учиться как на отзывах людей, так и на демонстрациях экспертов. Это может помочь агенту учиться быстрее и эффективнее, так как он может учиться как на положительной, так и на отрицательной обратной связи.

• Interactive Machine Learning (IML): IML Это метод, который предполагает активное взаимодействие между агентом и экспертом-человеком, позволяя эксперту предоставлять обратную связь о действиях агента в режиме реального времени. Это может помочь агенту учиться быстрее и эффективнее, так как он может получать обратную связь о своих действиях на каждом этапе процесса обучения.

• Human-in-the-Loop Reinforcement Learning (HITLRL): HITLRL Это метод, который включает в себя интеграцию обратной связи человека в процесс RL на нескольких уровнях, таких как формирование вознаграждения, выбор действий и оптимизация политики. Это может способствовать повышению эффективности и результативности RLHF Система использует преимущества как людей, так и машин.

Вот несколько примеров обучения с подкреплением обратной связью с человеком.RLHF):

• Game PlayingВ игре обратная связь с человеком может помочь агенту изучить стратегии и тактики, которые эффективны в различных сценариях игры. Например, в популярной игре Go специалисты-люди могут дать обратную связь агенту по его ходам, помогая ему улучшить свой геймплей и принятие решений.

• Personalized Recommendation SystemsВ системах рекомендаций обратная связь с человеком может помочь агенту узнать предпочтения отдельных пользователей, что позволяет предоставлять персонализированные рекомендации. Например, агент может использовать отзывы пользователей о рекомендуемых продуктах, чтобы узнать, какие функции наиболее важны для них.

• RoboticsВ робототехнике обратная связь с человеком может помочь агенту научиться безопасно и эффективно взаимодействовать с физической средой. Например, робот может научиться более быстро ориентироваться в новой среде с обратной связью от человека-оператора на лучшем пути или каких объектов избегать.

• EducationВ образовании обратная связь с человеком может помочь агенту научиться более эффективно обучать студентов. Например, репетитор на основе ИИ может использовать обратную связь от учителей, на которых стратегии обучения лучше всего работают с разными студентами, помогая персонализировать опыт обучения.

Документы

Вы также можете Посетить эту ссылку Чтобы получить опыт чтения бумаги с ИИ.

format:
- [title](paper link) [links]
  - author1, author2, and author3...
  - publisher
  - keyword
  - code
  - experiment environments and datasets

2026

• Why DPO is a Misspecified Estimator and How to Fix It

  • Aditya Gopalan, Sayak Ray Chowdhury, Debangshu Banerjee
  • Ключевое слово: DPO, RLHF, Preference, Alignment, LLM

• What’s In My Human Feedback? Learning Interpretable Descriptions of Preference Data

  • Rajiv Movva, Smitha Milli, Sewon Min, Emma Pierson
  • Ключевое слово: RLHF, Preference, Alignment, Safety, Human Feedback

• Multiplayer Nash Preference Optimization

  • Fang Wu, Xu Huang, Weihao Xuan, Zhiwei Zhang, Yijia Xiao, Guancheng Wan, Xiaomin Li, Bing Hu, Peng Xia, Jure Leskovec, Yejin Choi
  • Ключевое слово: PPO, RLHF, Preference, Alignment, LLM

• Token-Importance Guided Direct Preference Optimization

  • Ning Yang, Hai Lin, Yibo Liu, Baoliang Tian, Guoqing Liu, Haijun Zhang
  • Ключевое слово: DPO, RLHF, Preference, Alignment, LLM

• SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety

  • Geon-Hyeong Kim, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, Youngsoo Jang, Moontae Lee
  • Ключевое слово: DPO, RLHF, Reward Model, Preference, Alignment

• BaseReward: A Strong Baseline for Multimodal Reward Model

  • YiFan Zhang, Haihua Yang, Huanyu Zhang, Yang Shi, Zezhou Chen, Haochen Tian, Chaoyou Fu, Kai WU, Bo Cui, Xu Wang, Jianfei Pan, Haotian Wang, Zhang Zhang, Liang Wang
  • Ключевое слово: RLHF, Reward Model, Preference, Multimodal, LLM

• The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives

  • Matthieu Bou, Nyal Patel, Arjun Jagota, Satyapriya Krishna, Sonali Parbhoo
  • Ключевое слово: RLHF, Preference, Alignment, Safety, LLM

• Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs

  • Shangpin Peng, Weinong Wang, Zhuotao Tian, Senqiao Yang, Xing W, Haotian Xu, Chengquan Zhang, Takashi Isobe, Baotian Hu, Min Zhang
  • Ключевое слово: DPO, RLHF, Preference, Multimodal, LLM

• Learning to summarize user information for personalized reinforcement learning from human feedback

  • HyunJi Nam, Yanming Wan, Mickel Liu, Peter F. Ahnn, Jianxun Lian, Natasha Jaques
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, LLM

• Token-Guard: Towards Token-Level Hallucination Control via Self-Checking Decoding

  • Yifan Zhu, Huiqiang Rong, Haoran Luo
  • Ключевое слово: RLHF, LLM, Token-level, Reinforcement Learning, Human Feedback

• Pretrain Value, Not Reward: Decoupled Value Policy Optimization

  • Chenghua Huang, Lu Wang, Fangkai Yang, Pu Zhao, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan
  • Ключевое слово: RLHF, Reward Model, Preference, LLM, Optimization

• P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization

  • ruipeng zhang, Zhihao Li, Haozhang Yuan, C.L.Philip Chen, Tong Zhang
  • Ключевое слово: DPO, Preference, Optimization, Human Feedback

• Unifying Stable Optimization and Reference Regularization in RLHF

  • Li He, Qiang Qu, He Zhao, Stephen Wan, Dadong Wang, Lina Yao, Tongliang Liu
  • Ключевое слово: RLHF, Preference, Alignment, Optimization, Reinforcement Learning

• Text2Grad: Reinforcement Learning from Natural Language Feedback

  • Hanyang Wang, Lu Wang, Chaoyun Zhang, Tianjun Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
  • Ключевое слово: RLHF, Reward Model, Alignment

• ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoning

  • Zhengyue Zhao, YingziYingzi Ma, Somesh Jha, Marco Pavone, Patrick McDaniel, Chaowei Xiao
  • Ключевое слово: RLHF, Alignment, Safety, LLM, Optimization

• Reward Model Routing in Alignment

  • Xinle Wu, Yao Lu
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, LLM

• CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solving

  • Shuhang Chen, Yunqiu Xu, Junjie Xie, Aojun Lu, Tao Feng, ZEYING HUANG, ZHANG NING, Yi Sun, Yi Yang, Hangjie Yuan
  • Ключевое слово: Reward Model, Multimodal, Optimization

• All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning

  • Gokul Swamy, Sanjiban Choudhury, Wen Sun, Steven Wu, Drew Bagnell
  • Ключевое слово: PPO, Reward Model, Preference, Reinforcement Learning

• General Exploratory Bonus for Optimistic Exploration in RLHF

  • Wendi Li, Changdae Oh, Sharon Li
  • Ключевое слово: RLHF, Alignment, Reinforcement Learning, Human Feedback

• RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment

  • Xiaoyang Cao, Zelai Xu, Mo Guang, Kaiwen Long, Michiel A. Bakker, Yu Wang, Chao Yu
  • Ключевое слово: DPO, RLHF, Preference, Alignment, LLM

• Learning Correlated Reward Models: Statistical Barriers and Opportunities

  • Yeshwanth Cherapanamjeri, Constantinos Costis Daskalakis, Gabriele Farina, Sobhan Mohammadpour
  • Ключевое слово: RLHF, Reward Model, Preference, Reinforcement Learning, Human Feedback

• Verification and Co-Alignment via Heterogeneous Consistency for Preference-Aligned LLM Annotations

  • Cheng Chen, Haiyan Yin, Ivor Tsang
  • Ключевое слово: RLHF, Preference, Alignment, LLM

• Disentangling Length Bias in Preference Learning via Response-Conditioned Modeling

  • Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Yue Wang, Li Li, Wengang Zhou, Houqiang Li
  • Ключевое слово: DPO, RLHF, Reward Model, Preference, LLM

• Enforcing Axioms for AI Alignment under Loss-Based Rules

  • Alexandros Hollender, Sonja Kraiczy
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, Reinforcement Learning

• OPPO: Accelerating PPO-based RLHF via Pipeline Overlap

  • Kaizhuo Yan, YingJie Yu, Yifan Yu, Haizhong Zheng, Fan Lai
  • Ключевое слово: PPO, RLHF, Reward Model, Preference, LLM

• QuRL: Rubrics As Judge For Open-Ended Question Answering

  • Xiyu Wei, Qingwei Zong, Xiaoguang Li, Eugene J. Yu, Sujian Li
  • Ключевое слово: LLM, Optimization, Reinforcement Learning, Human Feedback

• Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations

  • Xinyi Yang, Liang Zeng, Heng Dong, Chao Yu, Xiaoran Wu, Huazhong Yang, Yu Wang, Milind Tambe, Tonghan Wang
  • Ключевое слово: RLHF, LLM, Reinforcement Learning

• Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy

  • Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, Safety

• Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game

  • Barna Pásztor, Thomas Kleine Buening, Andreas Krause
  • Ключевое слово: RLHF, Preference, Alignment, Nash, Optimization

• Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback

  • Gihoon Kim, Euntai Kim
  • Ключевое слово: RLHF, Reward Model, Preference, Reinforcement Learning, Human Feedback

• Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback

  • Amirhossein Afsharrad, Ruida Zhou, Luca Viano, Sanjay Lall, Mohammad Ghavamzadeh
  • Ключевое слово: Reward Model, Preference, Safety, Human Feedback

• RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards

  • Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Ellie Evans, Daniel Egert, Hoo-Chang Shin, Felipe Soares, Yi Dong, Oleksii Kuchaiev
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, LLM

• Reward Models Inherit Value Biases from Pretraining

  • Brian Christian, Jessica A F Thompson, Elle, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, Tsvetomira Dumbalska
  • Ключевое слово: Reward Model, Preference, Alignment, Safety, LLM

• Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignment

  • Byeonghu Na, Hyungho Na, Yeongmin Kim, Suhyeon Jo, HeeSun Bae, Mina Kang, Il-chul Moon
  • Ключевое слово: RLHF, Preference, Alignment, LLM, Reinforcement Learning

• RewardBench 2: Advancing Reward Model Evaluation

  • Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, Nathan Lambert
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, Safety

• Causally Robust Reward Learning from Reason-Augmented Preference Feedback

  • Minjune Hwang, Yigit Korkmaz, Daniel Seita, Erdem Biyik
  • Ключевое слово: Reward Model, Preference

• COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences

  • Yixin Liu, Argyris Oikonomou, Weiqiang Zheng, Yang Cai, Arman Cohan
  • Ключевое слово: RLHF, Preference, Alignment, Nash, Optimization

• Displacement-Resistant Extensions of DPO with Nonconvex $f$-Divergences

  • Idan Pipano, Shoham Sabach, Kavosh Asadi, Mohammad Ghavamzadeh
  • Ключевое слово: DPO, RLHF

• Keep the Best, Forget the Rest: Reliable Alignment with Order-Aware Preference Optimization

  • Jiahui Zhu, Yuanjie Shi, Xiyue Peng, Xin Liu, Yan Yan, Honghao Wei
  • Ключевое слово: DPO, PPO, RLHF, Preference, Alignment

• Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset

  • Lily H Zhang, Smitha Milli, Karen Long Jusko, Jonathan Smith, Brandon Amos, Wassim Bouaziz, Manon Revel, Jack Kussman, Yasha Sheynin, Lisa Titus, Bhaktipriya Radharapu, Jane Yu, Vidya Sarma, Kristopher Rose, Maximilian Nickel
  • Ключевое слово: Preference, Alignment, LLM

• Fair Reinforcement Learning for Just AI

  • Ezgi Korkmaz
  • Ключевое слово: Preference, Alignment, Optimization, Reinforcement Learning, Human Feedback

• Robust Reward Modeling via Causal Rubrics

  • Pragya Srivastava, Harman Singh, Rahul Madhavan, Gandharv Patil, Sravanti Addepalli, Arun Suggala, Rengarajan Aravamudhan, Soumya Sharma, Anirban Laha, Aravindan Raghuveer, Karthikeyan Shanmugam, Doina Precup
  • Ключевое слово: DPO, Reward Model, Alignment, Safety, LLM

• Escaping Policy Contraction: Contraction-Aware PPO (CaPPO) for Stable Language Model Fine-Tuning

  • Dun Yuan, Di Wu, Xue Liu
  • Ключевое слово: PPO, RLHF, Alignment, Optimization, Reinforcement Learning

• Beyond Pairwise: Empowering LLM Alignment With (Ranked) Choice Modeling

  • Yuxuan Tang, Yifan Feng
  • Ключевое слово: DPO, PPO, Preference, Alignment, LLM

• Learning Ordinal Probabilistic Reward from Preferences

  • Longze Chen, Lu Wang, Renke Shan, Ze Gong, Run Luo, Jiaming Li, Jing Luo, Qiyao Wang, Min Yang
  • Ключевое слово: Reward Model, LLM

• Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance

  • Zhuo Li, Pengyu Cheng, Zhechao Yu, FeifeiTong, Anningzhe Gao, Tsung-Hui Chang, Xiang Wan, erchao.zec, xiaoxi jiang, guanjunjiang
  • Ключевое слово: RLHF, Reward Model, Preference, LLM, Optimization

• Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment

  • Mengxuan Hu, Vivek Datla, Anoop Kumar, Zihan Guan, Sheng Li, Alfy Samuel, Daben Liu
  • Ключевое слово: DPO, RLHF, Preference, Alignment, Safety

• Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment

  • Hongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang, Min Zhang
  • Ключевое слово: Reward Model, Preference, Alignment, LLM, Reinforcement Learning

• Balancing the Experts: Unlocking LoRA-MoE for GRPO via Mechanism-Aware Rewards

  • Changlian Ma, Zizheng Huang, Xiangyu Zeng, Yi Wang, Cheng Liang, Kun Tian, Xinhai Zhao, Limin Wang
  • Ключевое слово: Alignment, Multimodal, Optimization, Reinforcement Learning

• Bradley-Terry and Multi-Objective Reward Modeling Are Complementary

  • Zhiwei Zhang, Hui Liu, Xiaomin Li, Zhenwei Dai, Jingying Zeng, Fali Wang, Minhua Lin, Ramraj Chandradevan, Linlin Wu, Zhen Li, Chen Luo, Zongyu Wu, Xianfeng Tang, Qi He, Suhang Wang
  • Ключевое слово: RLHF, Reward Model, Preference, LLM, Reinforcement Learning

• Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework

  • Kihyun Kim, Jiawei Zhang, Asuman E. Ozdaglar, Pablo A. Parrilo
  • Ключевое слово: Preference, Alignment

• ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment

  • Xiaoqiang Lin, Arun Verma, Zhongxiang Dai, Daniela Rus, See-Kiong Ng, Bryan Kian Hsiang Low
  • Ключевое слово: DPO, Reward Model, Preference, Alignment, LLM

• BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models

  • Yuming Li, Yikai Wang, Yuying zhu, Zhongyu Zhao, Ming Lu, Qi She, Shanghang Zhang
  • Ключевое слово: Preference, Alignment, Optimization

• Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimization

  • Tuan Nguyen, Long Tran-Thanh
  • Ключевое слово: Alignment, Safety, LLM, Reinforcement Learning, Human Feedback

• Threshold-Guided Optimization for Visual Generative Models

  • Jinbin Bai, Yu Lei, Qingyu Shi, Aosong Feng, Yi Xin, Zhuoran Zhao, Fei Shen, Kaidong Yu, Xiangtai Li
  • Ключевое слово: Reward Model, Preference, Alignment, Diffusion, Optimization

• Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients

  • Omar Elmansouri, Fathinah Izzati, Mohamed El Amine Seddik, Salem Lahlou
  • Ключевое слово: RLHF, Reward Model, LLM, Optimization, Reinforcement Learning

• Controllable and explainable personality sliders for LLMs at inference time

  • Florian Hoppe, David Khachaturov, Robert Mullins, Mark Huasong Meng
  • Ключевое слово: PPO, RLHF, Alignment, LLM, Optimization

• PS-PPO : Prefix-Sampling PPO for Critic-Free RLHF

  • Doo Hwan Hwang, Kee-Eung Kim
  • Ключевое слово: PPO, RLHF, Optimization, Reinforcement Learning, Human Feedback

• Unbiased Reward Modeling from Implicit Preference

  • Eric Wang, Haocheng Yang, Licheng Pan, Lei Shen, Xiaoxi Li, Yinuo Wang, Zhichao Chen, Yuan Lu, Haoxuan Li, Zhouchen Lin
  • Ключевое слово: RLHF, Reward Model, Preference, Reinforcement Learning, Human Feedback

• How RLHF Amplifies Sycophancy

  • Itai Shapira, Gerdus Benade, Ariel Procaccia
  • Ключевое слово: Preference, Alignment, Optimization, Human Feedback

• Real-Time Aligned Reward Model beyond Semantics

  • Zixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng, Xuefeng Xiao, Hongyan Xie, Huaqiu Li, Songshi Liang, Zhongxiang Dai, Fuzhen Zhuang, Jianxin Li, Yikun Ban, deqing wang
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, LLM

• Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

  • Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Du, Yejin Choi, Tim Althoff, Natasha Jaques
  • Ключевое слово: RLHF, Reward Model, Alignment, Safety, LLM

• B-Spar: Bayesian Sparse-Reward Modeling for RL-based Image Editing

  • shusong xu, Peiye Liu, Yongbin Liu, Bangjie Yin, Zhaomang Sun, Zhenyu Chen, Tianyi Zheng, Peng-Tao Jiang, Jian Zhang, Yuzhao Wang, Jinwei Chen, Zhen Gu, Bo Li
  • Ключевое слово: Reward Model, Alignment, Multimodal, LLM, Optimization

• Calibrated Preference Learning: The Case of Label Ranking

  • Santo Thies, Viktor Bengs, Timo Kaufmann, Sebastian Vollmer, Eyke Hüllermeier
  • Ключевое слово: RLHF, Reward Model, Alignment

• Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO

  • Ruizhe Shi, Minhak Song, Runlong Zhou, Zihan Zhang, Maryam Fazel, Simon Du
  • Ключевое слово: DPO, RLHF, Reward Model, Preference, Optimization

• DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding

  • mingxi Zou, Jiaxiang Chen, Junfan Li, Langzhang Liang, Qifan Wang, Xu Yinghui, Zenglin Xu
  • Ключевое слово: DPO, RLHF, Preference, Alignment, Optimization

• The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs

  • Pengrui Han, Rafal Kocielnik, Peiyang Song, Ramit Debnath, Dean Mobbs, Anima Anandkumar, R. Michael Alvarez
  • Ключевое слово: RLHF, Alignment, LLM

• Distributionally Robust Reinforcement Learning with Human Feedback

  • Debmalya Mandal, Paulius Sasnauskas, Goran Radanovic
  • Ключевое слово: DPO, RLHF, Reward Model, Preference, LLM

• Automatically Finding Reward Model Biases

  • Atticus Wang, Iván Arcuschin, Arthur Conmy
  • Ключевое слово: Reward Model, LLM, Reinforcement Learning, Human Feedback

• Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

  • Zichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang, Le Sun, Yaojie Lu, XingYu
  • Ключевое слово: RLHF, LLM, Optimization, Reinforcement Learning

• Convex Optimization for Alignment and Preference Learning on a Single GPU

  • Miria Feng, Mert Pilanci
  • Ключевое слово: DPO, RLHF, Preference, Alignment, LLM

• MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge

  • Baochen Fu, Yuntao Du, Cheng Chang, Baihao Jin, Wenzhi Deng, Muhao Xu, Hongmei Yan, Weiye Song, Yi Wan
  • Ключевое слово: RLHF, Multimodal, Reinforcement Learning, Human Feedback

• Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization

  • Yihang Yao, Zhepeng Cen, Haohong Lin, Shiqi Liu, Zuxin Liu, Jiacheng Zhu, Zhang-Wei Hong, Laixi Shi, Ding Zhao
  • Ключевое слово: LLM, Reinforcement Learning, Human Feedback

• Unbiased Alignment for Large Language Models with Noisy Preferences

  • Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu, Haoliang Li
  • Ключевое слово: DPO, Reward Model, Preference, Alignment, Optimization

• Unbiased Principles, Robust Rewards

  • Qingnan Ren, Zhen Fang, Shiting Huang, Yu Zeng, Lin Chen, Zehui Chen, Feng Zhao
  • Ключевое слово: RLHF, Reward Model, Reinforcement Learning, Human Feedback

• The Secret Engine Behind RLHF: It’s Contarstive Learning All Along

  • Xufei Lv, Kehai Chen, Haoyuan Sun, Xuefeng Bai, Min zhang, Houde Liu
  • Ключевое слово: DPO, RLHF, Preference, Alignment, LLM

• When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models

  • Tong Xie, Ching-Yuan Bai, Yuanhao Ban, Yunqi Hong, Haoyu Li, Cho-Jui Hsieh
  • Ключевое слово: RLHF, Reward Model, Alignment, LLM

• Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models

  • Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen
  • Ключевое слово: DPO, RLHF, Preference, Alignment, Safety

• TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

  • Abdulhady abas, Fatemeh Daneshfar, Seyedali Mirjalili, Mourad Oussalah
  • Ключевое слово: DPO, PPO, RLHF, Preference, Multimodal

• Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling

  • Yang Cai, Weiqiang Zheng
  • Ключевое слово: PPO, Preference, Alignment, LLM, Nash

• Reward Modeling from Natural Language Human Feedback

  • Zongqi Wang, Rui Wang, Yuchuan Wu, Yiyao Yu, Pinyi Zhang, Shaoning Sun, Yujiu Yang, Yongbin Li
  • Ключевое слово: Reward Model, Preference, Reinforcement Learning, Human Feedback

• Efficient Preference Poisoning Attack on Offline RLHF

  • Chenye Yang, Weiyu Xu, Lifeng Lai
  • Ключевое слово: DPO, RLHF, Preference, Optimization, Reinforcement Learning

• Position: Agentic Safety is an Epistemic Property, Not a Behavioral One

  • Charles Wang, Keir Dorchen, Peter Jin
  • Ключевое слово: RLHF, Preference, Alignment, Safety, Optimization

• Position: Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences

  • Cristina Garbacea
  • Ключевое слово: RLHF, Reward Model, Preference, Safety, Reinforcement Learning

• Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

  • Yucong Huang, Xiucheng Li, Kaiqi Zhao, Jing Li
  • Ключевое слово: PPO, RLHF, Preference, Alignment, Nash

• Factored Causal Representation Learning for Robust Reward Modeling in RLHF

  • Yupei Yang, Lin Yang, Wanxi Deng, Lin Qu, Fan Feng, Biwei Huang, Shikui Tu, Lei Xu
  • Ключевое слово: RLHF, Reward Model, Preference, LLM, Reinforcement Learning

• Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences

  • Jabin Koo, Hoyoung Kim, Minwoo Jang, Jungseul Ok
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, LLM

• Online Compatible Reward Identification from Preference Feedback

  • Simone Drago, Marco Mussi, Alberto Maria Metelli
  • Ключевое слово: Preference, Safety, Reinforcement Learning, Human Feedback

• $f$-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses

  • Di Wu, Chengshuai Shi, Jing Yang, Cong Shen
  • Ключевое слово: RLHF, Reinforcement Learning, Human Feedback

• Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases

  • Dongyoon Hahm, Dylan Hadfield-Menell, Kimin Lee
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, LLM

• IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models

  • Haonan Song, Qingchen Xie, Huan Zhu, Feng Xiao, Luxi Xing, Liu Kang, Fuzhen Li, Zhiyong Zheng, Feng Jiang, Ziheng Li, Kun Yan, Qingyi Si, Yanghua Xiao, Hongcheng Guo, Fan Yang
  • Ключевое слово: RLHF, Reward Model, Preference, Reinforcement Learning, Human Feedback

• Implicit Preference Alignment for Human Image Animation

  • Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng, bing ma, Kai Yu, Tianxiang Zheng, Qinglin Lu, Zhen Cui
  • Ключевое слово: Preference, Alignment, Optimization, Reinforcement Learning, Human Feedback

• Multilingual Safety Alignment Via Sparse Weight Editing

  • Jiaming Liang, Zhaoxin Wang, Handing Wang
  • Ключевое слово: RLHF, Alignment, Safety, LLM, Reinforcement Learning

• Gradient Regularization Prevents Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

  • Johannes Ackermann, Michael Noukhovitch, Takashi Ishida, Masashi Sugiyama
  • Ключевое слово: RLHF, Reward Model, LLM, Reinforcement Learning, Human Feedback

• Graph-Preference Learning: Debiasing Network-Sampled Human Feedback for Target Welfare Estimation

  • Guangrui Fan, DanDan Liu, AZNUL SABRI, Pan Lihu
  • Ключевое слово: DPO, RLHF, Reward Model, Preference

• COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space

  • Yao Luan, Ni Mu, Hanfei Ge, Yiqin Yang, Bo XU, Qing-Shan Jia
  • Ключевое слово: Human Feedback

• Optimal Transport for Reward Modeling from Noisy Feedback

  • Eric Wang, Licheng Pan, Haocheng Yang, Yunsheng Lu, Yongqi Tong, Yinuo Wang, Shijian Wang, Zhixuan Chu, Lei Shen, Haoxuan Li, Yuan Lu
  • Ключевое слово: RLHF, Reward Model, Preference, Reinforcement Learning, Human Feedback

• Reliability-Aware LLM Alignment from Inconsistent Human Feedback

  • Jingyi Huang, Ruohan Zong, Yujun Feng, Liran Ma, Lanyu Shang, Yang Zhang
  • Ключевое слово: DPO, RLHF, Preference, Alignment, LLM

• Position: We Need Large Language Models Optimized For Our Well-Being

  • Ashton Anderson, Harsh Kumar, Louis Tay, Karina Vold
  • Ключевое слово: RLHF, Preference, Optimization, Reinforcement Learning, Human Feedback

• Implicit Safety Alignment from Crowd Preferences

  • Qian Lin, Daniel S Brown
  • Ключевое слово: RLHF, Reward Model, Preference, Safety, Reinforcement Learning

• Contrastive Weak-to-Strong Generalization

  • Houcheng Jiang, Junfeng Fang, Jiaxin Wu, Tianyu Zhang, Chen Gao, Xiang Wang, Xiangnan He, Yang Deng
  • Ключевое слово: Reward Model, Alignment, LLM, Human Feedback

• Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

  • Kazusato Oko, Annie Ulichney, Nika Haghtalab, Han Bao
  • Ключевое слово: RLHF, Preference, Reinforcement Learning, Human Feedback

• Conversation for Non-verifiable Learning: Self-Evolving Large Language Models through Meta-Evaluation

  • Yuan Sui, Bryan Hooi
  • Ключевое слово: LLM, Optimization, Human Feedback

• Unifying Adversarial Robustness and Training Across Text Scoring Models

  • Manveer Tamber, Hosna Oyarhoseini, Jimmy Lin
  • Ключевое слово: PPO, RLHF, Reward Model, LLM

• ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

  • Davit Melikidze, Marian Schneider, Jessica Lam, Martin Wertich, Ido Hakimi, Barna Pasztor, Andreas Krause
  • Ключевое слово: RLHF, Preference, Alignment, LLM, Reinforcement Learning

• Layer-wise Gradient Disentanglement: Decoupling Semantics and Preferences in Direct Preference Optimization

  • Mengyang Li, Shuang Liu, Zhong Zhang
  • Ключевое слово: DPO, RLHF, Preference, Optimization

• The Sign Estimator: Preference Modeling for LLM Alignment under Heterogeneity

  • Aymane El Gadarri, Ali Aouad, Vivek Farias
  • Ключевое слово: RLHF, Reward Model, Preference, Alignment, LLM

• Leveraging Machine Unlearning for Cost-Efficient Preference Alignment

  • XiaoHua Feng, Yuyuan Li, HuWei Ji, Li Zhang, Jiaming Zhang, Tianyu Du, Chaochao Chen
  • Ключевое слово: Preference, Alignment, LLM, Optimization, Reinforcement Learning

• Regularization in the Axiomatic Approach to Learning from Human Preferences

  • Ezgi Korkmaz
  • Ключевое слово: RLHF, Preference, Reinforcement Learning, Human Feedback

• DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and Its Loss’ Convexity is Dispensable)

  • Wenxuan Zhou, Shujian Zhang, brice magdalou, John Lambert, Ehsan Amid, Richard Nock, Andrew Hard
  • Ключевое слово: DPO, PPO, RLHF, Reward Model, Preference

• Conditional Equivalence of DPO and RLHF: Assumptions, Failure Modes, and Provable Alignment

  • Yonggang Zhang, Zhiqin Yang, Wei Xue, Dong Fang, Bo Han, Yike Guo
  • Ключевое слово: DPO, RLHF, Preference, Alignment, Optimization

• A Regret Minimization Framework on Preference Learning in Large Language Models

  • Suhwan Kim, Taehyun Cho, Youngsoo Jang, Geon-Hyeong Kim, Yu Jin Kim, Moontae Lee, Jungwoo Lee
  • Ключевое слово: RLHF, Preference, Optimization, Reinforcement Learning, Human Feedback

• Position: Measuring Human Preferences in RLHF is a Social Science Problem

  • Bijean Ghafouri, Eun Cheol Choi, Priyanka Dey, Emilio Ferrara
  • Ключевое слово: RLHF, Preference, Alignment

• Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling

  • Zhibin Duan, Guowei Rong, Zhuo Li, Bo Chen, Mingyuan Zhou, Dandan Guo
  • Ключевое слово: Reward Model, Preference, LLM, Optimization, Reinforcement Learning

2025

• Position: The Complexity of Perfect AI Alignment — Formalizing the RLHF Trilemma

  • Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary
  • Ключевое слово: Alignment Bias, Safety, Interpretability

• What’s In My Human Feedback? Learning Interpretable Descriptions of Preference Data

  • Rajiv Movva, Smitha Milli, Sewon Min, Emma Pierson
  • Ключевое слово: Sparse Autoencoders, Interpretable Data Curation, Reward Hacking, Feature Attribution
  • Код: Official

• Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback

  • Gen Li, Yuling Yan
  • Ключевое слово: Online RL, Multi-armed Bandit, LLMs

• OpenRLHF: A Ray-based Easy-to-use, Scalable and High-performance RLHF Framework

  • Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Wenkai Fang, Xianyu, Yu Cao, Haotian Xu, Yiming Liu
  • Ключевое слово: Framework
  • Код: Official

• Language Models Learn to Mislead Humans via RLHF

  • Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Sam Bowman, He He, Shi Feng
  • Ключевое слово: Open-ended Task, Human Reward, Alignment Method, LLMs

• A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning

  • Shashank Gupta, Chaitanya Ahuja, Tsung-Yu Lin, Sreya Dutta Roy, Harrie Oosterhuis, Maarten de Rijke, and Satya Narayan Shukla
  • Ключевое слово: Diffusion Model, REINFORCE, PPO

• Differential Information: An Information-Theoretic Perspective on Preference Optimization

  • Yunjae Won, Hyunji Lee, Hyeonbin Hwang, Minjoon Seo
  • Ключевое слово: Preference Optimization, Information-Theoretic Analysis, Log-Ratio Reward Parameterization, Data Distribution, Log-Likelihood Displacement

• Generalist Reward Models: Found Inside Large Language Models

  • Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, Zhi-Hua Zhou
  • Ключевое слово: Offline Inverse RL, LLM-as-a-judge, Training-free, Alignment

• A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization

  • Wenyuan Xu, Xiaochen Zuo, Chao Xin, Yu Yue, Lin Yan, Yonghui Wu
  • Ключевое слово: Generative Pairwise Reward Model, Policy Optimization, Framework

• Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback

  • Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue, Lin Yan
  • Ключевое слово: Data Scaling, Reward Hacking, LLMs

• RLTHF: Targeted Human Feedback for LLM Alignment

  • Yifei Xu, Tusher Chakraborty, Emre Kıcıman, Bibek Aryal, Eduardo Rodrigues, Srinagesh Sharma, Roberto Estevao, Maria Angels de Luis Balaguer, Jessica Wolk, Rafael Padilha, Leonardo Nunes, Shobana Balakrishnan, Songwu Lu, Ranveer Chandra
  • Ключевое слово: Human-AI Hybrid Framework, Efficient, Alignment, LLMs

• Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models

  • Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, Bo Zheng
  • Ключевое слово: Safety, Framework, Adaptive Message-wise Alignment Method, LLMs

• MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

  • Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, Xue Wang, Yibo Hu, Bin Wen, Fan Yang, Zhang Zhang, Tingting Gao, Di Zhang, Liang Wang, Rong Jin, Tieniu Tan
  • Ключевое слово: Critique-based Reward Model, Dynamic Reward, Dataset
  • Код: Official

• Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback

  • Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li, Yu Cheng
  • Ключевое слово: Test-Time Optimization, Preference Learning, Iterative Feedback
  • Код: Official

• Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model

  • Yueqin Yin, Shentao Yang, Yujia Xie, Ziyi Yang, Yuting Sun, Hany Awadalla, Weizhu Chen, and Mingyuan Zhou
  • Ключевое слово: Segment-level Reward Model, Dense Reward RLHF Framework, Improved PPO training for LLMs
  • Код: Official

• REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models

  • Jian Hu
  • Ключевое слово: Efficient, Alignment, Reinforcement Learning
  • Код: Official

2024

• DPO Meets PPO: Reinforced Token Optimization for RLHF

  • Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, Liwei Wang
  • Ключевое слово: Token-wise Reward, DPO, PPO, RLHF
  • Код: Official

• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs

  • Shenao Zhang, Zhihan Liu, Boyi Liu, Yufeng Zhang, Yingxiang Yang, Yongfei Liu, Liyu Chen, Tao Sun, Zhaoran Wang
  • Ключевое слово: Reward-Augmented Data, DPO, LLMs
  • Код: Official

• The Accuracy Paradox in RLHF: When Better Reward Models Don’t Yield Better Language Models

  • Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, Xiaoyu Shen
  • Ключевое слово: Reward Model Evaluation, Accuracy Paradox, LLM Alignment
  • Код: Official

• Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

  • Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li, Weipeng Chen, Jun Song, Bo Zheng, Yaodong Yang
  • Ключевое слово: Multi-modality Alignment, Dataset, Training-evaluation Framework
  • Код: Official

• REvolve: Reward Evolution with Large Language Models using Human Feedback

  • Rishi Hazra, Alkis Sygkounas, Andreas Persson, Amy Loutfi, Pedro Zuidberg Dos Martires
  • Ключевое слово: Improved Reward Model with LLMs, Framework
  • Код: Official

• Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

  • Qining Zhang, Lei Ying
  • Ключевое слово: Reward inference-free RLHF, Zeroth-order optimization, Policy gradient

• Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment

  • Chenliang Li, Siliang Zeng, Zeyi Liao, Jiaxiang Li, Dongyeop Kang, Alfredo Garcia, Mingyi Hong
  • Ключевое слово: Joint Reward and Policy, Efficiency, Framework

• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions

  • Yekun Chai, Haoran Sun, Huang Fang, Shuohuan Wang, Yu Sun, Hua Wu
  • Ключевое слово: Macro action-level Reward, Efficiency, Framework
  • Код: Official

• Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

  • Shang Liu, Yu Pan, Guanting Chen, and Xiaocheng Li
  • Ключевое слово: Reward Modeling, Ordinal Feedback, Human Preference Dataset
  • Код: Official

• Aligning Few-Step Diffusion Models with Dense Reward Difference Learning

  • Ziyi Zhang, Li Shen, Sen Zhang, Deheng Ye, Yong Luo, Miaojing Shi, Bo Du, Dacheng Tao
  • Ключевое слово: Diffusion Models, Text-to-Image, Alignment, Reinforcement Learning
  • Код: Official

• HybridFlow: A Flexible and Efficient RLHF Framework

  • Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu
  • Ключевое слово: Flexible, Efficient, RLHF framework
  • Код: Official

• ALaRM: Align Language Models via Hierarchical Rewards Modeling

  • Yuhang Lai, Siyuan Wang, Shujun Liu, Xuanjing Huang, Zhongyu Wei
  • Ключевое слово: Hierarchical Reward, Open Text Generation Tasks
  • Код: Official

• TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback

  • Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Wontae Nam, Daejin Jo, Kyoung-Woon On, Mark A. Hasegawa-Johnson, Sungwoong Kim, Chang D. Yoo
  • Ключевое слово: Token-Level Continuous Reward, RLHF
  • Код: Official

• Aligning Large Multimodal Models with Factually Augmented RLHF

  • Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell
  • Ключевое слово: Factually Augmented RLHF, Vision & Language, Human Preference Dataset
  • Код: Official

• Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation

  • Aiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong, Simon Wang, Jiulong Shan, Meng Cao, Lijie Wen
  • Ключевое слово: Without Human Preference Data, Self-Reward, DPO
  • Код: Official

• Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards

  • Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, Tong Zhang
  • Ключевое слово: User Preference, Multi-objective Reward Model, Rejection Sampling Finetuning
  • Код: Official

• Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

  • Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker
  • Ключевое слово: Online RL Optimization, Low Computational Cost
  • Код: Official

• Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint

  • Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, Ji-Rong Wen
  • Ключевое слово: Token-level Reward, LLM
  • Код: Official

• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

  • Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash
  • Ключевое слово: RL from AI Feedback
  • Код: official

• Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF

  • Han Shen, Zhuoran Yang, Tianyi Chen
  • Ключевое слово: Bilevel optimization
  • Код: official

• Dense Reward for Free in Reinforcement Learning from Human Feedback

  • Alex James Chan, Hao Sun, Samuel Holt, Mihaela Van Der Schaar
  • Ключевое слово: reward shaping, RLHF
  • Код: official

• A Minimaximalist Approach to Reinforcement Learning from Human Feedback

  • Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, Alekh Agarwal
  • Ключевое слово: Minimax Winner, Self-Play Preference Optimization
  • Код: official

• Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

  • Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, Tat-Seng Chua
  • Ключевое слово: Multimodal Large Language Models, Hallucination Problem, Reinforcement Learning from Human Feedback
  • Код: official

• RLHF Workflow: From Reward Modeling to Online RLHF

  • Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, Tong Zhang
  • Ключевое слово: Online Iterative RLHF, Preference Modeling, Large Language Models
  • Код: official

• MaxMin-RLHF: Towards equitable alignment of large language models with diverse human preferences

  • Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, Mengdi Wang
  • Ключевое слово: mixture of preference distributions, MaxMin alignment objective
  • Код: official

• Dataset Reset Policy Optimization for RLHF

  • Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen Sun
  • Ключевое слово: Dataset Reset Policy Optimization
  • Код: official

• A Dense Reward View on Aligning Text-to-Image Diffusion with Preference

  • Shentao Yang, Tianqi Chen, Mingyuan Zhou
  • Ключевое слово: RLHF for Text-to-Image Generation, Dense Reward Improvement of DPO, Efficient Alignment
  • Код: official

• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

  • Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu
  • Ключевое слово: Self-Play Fine-Tuning
  • Код: official

• RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

  • Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, Bruno Castro da Silva
  • Ключевое слово: RLHF, Oracular Reward, Reward Model Analysis, Survey

• Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases

  • Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, Dacheng Tao
  • Ключевое слово: Diffusion Models, Alignment, Reinforcement Learning, RLHF, Reward Overoptimization, Primacy Bias
  • Код: official

• On Diversified Preferences of Large Language Model Alignment

  • Dun Zeng, Yong Dai, Pengyu Cheng, Tianhao Hu, Wanshun Chen, Nan Du, Zenglin Xu
  • Ключевое слово: Aligning shared preference, Reward modeling metrics, LLM
  • Код: official

• Aligning Crowd Feedback via Distributional Preference Reward Modeling

  • Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, Yong Liu
  • Ключевое слово: RLHF, Preference distribution, Aligning, LLM

• Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization

  • Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, Yu Qiao
  • Ключевое слово: Multi-objective RLHF without reward modeling, DPO
  • Код: official

• Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!

  • Zhanhui Zhou, Jie Liu, Zhichen Dong, Jiaheng Liu, Chao Yang, Wanli Ouyang, Yu Qiao
  • Ключевое слово: LLM inference-time attack, DPO, Producing harmful LLMs without training
  • Код: official

• A Theoretical Analysis of Nash Learning from Human Feedback under General KL-Regularized Preference

  • Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, Tong Zhang
  • Ключевое слово: Game-based RLHF, Nash Learning, Alignment under reward-model-free oracle

• Mitigating the Alignment Tax of RLHF

  • Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, Tong Zhang
  • Ключевое слово: RLHF, Alignment tax, Catastrophic forgetting

• Training Diffusion Models with Reinforcement Learning

  • Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, Sergey Levine
  • Ключевое слово: reinforcement learning, RLHF, diffusion models
  • Код: official

• AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model

  • Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng,Yujing Hu, Tangjie Lv, Changjie Fan, Zhipeng Hu
  • Ключевое слово: Reinforcement learning; Diffusion models; RLHF; Preference aligning
  • Код: official

• Dense Reward for Free in Reinforcement Learning from Human Feedback

  • Alex J. Chan, Hao Sun, Samuel Holt, Mihaela van der Schaar
  • Ключевое слово: RLHF
  • Код: official

• Transforming and Combining Rewards for Aligning Large Language Models

  • Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D’Amour, Sanmi Koyejo, Victor Veitch
  • Ключевое слово: RLHF, Aligning, LLM

• Parameter Efficient Reinforcement Learning from Human Feedback

  • Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, Lucas Dixon
  • Ключевые слова: RLHF, Parameter Efficient method, Low Computational Cost, LLM, VLM

• Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

  • Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, Chuang Gan
  • Ключевые слова: RLHF, Reward Ensemble, Efficient Ensemble Method

• RIME: Robust Preference-based Reinforcement Learning with Noisy Human Preferences

  • Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, Fei-Yue Wang
  • Ключевое слово:
  • Код: official

• Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback

  • Yifu Yuan, Jianye Hao, Yi Ma, Zibin Dong, Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, Yan Zheng
  • Ключевое слово:
  • Код: official
  • Набор данных: official

2023

• The Trickle-down Impact of Reward (In-)consistency on RLHF

  • Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, Dong Yu
  • Ключевое слово: Reward model, RLHF, Reward hacking
  • Код: official

• A General Theoretical Paradigm to Understand Learning from Human Preferences

  • Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, Rémi Munos
  • Ключевые слова: RLHF, Pairwise Preference

• Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

  • Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, Hannaneh Hajishirzi
  • Ключевое слово: RLHF, Sentence-level Reward, LLM
  • Код: official

• Preference-grounded Token-level Guidance for Language Model Fine-tuning

  • Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, Mingyuan Zhou
  • Ключевое слово: RLHF, Token-level Training Guidance, Alternate/Online Training Framework, Minimalist Training Objectives
  • Код: official

• Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems

  • Yihao Feng*, Shentao Yang*, Shujian Zhang, Jianguo Zhang, Caiming Xiong, Mingyuan Zhou, Huan Wang
  • Ключевое слово: RLHF, Genralized Reward Function Learning, Reward Function Utilization, Task-oriented Dialogue System, Learning-to-rank
  • Код: official

• Inverse Preference Learning: Preference-based RL without a Reward Function

  • Joey Hejna, Dorsa Sadigh
  • Ключевое слово: Inverse Preference Learning, without reward model
  • Код: official

• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

  • Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S. Liang, Tatsunori B. Hashimoto
  • Ключевое слово: RLHF, Simulation Framework
  • Код: official

• Adversarial Preference Optimization

  • Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, Nan Du
  • Ключевое слово: RLHF, GAN, Adversarial Games
  • Код: official

• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

  • Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, Tong Zhang
  • Ключевое слово: RLHF, Iterative DPO, Mathematical foundation

• Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration

  • Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, Willie Neiswanger
  • Ключевое слово: RLHF, sample efficience, exploration

• Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing

  • Feiyang Han, Yimin Wei, Zhaofeng Liu, Yanxing Qi
  • Ключевое слово: RLHF, AB testing, RLSF

• A Baseline Analysis of Reward Models’ Ability To Accurately Analyze Foundation Models Under Distribution Shift

  • Ben Pikus, Will LeVine, Tony Chen, Sean Hendryx
  • Ключевое слово: RLHF, OOD, Distribution Shift

• Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

  • Di Jin, Shikib Mehri, Devamanyu Hazarika, Aishwarya Padmakumar, Sungjin Lee, Yang Liu, Mahdi Namazifar
  • Ключевое слово: RLHF, data-efficient, Alignment

• Let’s Reinforce Step by Step

  • Sarah Pan, Vladislav Lialin, Sherin Muckatira, Anna Rumshisky
  • Ключевое слово: RLHF, reasoning

• Direct Preference-based Policy Optimization without Reward Modeling

  • Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, Hyun Oh Song
  • Ключевое слово: RLHF without reward modeling, Contrastive learning, Offline refinforcement learning

• AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model

  • Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv, Changjie Fan, Zhipeng Hu
  • Ключевое слово: RLHF, Alignment, Diffusion model

• Eureka: Human-Level Reward Design via Coding Large Language Models

  • Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, Anima Anandkumar
  • Ключевое слово: LLM based, reward functions design

• Safe RLHF: Safe Reinforcement Learning from Human Feedback

  • Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, Yaodong Yang
  • Ключевое слово: Sale RL, LLM fine-ture

• Quality Diversity through Human Feedback

  • Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, Joel Lehman
  • Ключевое слово: Quality Diversity, Diffusion model

• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

  • Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, Zhi-Quan Luo
  • Ключевое слово: computational efficiency, variance-reduction technique

• Tuning computer vision models with task rewards

  • André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, Xiaohua Zhai
  • Ключевое слово: Reward tuning in Computer Vision

• The Wisdom of Hindsight Makes Language Models Better Instruction Followers

  • Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, Joseph E. Gonzalez
  • Ключевое слово: Hindsight Instruction Relabeling, RLHF System, No Value Network Required
  • Код: official

• Language Instructed Reinforcement Learning for Human-AI Coordination

  • Hengyuan Hu, Dorsa Sadigh
  • Ключевое слово: Human-AI coordination, Human preference alignment, Instruction conditioned RL

• Aligning Language Models with Offline Reinforcement Learning from Human Feedback

  • Jian Hu, Li Tao, June Yang, Chandler Zhou
  • Ключевое слово: Decision Transformer-based Alignment, Offline Reinforcement Learning, RLHF System

• Preference Ranking Optimization for Human Alignment

  • Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li and Houfeng Wang
  • Ключевое слово: Supervised Human Preference Alignment, Preference Ranking Extension
  • Код: official

• Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation

  • Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, André F. T. Martins
  • Ключевое слово: Natural Language Generation, Human Feedback Integration, Feedback Formalization and Taxonomy, AI Feedback and Principles-Based Judgments

• GPT-4 Technical Report

  • OpenAI
  • Ключевое слово: A large-scale, multimodal model, Transformerbased model, Fine-tuned used RLHF
  • Код: official
  • Набор данных: DROP, WinoGrande, HellaSwag, ARC, HumanEval, GSM8K, MMLU, TruthfulQA

• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

  • Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang
  • Ключевое слово: Rejection Sampling Finetuning, Alternative to PPO, Diffusion Model
  • Код: official

• RRHF: Rank Responses to Align Language Models with Human Feedback without tears

  • Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang
  • Ключевое слово: New paradigm for RLHF
  • Код: official

• Few-shot Preference Learning for Human-in-the-Loop RL

  • Joey Hejna, Dorsa Sadigh
  • Ключевое слово: Preference Learning, Interactive Learning, Multi-task Learning, Expanding the pool of available data by viewing human-in-the-loop RL
  • Код: official

• Better Aligning Text-to-Image Models with Human Preference

  • Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, Hongsheng Li
  • Ключевое слово: Diffusion Model, Text-to-Image, Aesthetic
  • Код: official

• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation

  • Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, Yuxiao Dong
  • Ключевое слово: General-purpose text-to-Image human preference RM, Evaluating Text-to-Image Generative Models
  • Код: official
  • Набор данных: COCO, DiffusionDB

• Aligning Text-to-Image Models using Human Feedback

  • Kimin Lee, Hao liu, MoonKyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Shixiang Shane Gu
  • Ключевое слово: Text-to-Image, Stable diffusion model, Reward function that predicts human feedback

• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

  • Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, Nan Duan
  • Ключевое слово: Visual Foundation Models, Visual ChatGPT
  • Код: official

• Pretraining Language Models with Human Preferences (PHF)

  • Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, Ethan Perez
  • Ключевое слово: Pretraining, offline RL, Decision transformer
  • Код: official

• Aligning Language Models with Preferences through f-divergence Minimization (f-DPG)

  • Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, Marc Dymetman
  • Ключевое слово: f-divergence, RL with KL penalties

• Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons

  • Banghua Zhu, Jiantao Jiao, Michael I. Jordan
  • Ключевое слово: Pessimistic MLE, Max-entropy IRL

• The Capacity for Moral Self-Correction in Large Language Models

  • Anthropic
  • Ключевое слово: Improve moral self-correction capability by increasing RLHF training
  • Dataset; BBQ

2022

• Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (NLPO)
  • Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté,Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, Yejin Choi
  • Ключевое слово: Optimizing language generators with RL, Benchmark, Performant RL algorithm
  • Код: official
  • Набор данных: IMDB, CommonGen, CNN Daily Mail, ToTTo, WMT-16 (en-de),NarrativeQA, DailyDialog
• Scaling Laws for Reward Model Overoptimization
  • Leo Gao, John Schulman, Jacob Hilton
  • Ключевое слово: Gold reward model train proxy reward model, Dataset size, Policy parameter size, BoN, PPO
• Improving alignment of dialogue agents via targeted human judgements (Sparrow)
  • Amelia Glaese, Nat McAleese, Maja Trębacz, et al.
  • Ключевое слово: Information-seeking dialogue agent, Break down the good dialogue into natural language rules, DPC, Interact with the model to elicit violation of a specific rule (Adversarial Probing)
  • Набор данных: Natural Questions, ELI5, QuALITY, TriviaQA, WinoBias, BBQ
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
  • Deep Ganguli, Liane Lovitt, Jackson Kernion, et al.
  • Ключевое слово: Red team language model, Investigate scaling behaviors, Read teaming Dataset
  • Код: official
• Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning
  • Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg, Avinatan Hassidim, Michael Fink, Yossi Matias, Idan Szpektor, Craig Boutilier, Gal Elidan
  • Ключевое слово: Real-time, Open-ended dialogue system, Pairs the succinct embedding of the conversation state by language models, CAQL, CQL, BERT
• Quark: Controllable Text Generation with Reinforced Unlearning
  • Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, Yejin Choi
  • Ключевое слово: Fine-tuning the language model on signals of what not to do, Decision Transformer, LLM tuning with PPO
  • Код: official
  • Набор данных: WRITINGPROMPTS, SST-2, WIKITEXT-103
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  • Yuntao Bai, Andy Jones, Kamal Ndousse, et al.
  • Ключевое слово: Harmless assistants, Online mode, Robustness of RLHF training, OOD detection.
  • Код: official
  • Набор данных: TriviaQA, HellaSwag, ARC, OpenBookQA, LAMBADA, HumanEval, MMLU, TruthfulQA
• Teaching language models to support answers with verified quotes (GopherCite)
  • Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, Nat McAleese
  • Ключевое слово: Generate answers which citing specific evidence, Abstain from answering when unsure
  • Набор данных: Natural Questions, ELI5, QuALITY, TruthfulQA
• Training language models to follow instructions with human feedback (InstructGPT)
  • Long Ouyang, Jeff Wu, Xu Jiang, et al.
  • Ключевое слово: Large Language Model, Align Language Model with Human Intent
  • Код: official
  • Набор данных: TruthfulQA, RealToxicityPrompts
• Constitutional AI: Harmlessness from AI Feedback
  • Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, et al.
  • Ключевое слово: RL from AI feedback(RLAIF), Training a harmless AI assistant through selfimprovement, Chain-of-thought style, Control AI behavior more precisely
  • Код: official
• Discovering Language Model Behaviors with Model-Written Evaluations
  • Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, et al.
  • Ключевое слово: Automatically generate evaluations with LMs, More RLHF makes LMs worse, LM-written evaluations are highquality
  • Код: official
  • Набор данных: BBQ, Winogender Schemas
• Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning
  • Joseph Early, Tom Bewley, Christine Evers, Sarvapali Ramchurn
  • Ключевое слово: Reward Modelling (RLHF), Non-Markovian, Multiple Instance Learning, Interpretability
  • Код: official
• SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning
  • Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, Kimin Lee
  • Ключевое слово: Semi-supervised Reward Learning, Preference Data Augmentation, RLHF Efficiency
• Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
  • Xinran Liang, Katherine Shu, Kimin Lee, Pieter Abbeel
  • Ключевое слово: Preference-based RL (PbRL), Exploration, Reward Uncertainty, Feedback Efficiency
  • Код: official

2021

• WebGPT: Browser-assisted question-answering with human feedback (WebGPT)
  • Reiichiro Nakano, Jacob Hilton, Suchir Balaji, et al.
  • Ключевое слово: Model search the web and provide reference， Imitation learning， BC, long form question
  • Набор данных: ELI5, TriviaQA, TruthfulQA
• Recursively Summarizing Books with Human Feedback
  • Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, Paul Christiano
  • Ключевое слово: Model trained on small task to assist human evaluate broader task, BC
  • Набор данных: Booksum, NarrativeQA
• Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation
  • Samuel Kiegeland, Julia Kreutzer
  • Ключевое слово: The success of policy gradient is because of reward rather than the shape of output distribution, Machine Translation, NMT, DOmain Adaption
  • Код: official
  • Набор данных: WMT15, IWSLT14
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
  • Kimin Lee, Laura Smith, Pieter Abbeel
  • Ключевое слово: Preference-based RL (PbRL), Data Efficiency, Unsupervised Pretraining, Reward Relabeling
  • Код: official
• B-Pref: Benchmarking Preference-Based Reinforcement Learning
  • Kimin Lee, Laura Smith, Anca Dragan, Pieter Abbeel
  • Ключевое слово: Benchmark, Preference-based RL, Simulated Human Feedback, Robustness Evaluation
  • Код: official

2020 и ранее

• Learning to summarize from human feedback
  • Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano
  • Ключевое слово: Care about summary quality, Training loss affect the model behavior, Reward model generalizes to new datasets
  • Код: official
  • Набор данных: TL;DR, CNN/DM
• Fine-Tuning Language Models from Human Preferences
  • Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving
  • Ключевое слово: Reward learning for language, Continuing text with positive sentiment, Summary task, Physical descriptive
  • Код: official
  • Набор данных: TL;DR, CNN/DM
• Scalable agent alignment via reward modeling: a research direction
  • Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg
  • Ключевое слово: Agent alignment problem, Learn reward from interaction, Optimize reward with RL, Recursive reward modeling
  • Код: official
  • Среда: Atari
• Reward learning from human preferences and demonstrations in Atari
  • Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, Dario Amodei
  • Ключевое слово: Expert demonstration trajectory preferences reward hacking problem, Noise in human label
  • Код: official
  • Среда: Atari
• Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces
  • Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, Peter Stone
  • Ключевое слово: High dimension state, Leverage the input of Human trainer
  • Код: third party
  • Среда: Atari
• Deep reinforcement learning from human preferences
  • Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei
  • Ключевое слово: Explore goal defined in human preferences between pairs of trajectories segmentation, Learn more complex thing than human feedback
  • Код: official
  • Среда: Atari, MuJoCo
• Interactive Learning from Policy-Dependent Human Feedback
  • James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David Roberts, Matthew E. Taylor, Michael L. Littman
  • Ключевое слово: Decision is influenced by current policy rather than human feedback, Learn from policy dependent feedback that converges to a local optimal

Кодовые базы

format:
- [title](codebase link) [links]
  - author1, author2, and author3...
  - keyword
  - experiment environments, datasets or tasks

• Reinforcement Learning from Human Feedback (RLHF) in Notebooks
  • Ashwani Kumar
  • step-by-step, Video tutorial, Jupyter notebooks, GPT-2, Reward Model, PPO, Pedagogical
  • Набор данных: stanfordnlp/sst2
  • Задача: Создание текста с позитивными чувствами
  • Среда: Google Colab
• veRL: Volcano Engine Reinforcement Learning for LLM
  • ByteDance Seed MLSys Team & HKU: Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu
  • Ключевое слово: Flexible, Efficient, RLHF framework
  • Задачи: RLHFРазумные задачи, включая математику и code.
• OpenRLHF
  • OpenRLHF
  • Ключевое слово: 70B, RLHF, DeepSpeed, Ray, vLLM
  • Задача: Простой в использовании, масштабируемый и высокопроизводительный RLHF Framework (Поддержка полной настройки 70B + и LoRA & Mixtral & KTO).
• Potato
  • David Jurgens et al.
  • Ключевое слово: Annotation, Human Evaluation, Quality Control, AI-Assisted Labeling
  • Задача: Портативная аннотационная платформа для оценки человека и сбора обратной связи с 20+ типами аннотаций и оценкой следов агента
• PaLM + RLHF - Pytorch
  • Phil Wang, Yachine Zahidi, Ikko Eltociear Ashimine, Eric Alcaide
  • Ключевое слово: Transformers, PaLM architecture
  • Набор данных: enwik8
• lm-human-preferences
  • Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving
  • Ключевое слово: Reward learning for language, Continuing text with positive sentiment, Summary task, Physical descriptive
  • Набор данных: TL;DR, CNN/DM
• following-instructions-human-feedback
  • Long Ouyang, Jeff Wu, Xu Jiang, et al.
  • Ключевое слово: Large Language Model, Align Language Model with Human Intent
  • Набор данных: TruthfulQA RealToxicityPrompts
• Transformer Reinforcement Learning (TRL)
  • Leandro von Werra, Younes Belkada, Lewis Tunstall, et al.
  • Ключевое слово: Train LLM with RL, PPO, Transformer
  • Задача: IMDB sentiment
• Transformer Reinforcement Learning X (TRLX)
  • Jonathan Tow, Leandro von Werra, et al.
  • Ключевое слово: Distributed training framework, T5-based language models, Train LLM with RL, PPO, ILQL
  • Задача: Отличная настройка LLM с RL с использованием функции вознаграждения или набора данных с маркировкой вознаграждения
• RL4LMs (A modular RL library to fine-tune language models to human preferences)
  • Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté,Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, Yejin Choi
  • Ключевое слово: Optimizing language generators with RL, Benchmark, Performant RL algorithm
  • Набор данных: IMDB, CommonGen, CNN Daily Mail, ToTTo, WMT-16 (en-de), NarrativeQA, DailyDialog
• LaMDA-rlhf-pytorch
  • Phil Wang
  • Ключевое слово: LaMDA, Attention-mechanism
  • Задача: Предварительная подготовка с открытым исходным кодом исследовательской работы Google LaMDA в PyTorch
• TextRL
  • Eric Lam
  • Ключевое слово: huggingface’s transformer
  • Задача: Генерация текстов
  • Среда: PFRL, gym
• minRLHF
  • Thomfoster
  • Ключевое слово: PPO, Minimal library
  • Задача: образовательные цели
• DeepSpeed-Chat
  • Microsoft
  • Ключевое слово: Affordable RLHF Training
• Dromedary
  • IBM
  • Ключевое слово: Minimal human supervision, Self-aligned
  • Задача: Автономная языковая модель, обученная с минимальным человеческим контролем
• FG-RLHF
  • Zeqiu Wu, Yushi Hu, Weijia Shi, et al.
  • Ключевое слово: Fine-Grained RLHF, providing a reward after every segment, Incorporating multiple RMs associated with different feedback types
  • Задача: Фреймворк, который позволяет обучать и учиться на функциях вознаграждения, которые мелкозернистые по плотности и нескольким RM. -Safe-RLHF
  • Xuehai Pan, Ruiyang Sun, Jiaming Ji, et al.
  • Ключевое слово: Support popular pre-trained models, Large human-labeled dataset, Multi-scale metrics for safety constraints verification, Customized parameters
  • Задача: Сдержанная привязанность к ценности LLM Безопасный RLHF
• VinePPO
  • Amirhossein Kazemnejad, Milad Aghajohari, et al.
  • Ключевое слово: Performant Implementation of RL algorithms for Reasoning, PPO, DPO, RestEM, Monte Carlo Value Estimation
  • Задача: Основные задачи, включая MATH и GSM8K

Набор данных

format:
- [title](dataset link) [links]
  - author1, author2, and author3...
  - keyword
  - experiment environments or tasks

• HH-RLHF
  • Ben Mann, Deep Ganguli
  • Ключевое слово: Human preference dataset, Red teaming data, machine-written
  • Задача: Набор данных с открытым исходным кодом для данных о предпочтениях человека о полезности и безвредности
• Stanford Human Preferences Dataset(SHP)
  • Ethayarajh, Kawin and Zhang, Heidi and Wang, Yizhong and Jurafsky, Dan
  • Ключевое слово: Naturally occurring and human-written dataset,18 different subject areas
  • Задача: Предназначен для использования для обучения RLHF Модели вознаграждений
• PromptSource
  • Stephen H. Bach, Victor Sanh, Zheng-Xin Yong et al.
  • Ключевое слово: Prompted English datasets, Mapping a data example into natural language
  • Задача: Инструментарий для создания, обмена и использования подсказок естественного языка
• Structured Knowledge Grounding(SKG) Resources Collections
  • Tianbao Xie, Chen Henry Wu, Peng Shi et al.
  • Ключевое слово: Structured Knowledge Grounding
  • Задача: Сбор наборов данных связан со структурированным обоснованием знаний
• The Flan Collection
  • Longpre Shayne, Hou Le, Vu Tu et al.
  • Задача: Коллекция компилирует наборы данных из Flan 2021, P3, супер-естественных инструкций
• rlhf-reward-datasets
  • Yiting Xie
  • Ключевое слово: Machine-written dataset
• webgpt_comparisons
  • OpenAI
  • Ключевое слово: Human-written dataset, Long form question answering
  • Задача: Тренируйте длинную модель ответа на вопросы в соответствии с предпочтениями человека
• summarize_from_feedback
  • OpenAI
  • Ключевое слово: Human-written dataset, summarization
  • Задача: Обучите модель обобщения в соответствии с предпочтениями человека
• Dahoas/synthetic-instruct-gptj-pairwise
  • Dahoas
  • Ключевое слово: Human-written dataset, synthetic dataset
• Stable Alignment - Alignment Learning in Social Games
  • Ruibo Liu, Ruixin (Ray) Yang, Qiang Peng
  • Ключевое слово: Interaction data used for alignment training, Run in Sandbox
  • Задача: Поезд на записанных данных взаимодействия в смоделированных социальных играх
• LIMA
  • Meta AI
  • Ключевое слово: without any RLHF, few carefully curated prompts and responses
  • Задача: Набор данных, используемый для обучения модели LIMA

Блоги

• [OpenAI] ChatGPT: Optimizing Language Models for Dialogue
• [Hugging Face] Illustrating Reinforcement Learning from Human Feedback (RLHF)
• [ZhiHu] 通向AGI之路：大型语言模型 (LLM) 技术精要
• [ZhiHu] 大语言模型的涌现能力：现象与解释
• [ZhiHu] 中文hh-rlhf数据集上的ppo实践
• [W&B Fully Connected] Understanding Reinforcement Learning from Human Feedback (RLHF)
• [Deepmind] Learning through human feedback
• [Notion] 深入理解语言模型的突现能力
• [Notion] 拆解追溯 GPT-3.5 各项能力的起源
• [gist] Reinforcement Learning for Language Models
• [YouTube] John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges
• [OpenAI / Arize] OpenAI on Reinforcement Learning With Human Feedback
• [Encord] Guide to Reinforcement Learning from Human Feedback (RLHF) for Computer Vision
• [hijkzzz] A Survey of Reinforcement Learning from Human Feedback (RLHF)
• [Weixun Wang] Overview of RL(HF)+LLM
• [Lilian Weng] Reward Hacking in Reinforcement Learning

Книги

• Reinforcement Learning from Human Feedback by Nathan Lambert
• Reinforcement Learning for Business
• The RLHF Book

Поддержка других языков

Turkish

Вклад

Наша цель - сделать этот репо еще лучше. Если вы заинтересованы в участии, пожалуйста, обратитесь к HERE Инструкции по вкладу.

Лицензия

Великолепно RLHF Выпускается под лицензией Apache 2.0.