What is RLHF (reinforcement learning from human feedback)?

Asked 13 hours ago Updated 13 hours ago 33 views

1 Answer


0

RLHF stands for reinforcement learning from human feedback. It is a training technique that uses human judgements to make AI models more helpful, honest and safe. It played a major role in turning raw language models into useful assistants like ChatGPT.

Why was RLHF needed? A language model trained only on internet text learns to predict the next word. That makes it knowledgeable, but not necessarily helpful. Ask it a question, and it might continue with another question, ramble, produce offensive content or ignore your instructions. The model needed to learn what kind of responses people actually want.

How does RLHF work? It typically involves three stages:

1. Supervised fine-tuning. Human trainers write example conversations showing ideal assistant responses. The model is trained on these examples to learn the basic style of a helpful assistant.

2. Training a reward model. The model generates several responses to the same prompt. Human reviewers rank them from best to worst, based on helpfulness, accuracy, safety and clarity. These rankings are used to train a separate model, called a reward model, that learns to predict which responses humans would prefer.

3. Reinforcement learning. The main model generates responses, the reward model scores them, and the main model is gradually adjusted to produce higher-scoring responses. Over many iterations, the model learns behaviours that align with human preferences.

What did RLHF achieve? It made models much better at following instructions, more likely to give relevant and well-organised answers, more likely to decline harmful requests and generally more pleasant to interact with. OpenAI's InstructGPT research in 2022 showed that a smaller RLHF-trained model could be preferred by people over a much larger model without it.

There are limitations and challenges:

Sycophancy. Models may learn to tell people what they want to hear, because reviewers sometimes prefer agreeable answers.
Reviewer bias. Preferences of the human reviewers can shape the model's views and style.
Cost. Collecting high-quality human feedback is expensive and slow.
Reward hacking. Models may find ways to score highly without truly being better.

Related techniques have developed since. Anthropic's Constitutional AI uses written principles and AI feedback alongside human feedback. Methods like direct preference optimisation simplify the process. Reinforcement learning is also used to train reasoning models, rewarding correct answers on maths and coding problems.

RLHF and its successors are a key reason modern AI assistants feel helpful rather than merely knowledgeable.

Write Your Answer