RLHF (Reinforcement Learning from Human Feedback)
Short answer. RLHF trains a model from comparisons instead of from correct answers. Annotators see two responses to the same prompt and pick the better one, those pairs train a reward model, and the reward model then scores the policy during reinforcement learning. The reason it is used rather than supervised fine-tuning is that for many tasks nobody can write the correct answer, but almost anyone can say which of two answers is better. The cost is that the reward model learns whatever the annotators actually preferred, including the parts of that preference you did not intend to train.
How to do it
Why comparisons and not scores
Asking an annotator for a quality score from one to five produces numbers that drift between people and within the same person across a session, and the drift is not random: it correlates with how long they have been working. Asking which of two responses is better removes the scale, and the resulting pair carries one bit that is much more stable. The same annotator asked the same pair twice agrees with themselves far more often than they reproduce the same score twice.
The pair is also cheaper to collect per unit of signal, which is why RLHF datasets are counted in comparisons rather than in examples. A dataset of fifty thousand comparisons is a normal size for a narrow task, and the useful number is not the raw count but how many pairs are close calls. Pairs where one response is obviously broken teach the reward model almost nothing.
The reward model and the loss it minimizes
The reward model reads a prompt and a response and outputs one number. It is trained so that the preferred response of a pair scores higher, using the Bradley-Terry form: the probability that response A beats response B is the logistic function of the difference of their scores. Only differences matter, so the absolute level of the scores is arbitrary, and a model whose scores all shift by a constant is the same model.
A small weight decay is normally added. Without it the scores of a consistent preference set grow without bound while the loss keeps falling, because pushing every gap wider always reduces the loss. The decay is what keeps the numbers readable and keeps the reward model from becoming arbitrarily confident on pairs it has not seen.
- P(A beats B) = sigmoid(reward(A) - reward(B))
- loss = -log P(preferred) averaged over pairs, plus a small weight decay
- only the difference of two rewards is identified, the level is not
What the reward model actually learns
A reward model is a compression of the annotators preferences, including the shortcuts they used. If longer answers were rated better on average, the reward model learns that length is good, and the policy will then produce long answers. This is measurable before training a policy: score pairs that differ only in length and see how much of the reward is explained by the length difference alone.
The same test applies to formatting, to hedging language, and to confident tone. Run it on your own reward model with paired probes, one pair per suspected shortcut, and report the gaps. A reward model that separates the probes is not necessarily wrong, but you need to know which of its preferences came from the task and which came from the writing style of the people who annotated it.
Code
Fit a Bradley-Terry reward model on five preference pairs with gradient descent, and show how one pair reacts when a response is pushed down the ranking.
import numpy as np
# 4 candidate responses, 5 human preference pairs, each a (winner, loser) index pair
PAIRS = [(0, 1), (1, 2), (2, 3), (0, 2), (1, 3)]
N = 4
LR = 0.3
L2 = 0.01 # weight decay on the reward model, keeps scores from drifting
def sigmoid(x):
z = np.exp(-np.abs(x))
return np.where(x >= 0, 1.0 / (1.0 + z), z / (1.0 + z))
def loss_and_grad(r):
total = L2 * float(np.sum(r ** 2))
g = 2.0 * L2 * r
for w, l in PAIRS:
p = sigmoid(r[w] - r[l])
total -= np.log(p)
g[w] -= (1.0 - p)
g[l] += (1.0 - p)
return total / len(PAIRS), g / len(PAIRS)
r = np.zeros(N)
print("step loss reward 0..3 (centered)")
for step in range(1, 401):
L, g = loss_and_grad(r)
r = r - LR * g
r = r - r.mean() # only relative scores matter; recenter so values stay bounded
if step in (1, 5, 20, 100, 400):
print("%4d %.4f %s" % (step, L, np.round(r, 3)))
print()
order = np.argsort(-r).tolist()
print("reward ranking:", order)
print("human order: [0, 1, 2, 3]")
print("agrees:", order == [0, 1, 2, 3])
print()
r2 = r.copy()
r2[1] = r2[3] # push response 1 down to response 3's score
print("if answer 1 drops to answer 3's score (%.3f):" % r[3])
for w, l in PAIRS:
if (w, l) in ((1, 2), (2, 3), (1, 3)):
p = float(sigmoid(r2[w] - r2[l]))
print(" pair (%d,%d) now %.3f, loss %.3f" % (w, l, p, -np.log(p)))- pip install numpy
- Expected: the loss falls from 0.6931 at step 1 to 0.1136 at step 400, and the rewards settle at about 3.499, 1.09, -1.09, -3.499. The ranking comes out [0, 1, 2, 3], which is the order the human pairs implied. The five pairs are consistent with a single order, so the model can fit all of them at once and the loss keeps dropping.
- The last block is the part worth reading. Pushing answer 1 down to answer 3 scores leaves the pair (2,3) almost untouched at 0.917, wrecks the pair (1,2) down to 0.083, and lands the pair (1,3) exactly at 0.500 because the two scores are now identical. A reward model cannot tell you that two responses are tied, because a tie is a 0.500 probability rather than a separate state.
- Set L2 to 0 and rerun. The loss keeps falling and the rewards grow without bound, which is what an unregularized Bradley-Terry fit does on a consistent pair set.
Where this goes wrong
Collecting pairs where one answer is obviously broken
A dataset of pairs where the loser is incoherent trains a reward model that only detects incoherence, and the policy that follows it gets good at avoiding the obvious failure without getting any better at the task. The pairs that carry signal are the close ones. If you are buying preference data, ask for the distribution of annotator confidence or of inter-annotator agreement on the pairs, because a set with no disagreement at all is a set of easy pairs.
Assuming the reward model encodes the task and nothing else
The reward model learns whatever the annotators preferred, and annotators prefer longer answers, tidier formatting, and more confident tone independent of correctness. Probe for it before you train a policy: build pairs that differ only in length, only in formatting, only in hedging, and score them. The gap on each probe is the amount of reward your policy will be able to farm without doing the task.
Treating reward scores as absolute quality
Only differences between rewards are identified, so a score of 2.4 means nothing on its own and is not comparable across two separately trained reward models. Comparing raw scores from two runs is a common way to conclude that a model got worse when only the scale changed. Compare rankings and win rates within one model, and never across two.
Not holding out pairs by prompt
If the same prompt appears in both the training pairs and the evaluation pairs, the reward model has seen the prompt and the score on it is not evidence of generalization. Split by prompt, not by pair, and check that no prompt string appears on both sides. This is easy to get wrong when a prompt has many pairs, which is exactly the case where the leak matters most.
When to buy the data instead of building the pipeline
Build the reward model yourself, because it has to encode your preference and nobody else can hand you that. Buy the preference data when you need volume, when the annotators have to be people other than the engineers who wrote the product, or when the task needs domain knowledge your team does not have. Ask for the guideline, the agreement rate between annotators on a held-out set of pairs, and the prompt-level split, because those three decide whether the pairs teach the task or the writing style. We collect preference pairs against a written rubric, with prompt-level splits and measured agreement reported before delivery.
More technical reference
Need data for RLHF (Reinforcement Learning from Human Feedback)?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.