Instruction Tuning
Short answer. Instruction tuning continues training a base model on prompt and response pairs so that it follows instructions rather than continuing text. The mechanical detail that separates a working setup from a broken one is the loss mask: the loss is computed only on the response tokens, and the prompt tokens are masked out. Get the mask off by one and the first response token is never supervised, which shows up as a model that ignores the start of every answer. The data requirement is not volume but diversity of instruction type, since a set of fifty thousand paraphrases of one task teaches one task.
How to do it
Mask the prompt, supervise the response
A training step takes a token sequence and asks the model to predict each next token. For an instruction example the prompt is part of that sequence, but the model is not being asked to learn to produce the prompt, so those positions are masked out of the loss. What remains is the response, and the boundary between the two is where the bugs live.
The boundary is off by one relative to the input sequence, because position i predicts token i plus one. The response tokens begin at index n in the full sequence, so in the label array they begin at index n minus one. Writing n instead of n minus one masks the first response token, and the symptom is subtle: the model produces answers that start mid-thought.
- inputs = tokens[:-1], labels = tokens[1:]
- mask[n_prompt - 1:] = 1, not mask[n_prompt:] = 1
- padding is masked the same way, and forgetting to mask it teaches padding tokens
What makes an instruction set useful
The number that matters is the count of distinct instruction types, not the count of examples. A set built by paraphrasing one task fifty thousand times produces a model that is very good at that task and no better at following instructions in general. Counting instruction types means grouping by what the model is being asked to do: classify, extract, rewrite, summarize, transform a format, answer from a passage, refuse.
Response style matters as much as response content, because the model copies the style of the responses it is trained on. If every response in the set is three sentences and ends with a summary line, the tuned model will answer that way for questions that need one word. Measure the length distribution and the formatting conventions of the response side before you accept a set, and compare them against what the product needs.
How to tell a good set from a bad one before training
Sample fifty examples at random and read them. The failure that matters is not a wrong answer here and there, it is a structural problem: instructions that are all phrased the same way, responses that all have the same shape, or a large fraction of examples where the instruction can be answered without reading it. That last one produces a model that pattern-matches the instruction and ignores the input.
Then check the duplication rate. Near-duplicate instructions with near-duplicate responses are common in sets assembled from public sources, and they weight the training toward whatever was duplicated. A simple check is to normalize whitespace and case, hash the instruction, and count how many hashes appear more than once. A rate above a few percent is worth a conversation with the supplier.
Code
Build the loss mask for one instruction example, show the off-by-one that silently drops the first response token, and report what share of the sequence actually carries loss.
import numpy as np
PROMPT = "Classify the review: the box was crushed"
RESPONSE = " negative"
ids = [ord(c) for c in PROMPT + RESPONSE]
labels = np.array(ids[1:]) # each step predicts the next character
n_prompt = len(PROMPT)
print("prompt %d chars, response %d chars, %d training steps"
% (len(PROMPT), len(RESPONSE), len(labels)))
correct = np.zeros(len(labels))
correct[n_prompt - 1:] = 1.0 # the first response character must be supervised too
off_by_one = np.zeros(len(labels))
off_by_one[n_prompt:] = 1.0 # one short: the first response character never gets a loss
print("correct mask supervises %d steps" % int(correct.sum()))
print("off-by-one mask supervises %d steps" % int(off_by_one.sum()))
lost = np.nonzero(correct - off_by_one)[0]
print("the missing step is index %d, which is %r, the first character of the response"
% (lost[0], chr(labels[lost[0]])))
print()
lo = n_prompt - 6
print("steps %d to %d, 1 = supervised, 0 = masked" % (lo, len(correct) - 1))
print(correct[lo:].astype(int))
print("step %d is the last prompt character, step %d is the first response character"
% (n_prompt - 2, n_prompt - 1))
print()
print("share of the sequence that carries loss: %.1f%%"
% (100.0 * correct.sum() / len(correct)))
print("padding to 512 and masking it costs nothing; padding and NOT masking teaches the")
print("model to emit padding tokens, which is what shows up as runaway generation later")- pip install numpy. The example uses characters instead of tokens so the boundary is readable; the arithmetic is identical with a real tokenizer.
- Expected: 40 prompt characters, 9 response characters, 48 training steps. The correct mask supervises 9 steps and the off-by-one mask supervises 8, and the missing one is index 39, the space that begins the response. A model trained with that mask never learns to open an answer.
- Only 18.8 percent of the sequence carries loss. That is normal for instruction data and it is why the effective training signal is much smaller than the token count suggests.
Where this goes wrong
Masking the prompt with an off-by-one boundary
The response starts at index n in the token sequence and at index n minus one in the label sequence, and writing n is a one-character mistake with a quiet symptom: the first token of every response is unsupervised. The model still trains and still produces answers, so the bug survives to the end of a run. Assert the mask boundary against a tokenizer round trip before starting a long job.
Fifty thousand paraphrases of one task
A set that looks large can contain one instruction type repeated. The tuned model is then excellent at that task and no better at following instructions, which is a disappointing result to discover after a training run. Group the instructions by what they ask for and count the groups, not the rows, before you commit to the set.
Leaving padding unmasked
Padding tokens are added to make a batch rectangular, and if they are not masked the model is trained to predict them. At inference the model can then produce padding, which appears as generation that runs past the answer or emits filler. The mask is the same one you already built for the prompt, extended to cover the pad positions.
Judging the set by response correctness alone
Instruction sets are often accepted on whether the answers are right, while the response style is what the tuned model will actually copy. A set of correct answers all written in the same five-line format produces a model that answers everything in five lines. Check the length distribution and the formatting conventions of the response side against what the product needs.
When to buy the data instead of building the pipeline
Build the training loop and the mask yourself; they are small and the boundary has to match your tokenizer. Buy the instruction data when you need instruction types your team has not thought of, or volume beyond what your own people can write. Ask for the count of distinct instruction types, the length distribution of the responses, and the near-duplicate rate, because a set described only by its row count tells you nothing about whether it will teach the model to follow instructions. We deliver instruction sets with the type inventory and the duplicate analysis alongside the data.
More technical reference
Need data for Instruction Tuning?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.