What is LLM Training Data?
Text corpora for LLM pretraining and fine-tuning: instruction data, preference data, and multi-turn dialogue data.
LLM training data is text for pretraining and fine-tuning: raw corpora, instruction-response pairs, preference comparisons, and multi-turn dialogue. The four are different products with different quality bars and different legal exposure.
The distinguishing concern is provenance. Text corpora are the category where the legality of the source determines whether the resulting model can be shipped, and it is the first thing a commercial buyer asks about.
The second concern is duplication. A corpus with large internal duplication changes what a training run learns, and near-duplicate detection across a large corpus is a technical operation that has to be performed rather than assumed.
What buyers get wrong
Provenance legality and deduplication are what buyers care about most: train on copyrighted material and the model ships as a legal liability.
The specification detail that decides everything
Whether a source list and license documentation come with the data. This is the make-or-break item for commercial use.
This is the line item that most often separates a dataset that works from one that gets re-ordered. It belongs in the specification before production starts, not in the delivery review.
How we source it
We do not hold inventory in this category. A requirement comes in, we match it to a producer who can meet the specification, and we agree the quality bar with both sides before recording begins. A pilot batch goes first so misalignments surface early.
Scenarios this covers
Buyers rarely ask for a category by name. They describe a situation, and the situation is what tells us whether this category is the right one. If yours is in the list, name it in the brief and it goes straight into the specification. If it is not, describe it anyway, because it usually means the requirement sits in a neighbouring category.
What good LLM Training Data looks like
LLM training data is judged on provenance and deduplication, both of which are documented properties rather than qualities of the text itself.
-
Source list with license documentation
Where the text came from and under what terms, as a list rather than a description, because this is the item that decides whether the model can ship.
-
Deduplication performed and measured
Exact and near-duplicate removal, with the rate reported, because duplication is invisible in a sample and large in aggregate.
-
Contamination screening against evaluation sets
Whether the corpus was checked against common benchmarks, because a model that has seen the test set scores well and generalizes badly.
-
Filtering criteria stated
Quality filters, language identification and any content removal, described as criteria rather than as a claim that the data is clean.
-
Subtypes delivered separately
Instruction data, preference data and raw text have different collection methods and quality bars, and a corpus that mixes them serves neither purpose well.
Public datasets in this category
These are openly documented resources, not products we sell. Each one covers part of the category, and knowing where each stops is usually what decides whether a project needs a collection of its own.
-
Common Crawl
A monthly crawl of the open web, the raw material behind most public web-text datasets. It is unfiltered and enormous, and every downstream dataset is defined by how it was filtered from this.
-
C4 and mC4
Filtered web text in English and in over a hundred languages respectively. The filtering rules are documented and they are also the main criticism, since they remove material in ways that correlate with dialect and register.
-
The Pile and RedPajama
Composite corpora mixing web text, books, code, academic material and more, with the mixture documented. They exist to make the composition reproducible, which was the missing piece before them.
-
FineWeb and Dolma
More recent open pretraining corpora with published filtering pipelines and ablations. The ablations are the valuable part: they document what each filtering step actually did to model quality.
Questions to ask a supplier
Questions that decide whether the corpus is usable commercially, asked before any technical evaluation.
- What is the source list, and what license covers each source?
- Was deduplication performed, and what was the removal rate?
- Was the corpus screened against evaluation benchmarks, and which ones?
- What filtering criteria were applied, and what was removed by each?
- Is instruction data, preference data and raw text delivered as separate sets?
The vague answer in this category concerns provenance. A supplier who says the corpus is web text without recording which sources it came from has described a corpus that cannot be filtered. When a rights question arises about one source, the whole corpus becomes unusable rather than a subset of it.
A verifiable answer names the sources, the crawl dates, the filtering pipeline with its thresholds, and the deduplication method and rate. Ask for the contamination check against the benchmarks the model will be evaluated on, because without it the reported scores are not measuring generalisation.
Technical attributes
The items below are the ones a specification can hold a producer to, and the ones that decide whether a delivery is usable for training rather than only present.
| Source documentation | List of sources with the license covering each |
|---|---|
| Deduplication | Performed, with the removal rate reported |
| Contamination screening | Benchmarks checked, listed by name |
| Filtering criteria | Stated per filter, with removal rates |
| Data subtype | Raw text, instruction, preference, or dialogue, separated |
| Delivery format | Text with source and filter metadata per record or per shard |
The filtering pipeline column decides what the corpus is, and it is the one most often summarised as quality filtering. Filtering removes material by dialect, register and topic, and the removals are not neutral. A pipeline documented step by step can be argued with, and one described in a sentence cannot.
The values come from the model and the evaluation plan. A corpus for a general model needs breadth, while one for a domain model needs depth in that domain. The thresholds should be set against the evaluation rather than against a house default, because the two pull in opposite directions.
A mistake here is measured as inflated quality. Without deduplication, evaluation sets leak into training and the reported results are higher than the model deserves. The effect is large enough to change conclusions, and it stays invisible until the model meets genuinely new data.
How LLM Training Data is collected
For text, collection is licensing and processing rather than recording.
- Source identification and license review. Each source is identified and its license reviewed before inclusion, because a source added later is a source whose terms have to be renegotiated.
- Acquisition within terms. Text is acquired in the way the source permits, including rate limits and any prohibition on redistribution.
- Filtering with recorded criteria. Quality, language and content filters are applied against stated criteria, and the removal rate for each is recorded.
- Deduplication. Exact and near-duplicate removal runs across the whole corpus, and the rate is reported because it is a measure of the corpus rather than of the process.
- Contamination screening and delivery. The corpus is screened against evaluation sets, the result is reported, and the source and filter metadata is delivered alongside.
After assembly, the corpus is deduplicated, filtered against a documented pipeline, checked for benchmark contamination, and packaged with a manifest stating the source mix and the removal rates. The contamination report is the step that makes the evaluation numbers meaningful.
The pilot for a training corpus is a sample run through the full pipeline, from acquisition through filtering and deduplication to the contamination check against the benchmarks the model will be evaluated on. Its purpose is to measure the removal rates at each stage and the contamination result, because those figures decide what the corpus will actually teach. A sample run produces them in days rather than after the full pipeline.
Acceptance is checked on the per-source provenance record and on the contamination report, both against criteria stated before the run. The specification should name the sources and the filtering thresholds in advance, because a threshold set afterwards is a threshold set to the output. Where the pilot shows contamination, the exclusion is applied and the pilot run again rather than the benchmark changed.
How a delivery goes wrong
The defects below are the ones that recur across this category. Each is decided at collection or annotation time, which is why none of them can be repaired in delivery review.
-
Provenance stops at the aggregate
A corpus described as web text does not record which sites it came from. When a rights question arises about one source, the corpus cannot be filtered, and the whole corpus becomes unusable instead of a subset of it.
-
Deduplication was not done or not documented
Web text repeats itself heavily. Without deduplication, evaluation sets leak into training and the reported quality is inflated, and the effect is large enough to change conclusions.
-
Filtering removed material selectively
Quality and safety filters disproportionately remove text by dialect, by register and by topic. The resulting corpus is smaller and less representative in ways that only show up in per-group evaluation.
Why this matters to buyers
Provenance is the commercial gate. A model trained on text whose terms do not permit training is a liability that surfaces after the training run, and the cost of replacing the corpus at that point is the whole programme.
The second driver is that duplication and contamination distort training in ways invisible in a sample. A corpus that is thirty percent duplicated, or that contains the evaluation set, produces a model whose benchmark score does not describe its behaviour.
A model trained on a corpus of unknown provenance is a liability that grows with the model. The larger the model, the more expensive it is to retrain when one source turns out to be unusable, and the retraining is not optional. Provenance is what makes a partial withdrawal possible.
The second cost is the evaluation. Scores measured on a contaminated corpus do not predict field performance, so every downstream decision rests on a number that is too high. Teams usually discover this when a smaller competitor matches them on real tasks, and by then the training run has been paid for.
A corpus with documented provenance and a clean contamination report gives evaluation numbers that predict field performance. The team makes architecture and training decisions on measurements rather than on inflated scores, and a source that turns out to be unusable can be withdrawn as a subset rather than the whole corpus.
The second gain is that the pipeline is reproducible. Filtering thresholds and deduplication rules decide what the corpus is, and a delivery that documents them can produce a second batch comparable to the first. Programmes that train more than one model depend on that comparability.
Specification at a glance
| Category | LLM Training Data |
|---|---|
| Group | Non-speech data |
| Critical spec | Whether a source list and license documentation come with the data. This is the make-or-break item for commercial use. |
| Included by default | Pilot batch, metadata schema, source and consent documentation |
| Pricing | Quoted from specification. No published rates |
Related categories
-
Computer Vision Data
Annotated image and video data, with annotation types including classification, bounding boxes, segmentation masks, and keypoints.
-
NLP Data
Annotated text for NLP tasks: named entities, dependency syntax, sentiment polarity, and relation extraction.
-
Multilingual Language Data
Parallel or non-parallel text corpora spanning multiple languages, used for multilingual models and low-resource language research.
-
Translation Data
Parallel corpora pairing source and target language, used for machine translation training and evaluation.
-
Dictionary Data
Structured dictionary entries with definitions, parts of speech, pronunciation, and example sentences, used for dictionary products and lexical understanding tasks.
-
Text Data
General-purpose text corpora from web pages, forums, reviews and similar sources, used for language model pretraining and text classification.
Questions about LLM Training Data
Why is the source list more important than the text quality?
Because quality can be improved by filtering and provenance cannot be fixed after the fact. A high quality corpus with an unclear chain cannot be used commercially, and the chain is the first thing a buyer's legal review asks for.
What does contamination screening involve?
Checking the corpus against the benchmarks the model will be evaluated on, and reporting any overlap. A model that has seen the evaluation data scores well on it and tells you nothing about how it will behave elsewhere.
How much duplication is normal?
It varies by source type, which is why the rate matters more than a threshold. What matters is that deduplication was performed and the removal rate reported, so a buyer can compare corpora rather than guess.
Can you combine instruction and raw text in one delivery?
We deliver them separately. They are collected differently, annotated differently and used at different stages of training, and a combined delivery makes it impossible to apply a different standard to each.
Do you have LLM Training Data available now?
No. We do not hold inventory. Everything is produced against a specification. That means the timeline starts when the spec is agreed rather than immediately, and it also means the data matches your requirement instead of approximately matching something already built.
Can this be combined with other categories in one delivery?
Yes, and it is common. A single specification can cover several categories across several languages, with one pilot batch and one quality bar. It usually reduces total cost because contributor recruitment and project setup are shared.
What does the consent documentation cover?
Signed contributor consent stating the intended use, the collection methodology, and the annotation guideline the data was produced under. For projects involving EU data subjects the chain is built to satisfy GDPR, including the transfer mechanism. The terms for moving the data across a border.
Request LLM Training Data
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.