What is Text Data?
General-purpose text corpora from web pages, forums, reviews and similar sources, used for language model pretraining and text classification.
Text data is general-purpose text from the web, forums, reviews and similar sources, used for language model pretraining and text classification. It is the least structured category and the one where processing is most of the value.
The processing is the product. Raw crawled text is typically more than thirty percent duplicate or low-quality content, and the work of deduplication, filtering and scrubbing is what turns a crawl into a corpus.
The three processing steps are separate and are reported separately: deduplication, toxicity filtering, and personally identifiable information scrubbing. A corpus described as clean without separating them tells a buyer nothing about which was done.
What buyers get wrong
Deduplication and filtering are the most labor-intensive part: raw crawled data is usually more than 30% duplicate or low-quality content.
The specification detail that decides everything
Whether deduplication, toxicity filtering, and PII scrubbing have been done. The three must be reported separately.
This is the line item that most often separates a dataset that works from one that gets re-ordered. It belongs in the specification before production starts, not in the delivery review.
How we source it
We do not hold inventory in this category. A requirement comes in, we match it to a producer who can meet the specification, and we agree the quality bar with both sides before recording begins. A pilot batch goes first so misalignments surface early.
Scenarios this covers
Buyers rarely ask for a category by name. They describe a situation, and the situation is what tells us whether this category is the right one. If yours is in the list, name it in the brief and it goes straight into the specification. If it is not, describe it anyway, because it usually means the requirement sits in a neighbouring category.
What good Text Data looks like
General text is judged on the processing record, because the raw material is not the product.
-
Deduplication reported as a rate
Exact and near-duplicate removal, with the proportion removed, because duplication is invisible in a sample and large in aggregate.
-
Toxicity filtering reported separately, with the criteria
What was filtered and against what threshold, because a filter that removes too much changes the register of the corpus.
-
PII scrubbing reported separately, with the method
Whether personal identifiers were removed and by what method, since this determines the corpus's legal exposure.
-
Source composition described
Which domains and source types, with the proportions, because a corpus that is mostly forum text behaves differently from one that is mostly news.
-
Language identification and its accuracy
Which language each document is, with the identification method and its measured accuracy, because misclassified documents pollute both the language they are in and the language they are assigned to.
Public datasets in this category
These are openly documented resources, not products we sell. Each one covers part of the category, and knowing where each stops is usually what decides whether a project needs a collection of its own.
-
Common Crawl
Monthly crawls of the open web, the raw material behind nearly every web-text dataset. Unfiltered, and every use of it is a filtering decision that has to be documented to be reproducible.
-
Wikipedia dumps
Full exports of every language edition with structure and history. Curated, referenced, and narrower in register than the open web.
-
Project Gutenberg
Public-domain books in plain text, tens of thousands of titles. Clean and consistent, and the language is literary and largely historical.
-
C4 and FineWeb
Two generations of filtered web text with published pipelines, the second including ablations on what each filter step did. They are the standard starting point for open pretraining work.
Questions to ask a supplier
Questions that establish which processing steps were performed, asked separately because they are separate operations.
- What was the deduplication rate, with exact and near-duplicate reported separately?
- What toxicity filter was applied, and at what threshold?
- What PII scrubbing was performed, and by what method?
- What is the source composition, by domain type and proportion?
- How was language identified, and what is the accuracy of that identification?
The vague answer in this category is about the document boundary. A supplier who describes a corpus in tokens without saying whether documents were kept intact has described a stream rather than a corpus. Once the boundaries are gone, per-document filtering, deduplication and rights handling all become impossible.
A verifiable answer states the document count, the average length, the boilerplate removal method, the language distribution as measured, and the deduplication rate. Ask for the document count as well as the token count, because a corpus that reports only tokens cannot be split or filtered.
Technical attributes
The items below are the ones a specification can hold a producer to, and the ones that decide whether a delivery is usable for training rather than only present.
| Deduplication | Exact and near-duplicate rates reported separately |
|---|---|
| Toxicity filtering | Criteria and threshold stated |
| PII scrubbing | Method stated |
| Source composition | Domain types with proportions |
| Language identification | Method and measured accuracy |
| Delivery format | Text with per-document source and processing metadata |
The document boundary column decides what can be done with the corpus later, and it is the one most often sacrificed to save space. A corpus delivered as concatenated text cannot be deduplicated, cannot have a subset withdrawn, and cannot be split into training and evaluation sets without leakage. The boundary has to exist in the delivery.
The values come from the training plan. A model that will be fine-tuned needs documents short enough to sample, while one trained on long context needs documents that are long and intact. The average length and the length distribution are specification items rather than packaging details.
A mistake here is irreversible. Document boundaries that were not recorded cannot be reconstructed, and the corpus becomes a single undifferentiated block. Every downstream decision, from deduplication to rights withdrawal, is then made at the level of the whole corpus rather than a part of it.
How Text Data is collected
Collection is a pipeline, and each stage is reported so the buyer can see what was done.
- Source selection and rights review. Which sources, and under what terms, established before crawling rather than after.
- Crawling within terms. Acquisition respects each source's terms including rate limits and any prohibition on redistribution.
- Filtering and deduplication. Quality, language and content filters run against stated criteria, and deduplication runs across the whole corpus with the rate recorded.
- PII scrubbing. Personal identifiers are removed by a stated method, and the method determines what the guarantee actually covers.
- Composition review and delivery. The final composition is measured and reported by source type and language, and the processing record is delivered with the corpus.
After assembly, documents are kept intact with their metadata, boilerplate is removed per document, and the files are packaged with a manifest stating the document count, the length distribution and the removal rates. The document count is the figure that makes the rest of the manifest meaningful.
The pilot for a text corpus is a sample assembled with document boundaries intact, run through boilerplate removal and deduplication before the full build. Its purpose is to measure the document count, the length distribution and the removal rates, because a corpus described only in tokens cannot be split, filtered or withdrawn in part. Those figures are what the specification is then held to.
Acceptance is checked on the document count alongside the token count and on the per-source removal rates. The specification should state the length distribution and the deduplication method before assembly starts. Where the pilot shows the boundaries were lost, the extraction is redone rather than the corpus accepted, since document boundaries that were not recorded cannot be reconstructed.
How a delivery goes wrong
The defects below are the ones that recur across this category. Each is decided at collection or annotation time, which is why none of them can be repaired in delivery review.
-
Boilerplate and navigation were kept
Repeated page furniture across millions of documents dominates the token distribution. A model trained on it generates cookie notices and menus, and removing it afterwards is not possible.
-
Document boundaries were lost
Concatenated text without document separators prevents any per-document filtering, deduplication, or rights handling. The separator has to exist in the delivery, not be inferred.
-
Language mixture was not characterised
A corpus described as English that contains a substantial share of other languages performs differently from one that does not. The measured language distribution belongs in the manifest.
Why this matters to buyers
Pretraining corpora are judged on what was removed as much as on what is present. Duplication, contamination and low-quality text distort a training run in ways that are invisible in a sample and expensive to discover later.
The second driver is legal exposure. PII in a training corpus is a liability that persists in the model, and whether it was scrubbed, and how, is a question a buyer's legal review will ask.
A corpus without document boundaries forces all-or-nothing decisions. If one source turns out to be unusable, the whole corpus is affected rather than that source, and the retraining that follows is paid for entirely. The boundary is cheap to preserve and impossible to restore.
The second cost is evaluation leakage. Without boundaries, splitting into training and evaluation sets has to be done at the level of the block, which puts the same content on both sides. The reported quality is then inflated, and the error is large enough to change which model looks better.
A text corpus with intact documents gives the team options it would otherwise not have. Evaluation sets can be split without leakage, one source can be withdrawn without losing the corpus, and the length distribution means sampling for fine-tuning behaves predictably.
The second gain is that the corpus can be rebalanced per source. Sources and removal rates are documented, so a second batch can add depth in a domain or language without re-running the whole pipeline. That record is what makes a text programme grow by addition rather than by replacement.
Specification at a glance
| Category | Text Data |
|---|---|
| Group | Non-speech data |
| Critical spec | Whether deduplication, toxicity filtering, and PII scrubbing have been done. The three must be reported separately. |
| Included by default | Pilot batch, metadata schema, source and consent documentation |
| Pricing | Quoted from specification. No published rates |
Related categories
-
LLM Training Data
Text corpora for LLM pretraining and fine-tuning: instruction data, preference data, and multi-turn dialogue data.
-
Computer Vision Data
Annotated image and video data, with annotation types including classification, bounding boxes, segmentation masks, and keypoints.
-
NLP Data
Annotated text for NLP tasks: named entities, dependency syntax, sentiment polarity, and relation extraction.
-
Multilingual Language Data
Parallel or non-parallel text corpora spanning multiple languages, used for multilingual models and low-resource language research.
-
Translation Data
Parallel corpora pairing source and target language, used for machine translation training and evaluation.
-
Dictionary Data
Structured dictionary entries with definitions, parts of speech, pronunciation, and example sentences, used for dictionary products and lexical understanding tasks.
Questions about Text Data
Why report the three processing steps separately?
Because they are separate operations with separate failure modes. A corpus described as clean may have been deduplicated and not scrubbed, or filtered and not deduplicated, and a combined claim hides which.
How much duplication should we expect?
It varies by source, which is why the rate matters more than a threshold. What matters is that deduplication was performed across the whole corpus and the removal rate is reported.
What does PII scrubbing actually cover?
It depends on the method, which is why the method is stated. Pattern-based scrubbing catches structured identifiers and misses names in context; a model-based approach catches more and makes different errors. The guarantee follows the method.
How accurate is language identification?
It depends on document length and on how close the languages are, and it is measured and reported rather than assumed. Misclassified documents pollute both the language they belong to and the language they were assigned to.
Do you have Text Data available now?
No. We do not hold inventory. Everything is produced against a specification. That means the timeline starts when the spec is agreed rather than immediately, and it also means the data matches your requirement instead of approximately matching something already built.
Can this be combined with other categories in one delivery?
Yes, and it is common. A single specification can cover several categories across several languages, with one pilot batch and one quality bar. It usually reduces total cost because contributor recruitment and project setup are shared.
What does the consent documentation cover?
Signed contributor consent stating the intended use, the collection methodology, and the annotation guideline the data was produced under. For projects involving EU data subjects the chain is built to satisfy GDPR, including the transfer mechanism. The terms for moving the data across a border.
Request Text Data
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.