What is Multilingual Language Data?
Parallel or non-parallel text corpora spanning multiple languages, used for multilingual models and low-resource language research.
Multilingual language data is text corpora spanning multiple languages, either parallel (the same content in several languages) or non-parallel (independent text in each). It is the text counterpart to multilingual speech, and it shares the same seam problem.
The specific hazard of text is encoding. Low-resource language text comes from scattered sources, and inconsistent character encoding, normalization and script representation break downstream processing in ways that are hard to trace back to the corpus.
The uses are multilingual model training, low-resource language research, and cross-lingual transfer. All three depend on the languages having been normalized to the same standard, because a model cannot align what has been encoded differently.
What buyers get wrong
Text data in low-resource languages tends to come from scattered sources with inconsistent encoding. Skip normalization before delivery and it breaks downstream.
The specification detail that decides everything
Character encoding and text normalization status must be stated per language.
This is the line item that most often separates a dataset that works from one that gets re-ordered. It belongs in the specification before production starts, not in the delivery review.
How we source it
We do not hold inventory in this category. A requirement comes in, we match it to a producer who can meet the specification, and we agree the quality bar with both sides before recording begins. A pilot batch goes first so misalignments surface early.
Scenarios this covers
Buyers rarely ask for a category by name. They describe a situation, and the situation is what tells us whether this category is the right one. If yours is in the list, name it in the brief and it goes straight into the specification. If it is not, describe it anyway, because it usually means the requirement sits in a neighbouring category.
What good Multilingual Language Data looks like
Multilingual text is judged per language, and the first thing to check is whether the encoding was normalized.
-
Character encoding stated per language
UTF-8 throughout, with the normalization form named, because two corpora in the same script can differ in composition and will not compare equal.
-
Text normalization status per language
Whether case folding, punctuation normalization and whitespace normalization were applied, and where they were not.
-
Script representation declared
Where a language can be written in more than one script, which one is used, and whether transliterations are included as a separate layer.
-
Per-language volume and source
Reported separately, because a combined total hides a language with a thousand sentences behind one with a million.
-
Parallel alignment quality where the corpus is parallel
The alignment method and a measured accuracy, since a parallel corpus with bad alignment trains noise.
Public datasets in this category
These are openly documented resources, not products we sell. Each one covers part of the category, and knowing where each stops is usually what decides whether a project needs a collection of its own.
-
mC4
Filtered web text in over a hundred languages, derived from Common Crawl. Coverage is broad and the per-language depth tracks how much of the web is written in that language, which is the same inequality that exists offline.
-
OSCAR
Web text classified by language, with several releases differing in filtering aggressiveness. The releases differ enough that results are not comparable across them, which the documentation states.
-
CC-100
A hundred-language web corpus built with a documented pipeline and released for research. Its value is the pipeline documentation, which lets a buyer judge what the filtering removed.
-
Wikipedia dumps
Curated text in every language edition, from millions of articles down to a few thousand. It is the cleanest multilingual resource and the smallest for exactly the languages that need it most.
Questions to ask a supplier
Questions that establish whether the languages were normalized to one standard or assembled from several.
- What encoding and normalization form is used, and is it the same for every language?
- Which script is used where the language has more than one, and are transliterations included?
- What is the volume per language, reported individually?
- For parallel data, how was the alignment produced and what is its measured accuracy?
- Where did the text for each language come from?
The vague answer in this category is about composition per language. A supplier who lists a hundred languages without giving the volume per language has described coverage rather than depth. Most of the volume will sit in a handful of languages, and the rest will be too thin to train on.
A verifiable answer gives tokens, documents and domains per language, states the script handling where a language uses more than one, and reports the duplicate rate per language. Ask for the per-language table, and check one low-resource language against the delivered files, because that is where the shortfall always is.
Technical attributes
The items below are the ones a specification can hold a producer to, and the ones that decide whether a delivery is usable for training rather than only present.
| Encoding | UTF-8, with the normalization form named |
|---|---|
| Normalization | Case, punctuation and whitespace handling stated per language |
| Script | Declared per language, with transliterations noted |
| Volume | Reported per language, never pooled |
| Alignment | Method and measured accuracy, for parallel data |
| Delivery format | Per-language files with a per-language manifest |
The per-language volume column decides what the corpus can serve, and it is the one most often collapsed into a total. Low-resource languages on the web are also the ones with the most duplication, so a reported figure can overstate the usable content by a wide margin. The deduplicated per-language count is the only figure worth planning against.
The values come from the markets the product serves. A model for six markets needs six languages at depth, and the depth requirement comes from the task rather than from the language. Write the markets down, then set a floor per language from the task.
A mistake here is measured as uneven quality. A model trained on an unbalanced corpus performs well in the languages with volume and poorly in the rest, and the gap is hidden by an average across languages. Fixing it means per-language collection at a worse unit price than the original batch.
How Multilingual Language Data is collected
Normalization is the first task, because it is what makes the languages comparable.
- Source identification per language. Each language's sources are identified separately, since a language assembled from mixed sources carries mixed conventions.
- Encoding and normalization pass. Text is converted to a single encoding and normalization form, and the choice is recorded rather than left as whatever the source used.
- Script decision where applicable. Where a language can be written in more than one script, the script is fixed and transliterations are produced as a separate layer where the specification needs them.
- Parallel alignment where the corpus is parallel. Alignment is produced by a stated method and its accuracy is measured on a sample.
- Per-language verification and delivery. Each language passes its own check, and per-language manifests are delivered with the text.
After assembly, each language is deduplicated separately, script and encoding are normalised, and the files are packaged with a per-language manifest stating the deduplicated volume and the domain mix. The deduplication has to be per language, because cross-language deduplication hides exactly the duplication that matters.
The pilot for a multilingual text corpus is one language assembled through the full chain, from acquisition through deduplication and script normalisation to the per-language count. Its purpose is to establish what a language actually yields, because the per-language figure is always lower than the source suggests and the shortfall differs by language. One language measured end to end sets the plan for the rest.
Acceptance is checked per language against the deduplicated floor stated in the specification, with the counts reported separately rather than pooled. Where the pilot shows a language cannot reach the floor, it is dropped from the batch rather than reported as covered, because a language included at a fraction of the floor changes the aggregate figure without changing the model.
How a delivery goes wrong
The defects below are the ones that recur across this category. Each is decided at collection or annotation time, which is why none of them can be repaired in delivery review.
-
Language labels came from the source site
A site registered in one country does not publish in one language. Labels inherited from the domain put mixed-language documents into single-language buckets, and the error is concentrated in exactly the multilingual regions the corpus is bought for.
-
Script and language were conflated
A script can serve several languages and a language can be written in several scripts. A corpus that records only one of the two cannot be filtered on the other, and both are needed for training decisions.
-
Low-resource languages are mostly duplicates
Web text in a low-resource language is often the same content republished across mirrors. After deduplication the effective size can be a small fraction of the reported one, and the deduplication rate belongs in the manifest.
Why this matters to buyers
Low-resource language work is where the value concentrates, and it is where the data quality problems are worst. Text in these languages is scattered, inconsistently encoded and often unnormalized, and a corpus that has fixed that is directly usable where raw collected text is not.
The second driver is comparability. Multilingual evaluation requires that the languages be processed to the same standard, because otherwise a difference in measured performance could be a difference in the data pipeline rather than in the model.
A multilingual model that is strong in a few languages and weak in many is discovered by the markets it was built for. The users in the thin languages report it as a quality problem, and the reports are hard to act on because the aggregate metrics look healthy. The corpus composition is the cause and it is not visible from the outside.
The second cost is the second collection. Under-filled languages have to be collected on their own, without the shared setup, and the unit price is higher. Setting a per-language floor before the first batch is what avoids this, and it costs nothing to write down.
A multilingual corpus with per-language deduplicated counts gives a model whose performance per market can be predicted. The team allocates evaluation effort to the languages with the thinnest coverage before release, and the markets are served evenly rather than in the order the corpus happened to favour.
The second gain is that the composition is documented. Script handling and normalisation rules differ by language and decide what the corpus is, and a delivery that records them can take a second batch aimed at new markets without re-deriving the decisions. That record is what makes a language programme cumulative.
Specification at a glance
| Category | Multilingual Language Data |
|---|---|
| Group | Non-speech data |
| Critical spec | Character encoding and text normalization status must be stated per language. |
| Included by default | Pilot batch, metadata schema, source and consent documentation |
| Pricing | Quoted from specification. No published rates |
See how a Multilingual Language Data delivery is structured →
Related categories
-
LLM Training Data
Text corpora for LLM pretraining and fine-tuning: instruction data, preference data, and multi-turn dialogue data.
-
Computer Vision Data
Annotated image and video data, with annotation types including classification, bounding boxes, segmentation masks, and keypoints.
-
NLP Data
Annotated text for NLP tasks: named entities, dependency syntax, sentiment polarity, and relation extraction.
-
Translation Data
Parallel corpora pairing source and target language, used for machine translation training and evaluation.
-
Dictionary Data
Structured dictionary entries with definitions, parts of speech, pronunciation, and example sentences, used for dictionary products and lexical understanding tasks.
-
Text Data
General-purpose text corpora from web pages, forums, reviews and similar sources, used for language model pretraining and text classification.
Questions about Multilingual Language Data
Why does normalization matter so much for text?
Because it is invisible until it breaks something. Two corpora in the same script but different normalization forms do not compare equal, and a model trained across both sees a vocabulary split that has nothing to do with the language.
What about languages with more than one script?
The script is fixed and declared, and transliterations are produced as a separate layer where the specification needs them. Mixing scripts in one file without recording it makes the corpus unusable for most downstream tasks.
How is parallel alignment quality measured?
By sampling aligned pairs and checking them against a stated criterion, then reporting the accuracy. An unmeasured alignment is an assumption, and misaligned parallel data trains a model on noise.
Can you report a single figure for the corpus?
We report per language. A combined total is the least informative number for a multilingual corpus, because it hides the language that is thin.
Do you have Multilingual Language Data available now?
No. We do not hold inventory. Everything is produced against a specification. That means the timeline starts when the spec is agreed rather than immediately, and it also means the data matches your requirement instead of approximately matching something already built.
Can this be combined with other categories in one delivery?
Yes, and it is common. A single specification can cover several categories across several languages, with one pilot batch and one quality bar. It usually reduces total cost because contributor recruitment and project setup are shared.
What does the consent documentation cover?
Signed contributor consent stating the intended use, the collection methodology, and the annotation guideline the data was produced under. For projects involving EU data subjects the chain is built to satisfy GDPR, including the transfer mechanism. The terms for moving the data across a border.
Request Multilingual Language Data
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.