Script splits: when one spoken language needs two datasets
Hindi and Urdu are mutually intelligible when spoken and mutually unreadable when written. That single fact reshapes how the data has to be built.
A Punjabi speaker in Lahore and a Punjabi speaker in Amritsar can hold a conversation without difficulty. Ask them to write it down and they will produce text in two scripts that share almost no characters.
This is a script split: one spoken language, two writing systems. It appears in Punjabi, Hindi and Urdu, Serbian and Croatian, Kazakh, Mongolian, Azerbaijani, and several others, and it changes what "a dataset for this language" means.
Why it is a data problem and not just a writing problem
Speech data is a pairing of audio with text. If the same audio can legitimately be paired with two different texts, then the dataset is not one dataset.
Train a model on Hindi-script text and deploy it where users type in Urdu script, and the acoustic model may be fine while the language model fails completely. The failure looks like a model problem. It is a specification problem.
The three ways to handle it
There is no universally correct answer, but there are three coherent options and the specification needs to pick one.
- One script only: collect and annotate in the script your deployment target uses. Simplest, and correct when you know where the model will run.
- Parallel scripts: annotate the same audio in both scripts, producing two aligned text corpora from one recording. Roughly doubles annotation cost, and gives you a genuinely bilingual asset.
- Transliteration layer: annotate in one script and generate the other by rule. Cheapest, but transliteration between split scripts is not fully reversible, and the errors are systematic rather than random.
The related trap: script is not the same as language
Serbian and Croatian are the same language by most linguistic measures, written in Cyrillic and Latin respectively. Hindi and Urdu differ more in vocabulary than in grammar, but their scripts diverge completely.
This means a dataset labeled by script and a dataset labeled by spoken variety will not match. Buyers who specify "Serbian" and receive a dataset that is 40% Latin-script Ekavian will find that the label did not mean what they assumed.
How to specify it
State the script explicitly in the specification, alongside the language name. Then state whether transliteration is permitted, and if so, whether it must be human-verified.
This is a two-line addition to a specification that routinely prevents a full re-annotation.
What this does to your evaluation set
A script split propagates into evaluation, and this is where it does the most quiet damage. A test set written in Devanagari measures how the system performs for users who type in Devanagari. It does not measure performance for users who type in Nastaliq, even when both groups speak the same language, because the language model sees a different token sequence.
The failure mode is a benchmark score that looks healthy while a share of the real user population is being served badly. The score is not wrong. It is answering a narrower question than the one the team thinks it is answering.
The fix is structural rather than statistical: build the evaluation set from the same script distribution you expect in deployment, and report results per script rather than as a single aggregate. If deployment is 70 percent one script and 30 percent the other, an aggregate number will be dominated by the larger group and will hide the smaller one entirely.
Choosing between the three options
The three approaches are not ranked; they fit different situations, and the decision usually comes down to how much is known about deployment.
- One script only fits when the deployment target is known and stable. It is the cheapest option and the least forgiving: if the assumption about where the model runs turns out to be wrong, the annotation has to be redone rather than extended.
- Parallel scripts fit when the audience genuinely spans both, or when the deployment target is still undecided. It roughly doubles annotation cost and produces an asset that is unusually defensible, because very little of it exists.
- The transliteration layer fits when one script is clearly primary and the other is a convenience. Budget for verification: transliteration between split scripts is rule-based in principle and irregular in practice, and unverified output introduces systematic errors that are harder to detect than random ones.