Dataset Format
Short answer. A dataset format is a choice about three things: how a record is separated from the next one, how a field is typed, and whether nested structure survives the round trip. JSONL gives you one JSON object per line, so a corrupt line costs one record and nested fields survive. CSV is smaller and loads everywhere, but every field comes back as text and any nested field is flattened into a string. The failure is quiet: a label list written into a CSV cell reads back as a string, and code that counts the labels gets the length of the text instead.
How to do it
Line-delimited against whole-file
JSONL and CSV share a property that matters more than either format: they are line-delimited, so a reader can stream them and a corrupted line costs one record. A single JSON array holding the whole dataset has to be parsed in one piece, which means the file has to fit in memory and a syntax error anywhere makes the whole file unreadable.
The practical test is what happens when a job dies halfway. With a line-delimited file you keep everything written so far. With a whole-file format you keep nothing, and on a long annotation or export job that difference is the difference between resuming and restarting.
- JSONL: one object per line, nested fields survive, larger on disk
- CSV: smaller, opens in a spreadsheet, every field returns as text
- whole-file JSON: nothing streams, one syntax error loses everything
Where CSV quietly loses information
A CSV cell can only hold text, so a list or a dictionary gets converted to its printed form on the way out and stays a string on the way back. Nothing raises an error. The field that used to be a list of labels becomes a string that starts with a bracket, and the code that counts labels now counts characters.
Numbers go the same way. An identifier written as an integer comes back as a string, which matters when it is used as a dictionary key, and a long integer can lose precision in a tool that reads it as a float. The fix is to write the type down alongside the data, or to keep the nested fields encoded as JSON inside the cell and decode on read.
Choosing for the pipeline, not for the file
The format that loads fastest is not always the one to store. A common split is to keep the authoritative copy in a line-delimited format that a human can inspect, and to convert to a columnar format for training. The conversion is a build step that can be rerun, and the authoritative copy stays readable when something goes wrong.
Whatever you choose, validate on load rather than trusting the extension. Count the records, check that the fields your training code reads are present on every record, and compare the count against the manifest. A dataset that loads without error and is missing a field on one percent of records will train and produce a model that behaves strangely on that one percent.
Code
Write the same two records as JSONL and as CSV, round trip both, and show exactly which field types the CSV silently changes.
import csv, io, json
records = [
{"id": 1, "text": "the box was crushed",
"labels": ["negative"], "meta": {"lang": "en", "chars": 20}},
{"id": 2, "text": "arrived early and well packed",
"labels": ["positive", "fast"], "meta": {"lang": "en", "chars": 28}},
]
jsonl = "\n".join(json.dumps(r, ensure_ascii=False) for r in records) + "\n"
print("JSONL, %d bytes" % len(jsonl.encode()))
print(jsonl.strip()[:120] + " ...")
print()
buf = io.StringIO()
w = csv.writer(buf)
w.writerow(list(records[0].keys()))
for r in records:
w.writerow([r[k] for k in r])
naive = buf.getvalue()
print("CSV, naive, %d bytes" % len(naive.encode()))
print(naive.strip())
print()
back = list(csv.DictReader(io.StringIO(naive)))
print("round trip through the naive CSV:")
for field in ("id", "labels", "meta"):
orig, got = records[0][field], back[0][field]
print(" %-7s %-28r -> %-28r type %s -> %s"
% (field, orig, got, type(orig).__name__, type(got).__name__))
print()
print("labels comes back as a string, not a list. anything that does")
print("len(row['labels']) now gets 12 instead of 1, and nothing raises an error")
buf = io.StringIO()
w = csv.writer(buf)
w.writerow(list(records[0].keys()))
for r in records:
w.writerow([json.dumps(r[k]) if isinstance(r[k], (list, dict)) else r[k]
for k in r])
safe = buf.getvalue()
back2 = list(csv.DictReader(io.StringIO(safe)))
print()
print("CSV with nested fields encoded, %d bytes" % len(safe.encode()))
r2 = {k: (json.loads(v) if v and v[0] in "[{" else v)
for k, v in back2[0].items()}
print(" labels round trips as %r" % (r2["labels"],))
print(" costs %d extra bytes on %d records"
% (len(safe.encode()) - len(naive.encode()), len(records)))- Standard library only.
- Expected: JSONL is 222 bytes and the naive CSV is 173. The CSV is smaller because the keys are written once in the header instead of on every line.
- The round trip block is the reason to read this page. The id comes back as a string, and the labels and meta fields come back as their printed forms. The types changed and nothing raised.
- Encoding the nested fields as JSON inside the cell costs 20 extra bytes on two records, and the list round trips. At a million records that overhead is real, which is why line-delimited JSON is usually the better default when the records are not flat.
Where this goes wrong
Assuming the extension tells you the schema
Two files ending in .csv can disagree about the delimiter, the quoting, the character encoding and whether the first row is a header. Two files ending in .jsonl can disagree about which fields are optional. Validate the fields your code reads on load and fail loudly on the record that is missing one, rather than discovering it in the metrics.
Losing precision on identifiers
A long integer identifier read through a tool that treats it as a floating point number loses its last digits, and the record then belongs to a different item. This bites in spreadsheets and in any reader that infers types. Keep identifiers as strings in the file, or check that the round trip is exact before trusting the join.
Storing one giant JSON array
A single array has to be parsed whole, so the file must fit in memory and one syntax error makes it entirely unreadable. On an export job that takes hours, a crash near the end loses the whole run. Write line-delimited from the start, even if the consumer expects an array, and let the conversion be a separate step.
Treating the record count as validation
A file with the right number of lines can still have a missing field on some records, a truncated last line, or a duplicate id. Count the records, check the required fields on every record, check that ids are unique, and compare all three against the manifest the supplier sent. The count alone catches the least interesting failure.
When to buy the data instead of building the pipeline
Decide the format yourself, because it is tied to your training code. Buy the data with the format written into the specification: which fields, which types, whether nested structure is preserved, and what a null looks like. Ask for a sample in the exact format of the full delivery so the loader is tested before the batch arrives. We deliver in line-delimited JSON and in columnar formats, with the field list, the types and the null convention documented per delivery.
Need data for Dataset Format?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.