Parquet vs CSV
Short answer. CSV stores rows and Parquet stores columns. The two consequences are that Parquet reads only the columns a query touches, and that a column holding few distinct values can be stored once as a dictionary plus one index per row. On a training set where most columns are labels, splits and languages, that is where the size win comes from. Parquet does not shrink text: a column where every row is different has nothing to share, and it can even grow slightly because of the length prefixes.
How to do it
Row storage against column storage
In a CSV the fields of one record sit next to each other, so reading one column means reading every row and discarding the parts you do not want. In a columnar file the values of one column sit next to each other, so reading that column touches only its own bytes. The saving is proportional to how many columns you skip, which is why the gap is large on a forty-column table and small on a four-column one.
The cost is on the write side and on whole-record reads. Appending one row to a columnar file means touching every column, and reading one full record means gathering from every column. That is fine for training, where you read a fixed set of columns over many records, and awkward for a log you append to continuously.
Dictionary encoding, which is where the size goes
A column with eight distinct values repeated across a million rows does not need to store the strings a million times. Dictionary encoding stores the distinct values once and then one small index per row. The compression ratio is roughly the length of the average value over the size of the index, so it is large for a column of long repeated strings and zero for a column where every value is unique.
This is why a training set compresses well and a log file does not. Labels, splits, languages, categories and source names are all low-cardinality, and they are most of the columns in a typical dataset. Free text is the one column that does not compress, and it is usually the largest.
Where CSV is still the right answer
CSV opens in a spreadsheet, diffs as text, and can be inspected with standard tools, which matters when a human has to check a sample. A dataset that is a few thousand rows and gets read by a person is better off as CSV regardless of the size argument.
The other case is interoperability. Every tool reads CSV and a long tail of tools do not read Parquet. If the file leaves your pipeline and goes to a partner who will open it in Excel, the columnar format is a liability. Keep the authoritative copy in the format people can open, and convert for the training job.
Code
Measure the three layouts on the same 1000-row table, break the saving down per column, and show how many bytes a single-column read has to touch in each.
import csv, io, json
N = 1000
SPLIT = ["train", "val", "test"]
LANG = ["en", "es", "de", "fr", "ja"]
LABEL = ["positive", "negative", "neutral", "mixed",
"spam", "question", "request", "other"]
rows = [{"id": i,
"split": SPLIT[i % len(SPLIT)],
"lang": LANG[i % len(LANG)],
"label": LABEL[i % len(LABEL)],
"score": round(0.5 + (i % 97) / 200.0, 3),
"text": "sample text for row %d" % i}
for i in range(N)]
# row oriented: CSV
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=list(rows[0].keys()))
w.writeheader()
w.writerows(rows)
csv_bytes = buf.getvalue().encode()
# row oriented: JSONL
jsonl_bytes = ("\n".join(json.dumps(r) for r in rows) + "\n").encode()
# columnar + dictionary encoding: low-cardinality columns keep a dict plus indices,
# numeric columns store fixed-width ints
def dict_col(values):
order = sorted(set(values))
idx = {v: i for i, v in enumerate(order)}
return json.dumps(order).encode() + b"|" + bytes(idx[v] for v in values)
cols = {
"id": b"".join(int(r["id"]).to_bytes(4, "little") for r in rows),
"split": dict_col([r["split"] for r in rows]),
"lang": dict_col([r["lang"] for r in rows]),
"label": dict_col([r["label"] for r in rows]),
"score": b"".join(int(r["score"] * 1000).to_bytes(2, "little") for r in rows),
}
texts = [r["text"].encode() for r in rows]
cols["text"] = json.dumps([len(t) for t in texts]).encode() + b"|" + b"".join(texts)
col_total = sum(len(v) for v in cols.values())
print("same %d rows, three layouts" % N)
print(" CSV %7d bytes" % len(csv_bytes))
print(" JSONL %7d bytes (%.0f%% larger than CSV, keys repeat on every row)"
% (len(jsonl_bytes), 100.0 * (len(jsonl_bytes) / len(csv_bytes) - 1)))
print(" columnar+dict %7d bytes (%.0f%% smaller than CSV)"
% (col_total, 100.0 * (1 - col_total / len(csv_bytes))))
print()
print("per column: what CSV spends on the column vs what the columnar layout spends")
print(" %-6s %10s %10s %s" % ("col", "CSV", "columnar", "distinct values"))
for k in rows[0]:
csv_cost = len((",".join(str(r[k]) for r in rows) + "\n").encode())
distinct = len(set(r[k] for r in rows))
print(" %-6s %10d %10d %d" % (k, csv_cost, len(cols[k]), distinct))
print()
print("the win is entirely in the low-cardinality columns. label stores 8 distinct")
print("strings plus %d one-byte indices, where CSV repeats the string %d times." % (N, N))
print("the text column barely moves, because every row is different and there is")
print("nothing to share. parquet does not shrink text; it shrinks repetition.")
print()
print("reading one column:")
print(" CSV: scan all %d bytes, because every column sits on every row" % len(csv_bytes))
print(" columnar: touch %d bytes, which is the label column alone" % len(cols["label"]))
print(" that ratio is what makes a scan over 3 columns out of 40 cheap")- Standard library only. The columnar layout here is a simulation of what a columnar file does, built so the byte counts are real and reproducible; it is not an actual Parquet file. To write real Parquet, install pyarrow and use its ParquetWriter.
- Expected: CSV 49756 bytes, JSONL 112724 bytes, columnar 36032 bytes. JSONL is 127 percent larger than CSV because the six key names repeat on every row, and the columnar layout is 28 percent smaller than CSV.
- The per-column table shows where that comes from. The label column costs 7500 bytes in CSV and 1085 in the columnar layout, because 8 distinct strings plus 1000 one-byte indices replace 1000 copies of the string.
- Two columns got bigger, and both are worth understanding. The id column grew from 3890 to 4000 because a fixed-width four-byte integer costs more than decimal digits when the numbers are small. The text column grew from 23890 to 26891 because of the length prefixes. Neither breaks the argument, and both are the kind of thing a benchmark that only reports totals hides.
- The last block is the part that matters for a training job: reading one column means scanning all 49756 bytes from the CSV, against 1085 bytes from the columnar layout.
Where this goes wrong
Expecting Parquet to shrink free text
The size win comes from repeated values, so a table whose bulk is unique text barely compresses and can grow because of the length prefixes. If a dataset is mostly free text, converting it to a columnar format will not save the space you expect, and the real lever is compressing the text itself. Measure a sample before converting a terabyte.
Appending row by row to a columnar file
Writing one row at a time to a columnar format touches every column and produces many small row groups, which is the worst of both layouts. Columnar formats are written in batches. If the workload is an append-only log, keep it as line-delimited text and convert in a batch job.
Handing a columnar file to a person
A Parquet file does not open in a spreadsheet and does not diff as text, so a reviewer cannot check a sample without a tool. For a delivery where a human has to inspect the content, send CSV or line-delimited JSON, and keep the columnar file for the pipeline. Ask what the recipient will open it with before choosing.
Assuming every Parquet file has the same compression
Two files in the same format can differ by several times in size depending on the compression codec, the row group size and whether dictionary encoding is enabled. A benchmark that compares formats without stating these settings is comparing settings, not formats. Record the codec and the row group size with any size number you publish.
When to buy the data instead of building the pipeline
Choose the format yourself; it follows from how you read the data. Buy the data when the question is what is in it rather than how it is stored: a class balance you cannot collect, conditions your own capture has never seen, or annotation you cannot staff. Ask for a sample in the format of the full delivery and for the field list with types, because a format conversion is where a field silently changes type. We deliver in line-delimited JSON and in columnar formats, with the schema and the compression settings documented per delivery.
Need data for Parquet vs CSV?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.