CoNLL Format
Short answer. CoNLL format is a tab-separated table where one line is one token, a blank line ends a sentence, and each column carries one annotation layer: the token, its part of speech, its chunk tag, its named entity tag. It has no header and no schema, so the meaning of a column lives in whatever convention the corpus states. Two details cause most of the damage: a file that does not end with a blank line loses its final sentence in a naive parser, and a file written with spaces instead of tabs parses into one column with no error.
How to do it
One token per line, a blank line per sentence
The unit of the format is the token, not the sentence. Each line holds one token and its annotations for every layer, separated by tabs, and a blank line marks the end of a sentence. A reader accumulates lines until it meets a blank one. This is what makes the format streamable and what makes a partial file still readable.
The sentence boundary is a blank line and nothing else, so a file whose last sentence is not followed by a blank line will lose that sentence in a parser that only flushes on the blank. The guard is one line of code and it is the difference between a corpus of ten thousand sentences and one of nine thousand nine hundred, with no error either way.
- one token per line, columns separated by tab
- blank line ends the sentence
- a line starting with a hash is a comment in the newer variants
The columns are a convention, not a schema
The file carries no column names, so the meaning of the third column is whatever the corpus documentation says. In the widely used shared task convention it is the chunk tag, and in another it may be a fine-grained part of speech tag or a dependency relation. A file and its documentation are one artifact; a file without the column list is not usable.
The number of columns also varies between corpora, and a reader that assumes three will silently drop the fourth. Check the column count across the whole file before loading anything, and fail on a file where it changes. A file with mixed column counts usually means two corpora were concatenated without a conversion step.
Tag schemes and the boundary conventions
Named entity tags come in at least two schemes. The older one marks only the first token of a multi-token entity with a B prefix and leaves the rest as I. The newer one uses a distinct prefix for the first token of each entity and reserves I for continuation, which removes an ambiguity when two entities of the same type sit next to each other.
A corpus that does not state its scheme cannot be converted reliably, because the conversion between the two is not a pure function of the tags: the ambiguous case is exactly the case where two adjacent entities of the same type are written as one. Ask for the scheme, and if the answer is the older one, check whether the corpus contains adjacent same-type entities before converting.
Code
Parse a two-sentence CoNLL file, show the last-sentence guard that keeps a file without a trailing blank line from shrinking, and detect the space-for-tab mistake.
def parse(text):
sentences, cur = [], []
for line in text.split("\n"):
if not line.strip():
if cur:
sentences.append(cur)
cur = []
continue
if line.startswith("#"):
continue
cur.append(line.split("\t"))
if cur: # saves the last sentence when the file has no trailing blank line
sentences.append(cur)
return sentences
good = (
"# sent_id = 1\n"
"the\tDET\tO\n"
"box\tNOUN\tB-OBJ\n"
"was\tAUX\tO\n"
"crushed\tVERB\tO\n"
"\n"
"arrived\tVERB\tO\n"
"early\tADV\tO\n"
"\n"
)
no_trailing_blank = good.rstrip("\n")
for name, text in (("with trailing blank line", good),
("without trailing blank line", no_trailing_blank)):
sents = parse(text)
print("%-28s sentences %d tokens %s"
% (name, len(sents), [len(s) for s in sents]))
print()
print("the last-sentence guard is the line that keeps these equal. Remove it and the")
print("second file loses its final sentence with no error, which is the classic way a")
print("corpus quietly shrinks by one sentence per file.")
print()
sents = parse(good)
widths = {len(tok) for s in sents for tok in s}
print("column counts seen: %s" % sorted(widths))
if len(widths) > 1:
print("MIXED COLUMN COUNTS: at least one row has a different number of fields")
bad = good.replace("was\tAUX\tO", "was AUX O")
print()
print("if a file uses spaces instead of tabs:")
sents_bad = parse(bad)
widths_bad = {len(tok) for s in sents_bad for tok in s}
print(" column counts seen: %s" % sorted(widths_bad))
print(" rows now parse as %d field instead of 3, and the token count still looks right"
% sorted(widths_bad)[0])- Standard library only.
- Expected: both files report 2 sentences with 4 and 2 tokens. They are equal because of the guard at the end of the loop, which flushes the accumulated sentence when the text ends without a blank line.
- The column count check reports 3, which is what a well-formed file looks like. The last block replaces the tabs on one line with spaces and the counts become 1 and 3 together, so the file now has rows with different field counts while the token count is unchanged. That is the failure mode worth guarding against: nothing is missing, the columns are just wrong.
Where this goes wrong
Losing the last sentence of every file
A parser that only flushes on a blank line drops the final sentence of any file that does not end with one, and reports no error. Across a corpus of thousands of files the loss is systematic and invisible in any per-file check. Assert that the sentence count matches the count of blank-line-delimited blocks plus one when the file does not end with a blank line.
Spaces where tabs were intended
Splitting on a tab produces one field per line when the file was written with spaces, and the token count is still correct, so a count-based check passes. Check the field count per line across the whole file and fail on any file where it varies. This is also what a spreadsheet does to the file when someone opens and re-saves it.
Not recording the tag scheme
The older and newer entity schemes differ only on tokens inside an entity, and the difference matters exactly when two entities of the same type are adjacent. A corpus without its scheme recorded cannot be converted reliably, because that case is not recoverable from the tags. Record the scheme with the corpus, and check for adjacent same-type entities before converting.
Assuming the column order is universal
The file carries no header, so the column order is a convention that varies between corpora. A loader written against one corpus will read the wrong field from another and produce annotations that look plausible. Keep the column list with the file, and validate the column count on load rather than trusting the order.
When to buy the data instead of building the pipeline
Write the parser yourself; it is twenty lines and the boundary handling has to match the corpora you actually receive. Buy the data when you need token-level annotation in a language or domain your team cannot cover, or when the tag scheme has to be applied consistently across a corpus larger than your annotators can handle. Ask for the column list, the tag scheme, and a sample file, because a CoNLL file without its documentation is not a usable artifact. We deliver token-level annotations with the column list and the tag scheme documented, checked for consistent field counts across the delivery.
Need data for CoNLL Format?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.