What is Conversational Speech?
Speech from natural conversation between two or more people, on open or semi-structured topics, used for conversational AI, voice assistants, and small talk models.
Conversational speech is recorded dialogue between two or more people who are talking to each other rather than completing a task. The topics are open or lightly structured, the turn-taking is natural, and the audio contains everything natural turn-taking produces: interruptions, overlaps, back-channels, false starts and laughter.
The commercially important distinction is between this and task-oriented dialogue. A contact centre interaction has a goal and a process; a conversation has neither. That makes conversational speech harder to annotate and more valuable for a specific class of model, the one that has to behave like a person rather than complete a transaction.
It is the base material for conversational AI, for voice assistants outside the command path, and for small-talk behaviour in any spoken product. It is also the category most often under-specified, because the word conversation sounds like a description when it is actually a category containing several different things.
What buyers get wrong
Natural conversation is full of overlapping speech, interruptions, particles, and laughter. Whether to keep these "messy" parts is the first thing to lock down on a project like this.
The specification detail that decides everything
Whether overlapping speech is kept or split. The two outputs cannot be mixed in a single delivery batch.
This is the line item that most often separates a dataset that works from one that gets re-ordered. It belongs in the specification before production starts, not in the delivery review.
How we source it
We do not hold inventory in this category. A requirement comes in, we match it to a producer who can meet the specification, and we agree the quality bar with both sides before recording begins. A pilot batch goes first so misalignments surface early.
Scenarios this covers
Buyers rarely ask for a category by name. They describe a situation, and the situation is what tells us whether this category is the right one. If yours is in the list, name it in the brief and it goes straight into the specification. If it is not, describe it anyway, because it usually means the requirement sits in a neighbouring category.
What good Conversational Speech looks like
Conversational data is easy to collect badly. These five properties separate a usable corpus from hours of audio nobody can train on.
-
A stated position on overlap
Overlapping speech is either kept and annotated, or split and assigned. The two produce different datasets and cannot be mixed inside one delivery batch.
-
Speaker separation that survives the overlap
When two people talk at once, diarization has to assign both segments. A corpus where overlaps are silently attributed to one speaker teaches the model something false.
-
Topic and register spread
Conversations between friends, between strangers, and between a participant and an interviewer are three different distributions. State which one the model will meet.
-
Natural disfluency preserved
Filler words, repairs and abandoned sentences are the signal rather than the noise. A corpus cleaned into fluent prose is a read-speech corpus with extra steps.
-
Metadata per session
Who was talking, whether they knew each other, the recording environment, and the topic prompt. Without this the corpus cannot be sliced by anything later.
Public datasets in this category
These are openly documented resources, not products we sell. Each one covers part of the category, and knowing where each stops is usually what decides whether a project needs a collection of its own.
-
Switchboard
About 2,400 hours of spontaneous two-party telephone conversation between strangers. It is the reference corpus for conversational speech and remains the standard baseline, though the telephone channel and the 1990s vocabulary date it.
-
Fisher English
Roughly 2,000 hours of telephone conversations collected a decade after Switchboard, with better coverage of regional accents. Same limitation: telephone bandwidth, scripted topics.
-
ICSI and AMI meeting corpora
Multi-party meetings with overlapping speech, captured with close-talk and room microphones. Valuable precisely because overlap is present, and small relative to what a production model needs.
-
Why these do not replace a collection
All three were recorded for research on a fixed set of topics and channels. A conversational system deployed in a different domain meets vocabulary and interaction patterns none of them contain, and none of them can be extended by the buyer.
Questions to ask a supplier
Questions that reveal whether a supplier has produced conversational data before, as opposed to read speech with two people in the room.
- How do you prompt conversation without turning it into an interview?
- What is your rule for attributing overlapped speech, and how is it enforced in the guideline?
- What share of recorded audio is discarded, and for what reasons?
- How are speakers paired, and does that pairing bias the corpus toward people who already know each other?
- Can you deliver a session-level manifest with pairing and topic metadata?
The vague answer in this category hides behind the word natural. A supplier who says the conversations are natural without describing the elicitation is describing a hope. Naturalness is produced by a method: pairs who know each other, a topic with no right answer, and a recording length long enough that the participants stop performing. Ask which of those was controlled and which was left to chance.
A verifiable answer names the elicitation protocol and the turn-taking statistics that came out of it. Mean overlap, mean pause length, backchannel rate, and the share of sessions discarded for sounding scripted. Those figures can be checked against the delivered files, and a supplier who tracks them is a supplier who will notice when a batch drifts.
Technical attributes
The items below are the ones a specification can hold a producer to, and the ones that decide whether a delivery is usable for training rather than only present.
| Overlap policy | Kept and annotated, or split and assigned, stated up front |
|---|---|
| Speaker count per session | Two, or three and above, with diarization required for multi-party |
| Diarization | Speaker-labelled segments with a stated accuracy target |
| Disfluency handling | Preserved verbatim, or normalized, never left unstated |
| Topic structure | Open, or prompted from a prepared topic set |
| Delivery format | Per-session audio with time-aligned speaker-labelled transcripts |
Speaker overlap is the attribute that decides what the corpus can be used for, and it is the one most often flattened in delivery. Conversational speech is full of interruptions and backchannels that overlap the main speaker. If the delivered files are segmented into clean single-speaker turns, the overlap has been removed and the corpus can no longer train anything that depends on timing.
The values should come from the interaction you are trying to model rather than from a house default. A model for meeting transcription and a model for call analysis need different overlap tolerances, different session lengths and different speaker counts. Ask for the target conditions first, then derive the numbers from them.
Setting these wrong is cheap to do and expensive to discover. A corpus with the overlap removed still transcribes well in isolation, and then fails in the field on exactly the moments where two people talk at once, which in spontaneous conversation is a large share of the interesting ones. Recovering it means collecting again, because the timing information was never stored.
How Conversational Speech is collected
Conversational data is collected by creating the conditions for a conversation, then getting out of the way.
- Pairing and briefing. Speakers are matched against the specification, briefed on the recording setup, and told what not to do. They are not given lines.
- Topic prompting. A topic set is prepared so that sessions do not stall, and prompts are phrased to open a subject rather than to request information.
- Recording in a natural setting. Sessions run long enough for the participants to stop performing, which is the point at which usable conversation begins.
- Diarization and transcription. Speaker-attributed transcription against a guideline defining how overlaps, back-channels and unintelligible passages are marked.
- Overlap review. A second pass over the overlapped regions specifically, because that is where diarization errors concentrate and a single pass will not catch them.
After recording, the sessions are segmented, transcribed, checked for speaker attribution, and packaged with per-session metadata covering the relationship between the participants and the recording setting. The segmentation policy is the part to read closely, because it is where overlap is either preserved or quietly removed.
The pilot here is a handful of full conversations rather than fragments, because the property being tested is turn-taking and that only exists across a whole dialogue. A few hours of complete conversations shows whether the segmentation rule holds at interruptions and backchannels, which is where a conversation corpus usually breaks. Testing on excerpts will not show it.
Acceptance rests on the segmentation and the speaker attribution, checked by a second annotator on a held-back set of conversations. The specification should state the tolerated error rate on turn boundaries before the run starts. Where the pilot fails, the segmentation rule is rewritten and the pilot run again, since re-segmenting a finished batch means re-deriving every turn label from the audio.
How a delivery goes wrong
The defects below are the ones that recur across this category. Each is decided at collection or annotation time, which is why none of them can be repaired in delivery review.
-
Overlap was cleaned up without saying so
Some producers remove overlapping regions to raise transcription quality. That produces a cleaner corpus and a model that has never seen two people talk at once, which is the normal condition of conversation.
-
Topic coverage is one setting
A corpus recorded entirely in casual small talk teaches casual small talk. If the deployment includes task discussion, disagreement, or negotiation, those registers have to be in the specification.
-
Speaker count is high but each speaker is brief
A large pool with two minutes per person gives acoustic variety and almost no per-speaker data. Whether that is a defect depends on the use, and it belongs in the delivery manifest either way.
Why this matters to buyers
Models trained on task dialogue sound like task dialogue. A voice product that can complete a transaction but cannot handle a greeting, a joke or a change of subject fails at exactly the point where users decide whether it feels usable.
The second reason is evaluation. Conversational corpora are what make it possible to test whether a spoken system handles the messy parts of speech: a person talking over it, changing their mind mid-sentence, or saying something with no transactional content at all. Those cases dominate real usage and are absent from every task-oriented dataset.
A model trained on a corpus with the overlap flattened behaves confidently and wrongly in exactly the situations that matter. It produces fluent single-speaker transcripts of conversations where two people were talking at once, and the errors cluster in the segments a downstream system most needs to get right. The metric that hides this is word error rate on the corpus itself, which stays good.
The commercial cost shows up as a second collection. Teams discover the gap after the first model is deployed, when field data stops resembling the training data. At that point the choice is to ship a model known to be weak in the hard cases or to pay for the category twice, and the second collection is usually slower because the supplier has moved on to other work.
A dialogue corpus with a segmentation rule that survived the pilot produces turn labels a model can learn from directly. The team spends its time on the dialogue policy rather than on repairing the turn structure, and the interruption handling in the product comes from the data rather than from hand-written rules.
The second gain is that the corpus can be extended. Conversations vary by domain and by channel, and a delivery that recorded its segmentation rule and its agreement figure can take a second batch from a different channel without invalidating the first. Programmes that start with telephone conversations and later add chat usually find that the rule, not the audio, is what they need to carry forward.
Specification at a glance
| Category | Conversational Speech |
|---|---|
| Group | Speech data |
| Critical spec | Whether overlapping speech is kept or split. The two outputs cannot be mixed in a single delivery batch. |
| Included by default | Pilot batch, metadata schema, source and consent documentation |
| Pricing | Quoted from specification. No published rates |
Conversational Speech across languages
This category is available in every language we cover. The difficulty changes with the language, which is why each language page documents its own constraints.
Related categories
-
Call Center Speech
Customer-service calls, simulated rather than intercepted, capturing both the agent and the caller side, used for call center QC, intent recognition, and dialogue systems.
-
Singing Voice
Vocal recordings with melody, in both a cappella and accompanied form, used for singing voice synthesis, music information retrieval, and lyric alignment.
-
Speech Commands
Targeted recordings of short command words or phrases, usually with many speakers reading each entry several times over, used for wake words and on-device command recognition.
-
Read Speech
Recordings of speakers reading specified text, with clear pronunciation and known text, the base material for TTS and ASR.
-
Podcast Speech
Long-form podcast and interview audio, either solo monologue or two-person conversation, used for long-form speech recognition and speaker modeling.
-
Multilingual Speech
Speech data covering multiple languages within one project, used for multilingual ASR, cross-lingual transfer, and language identification.
Questions about Conversational Speech
What is the difference between this and call centre speech?
Call centre speech is a task interaction with a process and an outcome. Conversational speech has no task. The recording conditions differ, the annotation differs, and the models they train are different. A supplier who treats them as one category will deliver the wrong one.
Do you keep overlapping speech?
Per the specification. Keeping it produces a corpus that reflects real conversation and is harder to annotate; splitting it produces a cleaner corpus that has removed the phenomenon many buyers want. There is no sensible default, which is why the specification has to state one.
How long should each session run?
Long enough that the participants stop performing. In practice that is usually well past the first several minutes, and it is the reason conversational sessions are costed by the session rather than by the minute.
Can you collect conversations in a specific dialect?
Yes, and it is a recruitment constraint rather than a recording one. For dialect-specific work we will state the realistic contributor pool before quoting, because that number sets the schedule.
Do you have Conversational Speech available now?
No. We do not hold inventory. Everything is produced against a specification. That means the timeline starts when the spec is agreed rather than immediately, and it also means the data matches your requirement instead of approximately matching something already built.
Can this be combined with other categories in one delivery?
Yes, and it is common. A single specification can cover several categories across several languages, with one pilot batch and one quality bar. It usually reduces total cost because contributor recruitment and project setup are shared.
What does the consent documentation cover?
Signed contributor consent stating the intended use, the collection methodology, and the annotation guideline the data was produced under. For projects involving EU data subjects the chain is built to satisfy GDPR, including the transfer mechanism. The terms for moving the data across a border.
Request Conversational Speech
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.