Hallucination in Voice AI: The Layer Everyone Forgets

Your voice agent can hallucinate before the language model ever runs

Almost everything written about hallucination in voice AI is actually about hallucination in language models, with the word "voice" added. That misses the failure mode most specific to this medium, and arguably the more dangerous one.

Speech recognition hallucinates too. Not "mishears," it hallucinates. It generates fluent, grammatical sentences that were never spoken.

Koenecke et al.'s peer-reviewed study Careless Whisper: Speech-to-Text Hallucination Harms found that roughly 1% of Whisper transcriptions contained entirely hallucinated phrases or sentences with no basis in the underlying audio, invented sentences produced when nobody was speaking. Of those fabrications, 38% contained explicit harms: perpetuating violence, inventing associations, or implying false authority.

The mechanism is well understood. Whisper was trained on 680,000 hours of weakly supervised internet audio that was never manually curated, so segments containing silence, noise, or music got paired with arbitrary text. The model learned spurious correlations between non-speech input and fluent output. Researchers investigating this found recurring artifacts drawn from video-transcript training data, variations on "thanks for watching" and subtitle-credit boilerplate, alongside considerably more offensive fabrications.

Three things make this severe for voice agents specifically:

• Silence is the trigger, and conversation is full of silence. The study found that silences at the start and end of audio directly triggered hallucinations. Every natural pause in a phone call is an opportunity.

• It's an equity problem. Rates were higher for speakers with aphasia, 1.7% versus 1.2% for the control group. Any speech difference that produces more pauses or atypical prosody raises your hallucination rate. Your agent will fail disproportionately for disabled users, elderly users, and non-native speakers, and your aggregate metrics will hide it.

• Downstream, it's indistinguishable from truth. Your LLM receives a clean text string. It has no signal that the string was invented. It will reason over the fabrication faithfully and act on it, and if the fabricated text happens to contain something actionable, your agent takes a real action based on words nobody said.

If your architecture is STT to LLM to TTS, you have two hallucination surfaces, not one. Nearly all published guidance addresses only the second.

Why voice makes LLM hallucination worse than chat

The same fabrication does more damage spoken than written. Four structural reasons:

• No visible citations. In chat, you can render a source link under a claim. Reading a URL aloud is absurd, so the affordance that lets users verify simply doesn't exist in voice.

• No scrollback. A user can't re-read what the agent said three turns ago to check consistency. Memory of spoken content is worse than memory of read content, so contradictions go unnoticed.

• Speech carries unearned authority. Fluent, confidently-delivered speech is processed as more credible than the same words on a screen. Prosody adds a confidence signal that has nothing to do with accuracy.

• No time to think. Real-time conversation gives users no pause to evaluate. In chat, someone might read a claim twice. On a call, they've already moved on.

There's a compounding factor worth noting: MIT research has found that models use meaningfully more confident language when generating incorrect information than when generating correct information. In text, that's a subtle cue. Rendered as speech by a TTS engine that maps confident phrasing to confident delivery, it becomes an actively misleading signal: the agent sounds most certain precisely when it's wrong.

What the research actually says about inevitability

You'll frequently read that "a 2025 mathematical proof confirmed zero hallucination is architecturally impossible." That flattens a live academic disagreement into a settled fact, and it's worth getting right because the two positions imply different mitigation strategies.

The architectural position. Xu, Jain, and Kankanhalli's Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024) argues formally that LLMs, as computable functions, cannot provably learn all computable functions representing ground truth, so hallucination is structurally unavoidable. Bowen Xu's later work extends this, arguing that hallucination may be mitigable under a closed-world assumption but becomes inevitable in an open world.

The incentives position. Kalai, Nachum, Vempala, and Zhang's Why Language Models Hallucinate (OpenAI and Georgia Tech, September 2025) argues that models hallucinate because training and evaluation procedures reward guessing over admitting uncertainty, that hallucinations originate as ordinary binary classification errors under statistical pressure, and that reworking benchmarks to reward calibrated abstention would substantially reduce them.

These are not the same claim. One says the ceiling is architectural; the other says it's an artifact of how we score models. The honest summary: hallucination cannot currently be eliminated, and there is genuine disagreement about whether that's a permanent property of the architecture or a fixable consequence of training incentives. The practical implication is the same either way (build for containment, not elimination), but you should not tell readers a contested question is closed.

Mitigation, by layer

Because there are two hallucination surfaces, mitigation has to happen at both. Working from the audio inward.

Layer 1: Audio

• Trim silence before transcription. Koenecke's team found that removing silences reduces hallucinations, adjustable via the decibel threshold on the Whisper API. This is the cheapest fix available, and most teams don't do it.

• Gate on voice activity. Don't send audio segments to STT unless VAD confirms speech is present. Non-speech audio is precisely what triggers fabrication.

• Filter known artifacts. Researchers have constructed a "bag of hallucinations," a set of frequently recurring fabricated phrases, that can be stripped in post-processing. Subtitle boilerplate and video-transcript residue are highly repetitive and trivially filterable.

Layer 2: Transcription

• Check confidence and act on low scores. Rather than passing an uncertain transcript downstream, trigger a clarifying question. "Sorry, could you repeat that?" is nearly always better than confidently acting on a guess.

• Confirm consequential inputs. Anything that drives an action, such as account numbers, dates, amounts, or names, gets read back for confirmation. This is standard practice in human call centers for exactly this reason.

• Consider dual transcription for high-stakes flows. Running two STT engines and flagging disagreements is expensive, but for medical or financial contexts the cost is easy to justify.

Layer 3: Retrieval and grounding

• RAG remains the highest-return mitigation for factual queries. Retrieve from a verified knowledge base first, then generate strictly from what was retrieved.

• Make abstention the default, and make it sound natural. The system should be configured to say it doesn't know when retrieval returns nothing relevant. In voice, this needs writing care: "I don't have that information, but I can connect you with someone who does" works, while a flat "I don't know" sounds broken.

• Constrain actions with hard rules, not prompts. Anything with real consequences, such as issuing a refund, changing an order, or giving a dosage, should be gated by deterministic logic, not by asking the model nicely. A system prompt is a suggestion; a code path is a guarantee.

Layer 4: Verification

• Validate before synthesis. In a streaming pipeline, this is the hard part: you're synthesizing the first sentence while the model writes the third, so a post-hoc validator arrives too late. A practical compromise is to validate the first clause synchronously, stream the remainder, and be prepared to interrupt with a correction.

• Log the transcript, not just the audio. You need both, and you need them aligned, because diagnosing a failure requires knowing whether the agent misheard or misreasoned. Teams that log only audio spend enormous effort re-deriving this.

What to actually measure

Hallucination rate as a single percentage isn't operationally useful. Track these separately.

Build an adversarial test set from your own failures. Generic benchmarks won't predict your failure modes. Every production correction becomes a regression test. This suite is the most valuable asset you'll build, and it compounds.

Canary your releases. Roll knowledge-base and prompt changes to a small traffic slice first. Hallucination regressions from a prompt edit are common and don't show up in offline evals.

A note on the statistics you'll encounter

Writing about this topic, you will repeatedly meet the claim that AI hallucinations cost businesses $67.4 billion globally in 2024.

Trace it. Some sources attribute it to a study by AllAboutAI, an SEO-driven affiliate site. Others attribute the identical figure to Forrester Research. Others give no attribution at all. A single number credited to entirely different originators across dozens of articles is the signature of citation laundering: a figure that has been repeated until it acquired the appearance of authority.

Don't use it. The same caution applies to the widely quoted legal-hallucination range, which appears variously as 17 to 88% and 69 to 88% depending on the source, and to per-employee verification costs attributed alternately to Forrester and Microsoft.

There is plenty of genuinely rigorous work in this area: Koenecke et al. on ASR fabrication, Kalai et al. on training incentives, Xu et al. on architectural limits, and the Vectara hallucination leaderboard for model-level comparison. Cite that instead. In a piece arguing that fabricated confident claims are dangerous, using unverifiable statistics is a self-inflicted wound.

The short version

• You have two hallucination surfaces. Almost everything written on this covers only one.

• Trim silence before transcription. Cheapest available fix, rarely implemented.

• Stratify your accuracy metrics by speaker group, or you'll ship something that fails your most vulnerable users invisibly.

• Confirm consequential inputs by reading them back. Gate consequential actions in code, not in prompts.

• Ground with RAG, make abstention natural-sounding, and treat a zero abstention rate as a bug.

• Build your regression suite from real production failures, not from benchmarks.

• Check the provenance of any statistic before you publish it, including the ones in this article.