Voice AI Latency: Why Your Agent Feels Slow, and What Actually Fixes It

July 2026. Figures below are attributed to named sources. Where a number is a vendor claim rather than an independent measurement, it says so.

The number most teams optimize is the wrong number

Here is the standard latency budget you'll find in most articles on this topic.

This table is not wrong, exactly. It's measuring the wrong thing.

In a streaming pipeline, which is how every production voice agent should be built, you never wait for a stage to finish. You wait for it to start producing. The LLM figure of 1,000 to 4,000ms is total generation time for a complete response. What actually matters is time-to-first-token, typically 200 to 500ms, because your TTS begins synthesizing the first clause while the model is still writing the third sentence.

Add up first-chunk times instead of completion times and the same pipeline looks completely different. The "1.6 to 6.5 seconds" figure describes a naively sequential implementation that nobody should ship. Quoting it and then recommending streaming as the fix, without redoing the math, leaves readers with a mental model that's off by several hundred percent.

What to measure instead: end-to-end time from the moment the user stops speaking to the moment the first audio sample reaches their ear. One number, measured in production, reported as a distribution.

The dominant term is the one nobody talks about

Ask most teams where their latency goes and they'll say the LLM. Usually they're wrong.

According to Switchboard's latency analysis, VAD processing itself runs in under a millisecond on models like Silero, but end-of-speech detection adds real delay because the system must wait through a silence period to confirm the utterance is finished, typically 300 to 800 milliseconds, and often the single largest contributor to perceived latency. It is also the most tunable knob in the entire stack, which makes it the highest-leverage thing you can touch.

The trap is that tuning it is a genuine tradeoff, not a free win. As Gradium puts it, a 400ms pause in the middle of "I'd like to book a flight to... uh... Lisbon" is acoustically identical to a 400ms pause at the end of a sentence. A system measuring only whether sound is present cannot distinguish them. Raise the threshold to avoid interrupting people, and every completed turn now carries an extra half-second of dead air. Lower it, and you cut users off mid-thought.

For reference, natural human turn-taking gaps sit around 200ms.

Three ways out, in increasing order of sophistication:

• Tune per use case. Coval's guidance puts conversational endpointing at 200 to 400ms, with below 200ms risking clipped speech and above 500ms adding noticeable lag to every turn. They suggest keeping premature-endpointing false positives under 5% of turns. A transactional agent checking order status can run aggressive; an appointment scheduler where people speak in full sentences should not.

• Use your STT provider's end-of-utterance signal. LiveKit notes that many managed STT services emit an explicit event when they determine speech has stopped, faster than waiting out a full silence window, though you inherit the quality of their tuning.

• Move to semantic turn detection. A separate model reads the partial transcript in real time and predicts turn completion from meaning rather than silence. OpenAI's Realtime API exposes this as a semantic_vad turn-detection mode. This is where the field is heading, and it's the only approach that resolves the mid-sentence-pause problem rather than trading one failure mode for the other.

If you optimize one thing this quarter, optimize this. It's cheaper than switching models and it moves the number more.

A realistic streaming budget

Assuming semantic or well-tuned endpointing, streaming at every stage, and a co-located deployment:

Two things this table makes obvious that the sequential version hides. First, endpointing and TTFT together are roughly two-thirds of your budget, and that's where optimization effort belongs. Second, STT and TTS barely matter once you're streaming, which means switching speech vendors to chase latency is usually wasted effort.

Measure it as a distribution or don't bother

A single average latency number is close to useless.

Evaluation guidance from Hamming is explicit that you should track time-to-first-audio and end-to-end latency with percentile distributions, P50, P90, P95, and P99, not just averages, because a 500ms average can conceal 10% of calls spiking past three seconds. Those spiking calls are the ones that get abandoned, and they're invisible in your mean.

For a target, production telemetry published by DestiLabs across 10+ deployments puts median end-to-end latency at 680ms P50 and 1,180ms P95. That's a reasonable bar: P50 under 800ms, P95 under 1.5s. Hit those and the conversation feels smooth. Miss the P95 badly and a meaningful slice of your callers experiences a system that seems broken, regardless of what your dashboard average says.

Measure under concurrency. Latency on an idle system tells you nothing about latency during a Monday-morning queue. Load-test at your expected peak and report the percentiles from that run.

Fixing the LLM contribution

Once endpointing is handled, time-to-first-token is your next-largest term.

• Route by complexity. Most turns in a support conversation are simple. Send them to a small fast model and escalate to a larger one only when the query warrants it. This is the highest-return change available, because the TTFT gap between a small and a frontier model is large and most of your traffic doesn't need the frontier.

• Shorten your prompt. TTFT scales with input length. A 4,000-token system prompt costs you real milliseconds on every single turn. Audit it: most production prompts contain instructions that fire on 2% of calls and tax the other 98%.

• Cache aggressively. Prompt caching on the static portion of your system prompt, plus response caching for genuinely common queries.

• Don't stream tool calls into speech. If a turn requires a database lookup, the model can't answer until it returns. Either mask it with a brief acknowledgment ("let me pull that up") or design the flow so the lookup happens before the user finishes asking.

Transport and placement

Two infrastructure choices that are easy to get wrong and hard to fix later.

• Use WebRTC, not raw WebSockets, for browser and mobile audio. WebRTC handles jitter buffering, packet loss concealment, and adaptive bitrate. WebSockets require you to build all of that yourself, and you will build it worse.

• Co-locate. Put your orchestration in the same region as your users and, where possible, the same region as your model endpoints. Cross-region hops add 50 to 150ms per direction and they add it to every single turn.

Speech-native models: real, but not a free lunch

The architectural alternative to the cascaded STT to LLM to TTS pipeline is a model that processes audio directly and emits audio directly. OpenAI's Realtime API and Gemini Live are the prominent examples.

The genuine advantages: substantially lower latency, since you skip two model boundaries. Access to prosody, hesitation, and tone that the transcription step discards. And better barge-in, because the model is continuously listening rather than switching between listen and speak states.

The genuine costs, which most coverage skips:

• Less control. You can't swap the language model independently of the speech layer, inspect the intermediate transcript, or apply text-level guardrails before synthesis.

• Different failure modes. A model that reasons over audio tokens has a different error profile than one reasoning over text, and the evaluation tooling for it is less mature.

• Cost at volume. Cascaded pipelines remain cheaper to run at scale and easier to optimize component-by-component.

• Harder compliance. If you need every utterance logged, reviewed, and grounded, an architecture that skips the text representation makes that meaningfully more work.

Reasonable position for most teams in 2026: cascaded with semantic endpointing gets you to sub-second, which is the threshold that matters. Speech-native is worth adopting when you need the sub-400ms tier or when prosody is genuinely load-bearing for your use case, such as emotional support, sales, or anything where how someone said it changes the right response.

What about "users drop off at X seconds"?

You'll see completion-rate tables in a lot of articles on this subject, claims like 85% completion under one second dropping to 20% above three. Suspiciously round numbers, usually attributed to "internal studies" with no named study.

The directional claim is almost certainly true: latency hurts completion. The specific figures should not be repeated as fact, and you should not build a business case on them.

Measure it on your own traffic instead. Bucket your calls by observed end-to-end P50, and plot completion rate per bucket. You'll get a curve that's actually about your users, your use case, and your callers' patience, and you'll be able to defend it in a room. This takes an afternoon and it's worth more than any industry statistic.

Perceptual tricks, honestly assessed

Filler phrases ("let me check that for you"), typing sounds, and backchannels ("mm-hm") all reduce perceived latency by filling dead air. They work, within limits.

The limits: they only buy you about a second. Overuse makes the agent sound evasive, and a filler phrase followed by two more seconds of silence is worse than the silence alone, because you've now promised something and failed to deliver. Use them to smooth a known-slow operation like a database lookup. Don't use them to paper over a pipeline you haven't optimized.

The short version

• Instrument end-to-end first-audio latency, reported at P50/P95/P99, under realistic concurrency.

• Fix endpointing before you touch anything else. It's usually the biggest term and always the most tunable.

• Stream at every stage, and measure first-token rather than completion time.

• Route simple turns to small models; audit your system prompt for dead weight.

• Use WebRTC and co-locate.

• Target P50 under 800ms, P95 under 1.5s. Then stop optimizing latency and go work on accuracy.

That last point matters. Below roughly 800ms, further latency gains produce diminishing returns in user experience, while the failure modes that actually lose you customers, like the agent confidently saying something false, remain wide open.