How to Build a Low-Latency Voice AI Agent: Where the Milliseconds Go

Voice is the most latency-sensitive AI application in production, and the reason is human. A chat user waits three seconds for a reply and keeps using your product; a voice user waits three seconds and the conversation is broken, because the person on the other end assumes the call dropped, starts talking again, and the agent talks over them [1]. Conversational-linguistics research puts the average gap between human turns at around 200 milliseconds, with anything up to roughly 500 milliseconds still feeling natural; past about one second callers notice the delay, and past two seconds they assume the line is dead [2].

Building a voice agent that responds inside that window is an engineering achievement, not a configuration default. This guide opens up the pipeline, shows exactly where the milliseconds go, and explains how to control each one. It is the technical companion to the pillar, Voice AI Agents for Enterprise.

The pipeline, in one paragraph

A real-time voice agent is a three-stage pipeline with two pieces of orchestration wrapped around it. Audio comes in, speech-to-text (STT) turns it into words, a large language model (LLM) decides what to say, and text-to-speech (TTS) turns the reply back into audio. Around all of it sits turn-taking (has the caller finished speaking?) and barge-in handling (what happens if they interrupt?) [1][3]. That is the whole architecture. A well-tuned version of it lands at roughly 600 milliseconds from end of user speech to first audio out [2][4], which is fast enough to feel conversational.

Where the milliseconds actually go

Here is the layer-by-layer budget for a well-tuned pipeline, measured as the incremental latency each stage adds at the critical moment between the caller finishing and the agent’s first audio [2]:

• Network and transport: 30 to 80 ms
• Turn detection (VAD and endpointing): 150 to 300 ms
• LLM time-to-first-token: 150 to 400 ms
• TTS time-to-first-audio: 100 to 200 ms

The single most useful thing to understand about this budget is counterintuitive: the latency hides where you do not expect it, in turn-taking and LLM time-to-first-token, not in STT or TTS [1]. Speech recognition and speech synthesis get blamed for slow agents, but STT runs while the caller is still talking, so its contribution at the end is small, and TTS begins producing audio while the LLM is still generating tokens, so its impact is smaller than its per-layer number suggests. If a voice agent feels sluggish, the endpointing decision and the model’s time-to-first-token are the first two places to look [1].

While the caller is still speaking

Network and transport 30–80 ms. Small, unless the call path and the inference sit in different regions.

Streaming STT Runs continuously, so it is essentially finished the moment the caller stops. Batch transcription instead of streaming adds 600–1,200 ms and ends the conversation.

The caller stops. The budget starts here.

The window that decides whether it feels human

Turn detection 150–300 ms. Deciding the caller has actually finished. Often the largest single lever.

LLM first token 150–400 ms. Only the first token matters; the rest streams out behind it.

TTS first audio 100–200 ms. Starts before the model has finished writing the reply.

First audio out, at roughly 600 ms end to end

Minimizing each stage

Turn detection is the sneaky one. Deciding that the caller has finished speaking forces a trade-off: wait longer to be certain and add latency, or respond quickly and risk cutting them off mid-sentence. Naive silence-timeout endpointing handles this badly. Better approaches combine voice activity detection, STT endpointing signals, and model-based classifiers that recognize when an utterance is actually complete [5]. At 150 to 300 milliseconds of the budget, this is frequently the biggest single lever you can pull.

STT must stream. Use streaming speech-to-text that returns partial transcripts as the caller speaks, not batch transcription that waits for them to finish. Batch transcription alone adds 600 to 1,200 milliseconds and makes low latency impossible; the choice of streaming versus batch matters more than the choice of STT model [6]. With streaming, transcription is essentially done the moment the caller stops, and fast streaming engines contribute on the order of 60 to 150 milliseconds [6][7].

The LLM metric is time-to-first-token, not total generation. Because the agent starts speaking as soon as the first words arrive, what matters is how quickly the model begins responding, and model choice affects latency mostly through time-to-first-token rather than raw quality [1]. Fast, low-latency models can begin producing tokens in 100 to 350 milliseconds, while larger models can take 700 to 800 milliseconds or more, so the model is chosen against the latency budget, not just the capability wish list [7]. Then stream the tokens out.

TTS should stream too. Use synthesis that begins producing audio before the full text is ready, so the reply starts playing as the first sentence is generated [3]. Fast engines deliver time-to-first-audio in the 40 to 150 millisecond range [6][8].

The discipline that ties all four together is streaming everywhere and overlapping the stages. The 600-millisecond figure is not one thing done quickly; it is four streaming layers, each tuned independently, running in parallel [2]. A pipeline that runs its stages sequentially, each waiting for the previous to finish, cannot hit a conversational budget no matter how fast the individual components are.

The architectural lever most teams miss

Even with fast components, many production stacks are slow, and the culprit is usually not a model, it is network topology. The largest single contributor in real deployments is often the LLM hop to a public cloud region, followed by the telephony handoff between SIP and the communications platform, and together these can consume the entire latency budget in round trips before the LLM has generated a single token [8]. The fix is architectural rather than prompt-level: co-locate inference infrastructure with the media and call path, minimize cross-data-center round trips, and use efficient real-time transport [8][5]. This is why pipeline architecture beats model choice for latency [6]: you cannot prompt your way out of a network problem.

Pipeline versus speech-to-speech

The cascaded STT-LLM-TTS pipeline is the most common pattern because it gives you control: you can swap components, add function calling and retrieval, and observe every stage. Speech-to-speech models take a different approach, collapsing recognition, reasoning, and synthesis into a single model that processes audio more directly, which shaves roughly 100 to 200 milliseconds and preserves more emotional nuance, at the cost of the modular control the pipeline offers [2]. Research in this direction is active and validates the underlying techniques: one recent telecommunications-focused system combines streaming ASR, a 4-bit quantized LLM, retrieval-augmented generation over domain documents, and real-time TTS, coordinated through sentence-level streaming and concurrent, non-blocking module execution to minimize end-to-end latency [9]. The right choice depends on how much modular control you need versus how much you are optimizing for the last few milliseconds and maximum naturalness.

The hard parts beyond the happy path

A demo only has to handle the ideal conversation. Production has to handle the rest.

Barge-in. When a caller talks over the agent, a good stack stops TTS within about 100 milliseconds, discards the rest of the planned reply, and starts a fresh STT stream on the caller’s new speech. A bad stack keeps talking, which is the worst feeling on a phone call [1]. Clean interruption handling is central to natural conversation and is genuinely hard to get right.

Retrieval and function calls mid-call. When the agent needs to look something up, in a knowledge base or a CRM, that work adds latency. Streaming the retrieval and caching results where context allows keeps it inside the budget [8], and grounding spoken answers in real content is the same discipline covered in the enterprise RAG guide.

Consistency under load. Hitting the target at the 95th and 99th percentile across many concurrent calls, not just on average in a quiet demo, is the real bar. Voice pipelines also fail silently, a dropped transport connection looks like the agent not hearing the caller, a synthesis timeout manifests as dead air, so per-component observability is essential: record a per-call latency histogram broken down into STT, LLM first-token, and TTS first-chunk, and alert when any single stage exceeds its threshold [6].

Build or buy the orchestration

A practical note on where to spend engineering effort. The orchestration layer, voice activity detection, turn-taking, barge-in, streaming coordination, and function-call routing, is where the serious engineering investment goes, and teams that try to build all of it from scratch often end up with a slower, worse version of what already exists [1]. The pragmatic path is to build on a real-time voice framework such as Pipecat that handles the orchestration, and to concentrate your own effort on what is proprietary to the business and on the last-mile work that actually decides quality: the prompt and knowledge base, the workflows and integrations, and the latency and reliability tuning that gets the whole loop consistently inside the conversational window. That last-mile tuning is unglamorous and it is exactly where a production voice agent is won or lost.

The bottom line

A voice agent that feels human is not one fast component; it is an entire pipeline engineered to stream and overlap so that the first audio comes back inside the roughly 300-to-500-millisecond window that reads as natural. The milliseconds hide in turn-taking, in time-to-first-token, and in network round trips, not where most teams look first. Getting it right is detailed, unglamorous engineering, streaming at every stage, co-located inference, smart endpointing, clean barge-in, and real observability, and it is precisely what separates a voice agent people talk to from one they hang up on.

---

Building or evaluating a voice agent and fighting latency? The wins are almost always in turn-taking, time-to-first-token, network topology, and streaming discipline, not in swapping models.

---

Sources

1. Retell AI (2026). How Real-Time Voice AI Actually Works (STT → LLM → TTS, Explained) (the pipeline and orchestration; latency hiding in turn-taking and time-to-first-token; barge-in handling; model choice and TTFT; build-versus-buy on orchestration). https://www.retellai.com/blog/how-real-time-voice-ai-works-stt-llm-tts
2. The Prompt Bench (2026). Latency Budgets for Real-Time Voice (the per-layer budget split; streaming in parallel; speech-to-speech savings; perception thresholds). https://thepromptbench.com/voice-and-realtime/latency-budgets-for-realtime-voice/
3. AssemblyAI (2026). Voice Agent Architecture: Build STT-LLM-TTS Pipeline (streaming pipeline; streaming everywhere; orchestration as the hardest part). https://www.assemblyai.com/blog/voice-agent-architecture
4. Nurix (2026). Best AI Voice Agents for Enterprise in 2026 (voice-agent fundamentals and sub-second targets). https://www.nurix.ai/blogs/best-ai-voice-agents-enterprise-2026
5. LiveKit (2026). Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained (the four sources of latency; co-locating inference; turn detection via VAD, endpointing, and classifiers; streaming STT). https://livekit.com/blog/voice-agent-architecture-stt-llm-tts-pipelines-explained
6. Prodinit (2026). Voice AI Latency: Sub-250ms Architecture Guide (component-level budgets; batch transcription penalty; per-component observability; perception thresholds). https://prodinit.com/blog/production-voice-ai-agents-latency-architecture
7. Introl (2026). Voice AI Infrastructure: Building Real-Time Speech Agents (LLM time-to-first-token tiers by model; component latency numbers). https://introl.com/blog/voice-ai-infrastructure-real-time-speech-agents-asr-tts-guide-2025
8. Telnyx (2026). Voice AI Agents Compared on Latency (measuring per-layer breakdown; the network round-trip and SIP handoff as the largest contributors; co-locating inference; caching retrieval). https://telnyx.com/resources/voice-ai-agents-compared-latency
9. Ethiraj, V., David, A., Menon, S., & Vijay, D. (2025). Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTS. arXiv:2508.04721. https://arxiv.org/abs/2508.04721