Turn Detection and Interruption Handling in Voice Agents

Users do not report turn detection problems as turn detection problems. They say the agent keeps cutting them off, or that it takes forever to respond, or simply that talking to it feels wrong. Those are the same bug seen from two directions, and it is the highest-leverage thing to fix in most voice agents [1].

The two failure modes [2]:

The interrupter. A caller pauses half a second mid-sentence to check a date, and the agent jumps in. The caller has to start over, and the conversation now has a collision to recover from.

The laggy robot. The agent waits to be certain the caller finished. On a phone call, 400 milliseconds of silence after someone stops talking already reads as lag, and a full second feels broken.

The middle ground is narrow because real speech is messy. People pause to think, say “umm,” trail off, cough, get interrupted by background noise, and call from poor connections [2].

This guide covers how to navigate that. It builds on our voice AI agents guide, and is scoped apart from how voice AI latency works and voice agent telephony.

Terminology, because the field uses six names for one thing

Searching for help here is harder than it should be, since providers each named the capability differently. Endpointing, sometimes phrase endpointing, is the classic speech-recognition term. Semantic endpointing and semantic VAD are the two labels most common today for deciding turn completion from linguistic content rather than silence alone. Others call it end-of-turn (EOT) detection, end-of-utterance (EOU) detection, or just turn detection [3].

They are all answering the same question: has the caller actually finished?

Turn detection and interruption handling are different problems

This distinction matters more than any threshold value. Turn detection decides when the system thinks speech started or ended. Interruption handling decides what the agent does with that signal while the agent is already speaking [1].

Conflating them produces the single most common design failure: treating barge-in as a boolean. Turn it on and callers can interrupt; turn it off and they cannot. Production is messier than that [1], and some provider APIs already reflect the split by exposing separate controls for whether caller input stops playback and whether the application receives input during agent speech [1].

The detection layers

Production systems stack three layers, each doing a job the others cannot.

Acoustic voice activity detection. Classifies whether audio contains speech. The first cut is an energy threshold: frames below roughly -45 dBFS treated as silence, frames above roughly -35 dBFS treated as candidates, with the exact values depending on codec, microphone, and call conditions [4]. Treat those numbers as a starting point to calibrate, not a default to adopt. VAD’s job is opening the microphone and rejecting noise [3].

Silence-based endpointing. Heuristics layered over VAD that declare a turn over after a silence duration [5]. This is still what the majority of production voice agents in 2026 use [6], and it is the source of most turn-taking complaints, because silence duration is a poor proxy for whether someone finished a thought.

Semantic or model-based turn detection. Rather than classifying whether audio contains speech, this predicts whether the utterance is meaningfully complete, using lexical content, syntactic structure, intonation, and the filler words actually present [3]. The canonical example: a pause in the middle of a phone number stays open, while a pause after “that’s all, thanks” ends the turn [7].

The layers are complementary, not competing. Acoustic VAD is the right tool for opening a microphone and rejecting noise; semantic detection is the right tool for deciding a turn has ended [3]. Model-based detection can be the most accurate and costs more compute and tuning [5].

The direction of travel is clear. The production stack is migrating toward dedicated turn-taking models that classify backchannel, barge-in, and continued silence as a learned signal instead of an energy threshold, with implementations available in the major agent frameworks and as provider modes [4][3].

Barge-in is not one event

Before tuning any threshold, separate the things that look identical to a naive detector [1]:

Pure acoustic VAD handles the second row particularly badly, treating a backchannel as either silence or a full barge-in attempt, and neither is right [4]. Backchannel words differ by language and culture, which makes this worse for non-English agents.

The word-count heuristic helps here: requiring a minimum number of words before accepting an interruption filters most single-word backchannels while still letting a real correction through quickly.

Barge-in requires the detector to stay awake

Handling interruptions correctly means keeping the turn detection layer active while the agent is playing audio. When caller speech is detected during output, the agent cancels the current synthesis stream and hands control back to the recognition pipeline immediately [5].

This is harder than it sounds, mostly because of echo. Echo cancellation has to happen on the client side, where the speaker is [5]. Devices with built-in cancellation filter the agent’s own audio before it reaches your pipeline. Devices without it need a fallback, such as push-to-talk [5]. On telephony paths the echo situation differs again, which is one more reason to test on real calls rather than in a browser.

Cancelling synthesis also raises a state question most teams get wrong: what does the agent believe it said? If the caller interrupted after the first sentence of a three-sentence answer, only the first sentence was heard, and the conversation history should reflect that rather than the full generated text. Otherwise the agent will later reference information the caller never received.

Make thresholds context-aware

A single global silence threshold is the wrong shape for the problem, because how long someone pauses depends entirely on what they were asked.

Expected answer type. Someone reciting a phone number, email address, or account number pauses inside the utterance. Some providers now wait longer automatically when a tool parameter expects that kind of value [7]. If yours does not, you can do the same thing yourself: widen thresholds when the agent has just asked for structured data, tighten them for yes/no questions.

Workflow risk. Interruption policy should differ by what the agent is doing [1]. A legal disclosure may need to complete. A payment confirmation may need the caller to hear the full amount before anything cancels. An urgent support path should let the caller interrupt anything at any time. One global barge-in setting cannot express that.

Mid-tool interruption. If the caller interrupts while a tool call is running, what happens to the tool call? A reasonable default is that idempotent reads complete and mutations cancel when the caller contradicts them [4]. Whatever you pick, put it in code where it gets reviewed, not in a config flag someone flips without thinking about the semantics.

Speaker adaptation. Pace varies enormously between people. Systems that adapt thresholds to the individual speaker over the course of a call handle that better than any fixed value [7].

Multilingual turn-taking needs its own tuning

Thresholds and models tuned on English prosody misfire elsewhere. Pause patterns, sentence structure, and filler words differ by language, and code-switching speakers who mix two languages mid-sentence break assumptions in both.

Two specific traps. First, backchannel vocabulary is language-specific, so a filter built on English backchannels will let other languages’ equivalents through as false interruptions. Second, generating on partial transcripts is risky for code-switching, because partial hypotheses in mixed-language speech are frequently revised mid-utterance, so eager generation ends up discarded and repeated more often than it saves time.

The practical implication is to tune and measure per language rather than globally, and to treat a working English configuration as a starting point rather than a baseline you can port.

What to log

Turn-taking problems are impossible to debug from a transcript alone, because the transcript does not record what the agent believed was happening. Capture per interruption event [1]:

• Caller speech state and agent speech state at the moment of detection
• The interruption decision and which layer made it
• The playback action taken (continue, cancel, fade)
• What the caller actually said, and whether it was a correction, backchannel, or noise
• The transcript outcome and whether recovery succeeded
• Which threshold configuration was in effect

Alongside that, track aggregate signals that tell you which direction you are failing in: interruption rate, how often callers repeat themselves, dead air duration after end of speech, and where in calls people hang up. Instrumentation approach in LLM observability and tracing; alerting in monitoring LLM quality in production.

Test both directions separately

False positives and false negatives feel completely different to a caller and need separate test sets [1].

False positive tests (agent interrupts wrongly): mid-sentence pauses, recited numbers and addresses, thinking sounds and fillers, background conversation, coughs and door noise, backchannels in every language you support.

False negative tests (agent fails to yield): a caller correcting the agent mid-sentence, a caller saying stop, a caller answering before the agent finishes the question, urgent interruptions during a long response.

Run both against real call recordings rather than clean studio audio, and segment results by language, device type, and transport path. A configuration that scores well on browser WebRTC can fail on narrowband telephony audio, which is covered in voice agent telephony.

Then treat the review loop the way you would any other quality problem: sample real calls, categorize what went wrong, and let the counts tell you which threshold to move. The method is the same as error analysis for LLM failures, applied to turn-taking events rather than text outputs.

Common mistakes

• Treating barge-in as a single on/off setting.
• Tuning thresholds by feel instead of by measured interruption and repeat rates.
• Disabling a turn-detection model because it adds latency, rather than loosening its sensitivity.
• Using one global silence threshold regardless of what was asked.
• Ignoring backchannels, so “mhm” cancels the agent mid-sentence.
• Leaving conversation history as the full generated text after an interruption.
• Handling echo cancellation server-side, or assuming every device does it.
• Testing only false positives, or only false negatives.
• Porting an English configuration to another language unchanged.
• Validating on clean audio rather than real calls.

Checklist

• Turn detection and interruption policy are designed and configured separately.
• Acoustic VAD, endpointing, and semantic or model-based detection each have a defined role.
• Interruption causes are classified, not collapsed into one event.
• Backchannels are filtered, with vocabulary per supported language.
• Turn detection stays active during agent playback.
• Echo cancellation is handled client-side, with a fallback for devices without it.
• Conversation history reflects what the caller actually heard.
• Thresholds vary by expected answer type and workflow risk.
• Mid-tool interruption semantics are defined in reviewed code.
• Interruption events are logged with full state, and both failure directions are tested per language.

Keep reading

• Voice AI agents for the enterprise (the pillar for this series)
• How voice AI latency works
• Telephony for voice agents: SIP, PSTN, and the narrowband problem
• Voice AI: build vs buy
• Error analysis for LLM failures

CMW AI Labs builds production voice agents, including multilingual and code-switching deployments where English-tuned turn detection does not transfer. If callers say your agent talks over them or feels slow, talk to our team.

Sources

1. Hamming AI, “Voice Agent Interruption Handling: Barge-In, Backchannels, and Turn Detection.” https://hamming.ai/resources/voice-agent-interruption-handling-runbook
2. Cekura, “What Is Endpointing in Voice AI? A Guide to Turn Detection,” June 2026. https://www.cekura.ai/blogs/endpointing-in-voice-ai-turn-detection
3. Gradium, “Semantic VAD for Voice Agents: Turn Detection 2026,” June 2026. https://gradium.ai/content/semantic-vad-voice-agents-turn-detection-2026
4. Future AGI, “Voice AI Barge-In and Turn-Taking: A 2026 Implementation Guide,” May 2026. https://futureagi.com/blog/voice-ai-barge-in-turn-taking-2026/
5. LiveKit, “Turn detection for voice agents: VAD, endpointing, and model-based detection,” February 2026. https://livekit.com/blog/turn-detection-voice-agents-vad-endpointing-model-based-detection
6. Gradium, “Turn-Taking in Voice Agents: Why VAD Is Hard,” May 2026. https://gradium.ai/content/turn-taking-voice-agents-vad
7. AssemblyAI, “How intelligent turn detection (endpointing) solves the biggest challenge in voice agent development,” August 2026. https://www.assemblyai.com/blog/turn-detection-endpointing-voice-agent