What is a simulated caller, and why do scripts not work?

A simulated caller is an agent that phones your agent and plays a role. The property that matters is that it adapts in real time rather than replaying a fixed script. Scripted playback exercises the happy path and little else, while an adaptive caller changes what it says based on what your agent said, which is what exposes lost context, mishandled corrections and broken escalations. Structure each test as a persona, a goal, an expected outcome and the metrics to score, and the library becomes your regression suite.

What should you measure when testing a voice agent?

Lead with task completion, because it is the metric that maps to value and the one a client asks about; everything else is a diagnostic that explains why it moved. Around it, score transcription, turn-taking, latency at p50 through p99, conversation quality, safety and compliance, escalation and operational metrics such as transfer rate. Measure entity errors separately from overall word error rate: a mistake on "um" costs nothing, one on an account number or a medication name costs the whole interaction.

What acoustic conditions should voice agent testing cover?

The axis with no text equivalent, and where demo-quality agents fall apart. Cover the transport path, browser WebRTC against real telephony, since narrowband audio degrades recognition materially. Then background noise from a quiet room through to a vehicle, connection quality including packet loss, jitter and cellular handoff, device variation across handset, speakerphone, headset and car systems with and without echo cancellation, and speaker variation in pace, volume, accent and age.

Evaluating and Testing Voice Agents Before Real Callers Do

Q: Why can a voice agent not be tested like a chat feature?

Because there is no single correct response to assert against, the audio channel is itself a variable, and most real failures are conversational rather than per-response: context lost across turns, a correction mishandled, an escalation that never happened. The general evaluation discipline still carries over, but voice adds axes text does not have, and you cannot wait for production to explore them, because a bad phone call is harder to recover from than a bad chat response.

Q: How much voice agent evaluation can be automated?

Published practitioner guidance puts roughly 80% as automatable, covering latency, recognition accuracy and task completion, with the remainder needing human spot-checks for tone, genuine edge cases, and calibrating the automated grader itself. That last item is the one teams skip, and it matters most: a grader nobody checks against human judgment drifts, and every number it produces afterwards is unanchored.

A voice agent cannot be tested the way a chat feature can. There is no single correct response to assert against, the audio channel is itself a variable, and most real failures are conversational rather than per-response: context lost across turns, a correction mishandled, an escalation that never happened.

The general evaluation discipline still applies, and everything in our evals and LLMOps series carries over. What changes is that voice adds axes text does not have, and that you cannot wait for production to explore them, because a bad phone call is a worse experience than a bad chat response and a lot harder to recover from.

This guide covers how to build a voice testing practice that catches failures before callers do. It closes our series on production voice agents.

Layer the testing, because each layer has holes

A useful framing borrowed from aviation safety is the Swiss cheese model: each evaluation layer catches some failures and misses others, and the point is to stack them so the holes stop aligning [1].

Layer	Catches	Misses
Regression testing	Known functional failures	Only the ones you already thought of
Adversarial and environmental testing	Conditions you did not design for	Simulated adversity is not real adversity
Compliance testing	Regulatory and policy violations	Assumes good-faith callers
Production monitoring	Everything else	Only after it has happened at least once

Stacked, a failure that slips past regression gets caught in production, routed to human review, and turned into a regression case so it cannot recur [1]. That loop is the whole system. Any single layer on its own gives false confidence.

There is also a ceiling worth knowing about: teams relying on manual call-listening QA tend to plateau, because listening to calls does not scale past a certain volume, and moving beyond that plateau requires programmatic evaluation [1]. If your QA process is a person with headphones, that is where your quality will stop.

Simulated callers, and why scripts are not enough

The core tool of pre-production voice testing is the simulated caller: an agent that phones your agent and plays a role.

The important property is that it adapts in real time rather than replaying a fixed script [2]. Scripted playback exercises the happy path and little else. An adaptive simulated caller changes what it says based on what your agent said, which is what exposes lost context, mishandled corrections, and broken escalations [2].

Structure each test as a reusable unit combining a caller persona, the instructions and goal for that caller, the expected outcome, and the metrics to score [3]. A library of those becomes your regression suite, and the same discipline from regression testing AI features in CI applies: run it on every prompt, model, and retrieval change, and compare against a baseline rather than a vibe.

Build a persona library

Personas are how you get coverage of the conversations that actually break production rather than the ones you imagined [3].

Persona	What it tests
The interrupter	Barge-in handling, recovery after collision
The rambler	Endpointing under long pauses, context retention
The terse caller	Endpointing on one-word answers, over-eager prompting
The non-native or accented speaker	Recognition robustness, escalation when confidence drops
The code-switcher	Mixed-language handling, see multilingual voice agents
The caller in noise	Voice activity detection, false triggers
The confused caller	Clarification behavior, loop avoidance
The hostile or evasive caller	Tone under pressure, refusal handling, escalation
The correction-maker	Whether the agent updates state or keeps the stale value
The out-of-scope caller	Graceful boundaries rather than confabulation

Add the ones specific to your domain. A healthcare intake agent and an outbound collections agent fail in different ways, and the personas should reflect that.

Cover the acoustic axis

This is the dimension that has no text equivalent, and it is where demo-quality agents fall apart. Test across:

Transport path: browser WebRTC versus real telephony, since narrowband audio degrades recognition materially. See voice agent telephony.
Background noise: quiet room, open office, call centre, street, vehicle.
Connection quality: packet loss, jitter, cellular handoff, brief dropouts.
Device variation: handset, speakerphone, headset, car system, and devices with and without echo cancellation.
Speaker variation: pace, volume, accent, age.

Testing only on clean studio audio tells you almost nothing about how your agent performs on a call from a moving car, which is where a meaningful share of real calls come from.

Score more than word error rate

Recognition accuracy is a component metric, not an outcome. An agent can transcribe every word correctly and still fail the caller, and it can mis-transcribe a filler word and succeed completely.

Score across these dimensions, and weight them by what actually matters to the business:

Dimension	Example metrics
Transcription	Word error rate, error rate on critical entities such as numbers, names, identifiers
Turn-taking	Interruption rate, dead air after end of speech, caller repeat rate
Latency	End-of-speech to first audio at p50, p95, p99
Task performance	Task completion rate, correct data captured, tool-call success
Conversation quality	Instruction following, groundedness, appropriate tone
Safety and compliance	Disclosure delivered, consent captured, no unsafe or out-of-scope completions
Escalation	Correct trigger, context handed off, caller did not have to repeat
Operational	Call duration, transfer rate, abandonment point

Task completion is the metric to lead with. It is the one a client asks about and the one that maps to value. The rest are diagnostics that explain why it moved.

Weight entity errors separately from overall word error rate. A transcription error on “um” costs nothing; one on an account number or a medication name costs the entire interaction. An aggregate WER treats them identically, which is why an agent with a good WER can still be unusable.

Judges apply here too, with the same caveats

Scoring conversation quality, tone, and groundedness at scale means an LLM judge, and every caution from validating an LLM judge applies unchanged: validate it against labels from a domain expert, split train, dev, and test, report true positive and true negative rates rather than raw agreement, and re-validate when anything changes.

There is one voice-specific addition. Decide whether the judge grades the transcript or the audio. Grading the transcript is cheaper and misses everything about delivery: pace, mispronunciation, an awkward pause, a voice that sounds wrong reading a mixed-language sentence. Those are real quality problems that a perfect transcript hides. A practical split is to grade transcripts at volume and spot-check audio by hand.

Test adversarially

Beyond hard conversations, test callers who are actively trying to break things: attempts to extract the system prompt, to bypass identity verification, to push the agent outside its defined scope, or to elicit unsafe completions [3]. Add repeated-failure scenarios, where the same request fails several times in a row, since that is where agents tend to loop or degrade [5].

For agents handling payments, health information, or identity, this is not optional, and it overlaps directly with the governance requirements in enterprise AI governance.

Test escalation properly

Every phone agent eventually hands off to a human, and escalation is consistently under-tested. Cover the cases where the caller’s request falls outside scope, where the caller explicitly asks for a person, and where the agent has failed repeatedly, then verify the handoff actually happened and that context transferred with it [2].

Also test the path nobody specifies: what happens when no human is available. That branch gets exercised constantly in production and is usually undefined.

Test under load

Voice agents degrade differently under concurrency than chat features do. Media gateways are CPU-bound when transcoding, provider rate limits bite, and websocket connections drop under pressure. Run concurrency tests that reflect your actual peak, and watch for audio artifacts and reconnections rather than just error rates, because degradation shows up in the audio before it shows up in the logs.

Keep humans in the loop for the last part

Published practitioner guidance puts roughly 80% of voice evaluation as automatable, covering latency, recognition accuracy, and task completion, with the remainder needing human spot-checks: tone, genuine edge cases, and calibrating the automated grader itself [4].

That last item is the one teams skip. An automated grader that nobody checks against human judgment drifts, and every number it produces afterwards is unanchored.

The human cadence that works is the same as in error analysis for LLM failures: listen to a sample of real calls regularly, categorize what went wrong in your own words, group those notes into failure modes, and count them. The counts tell you which persona or acoustic condition to add to the suite next.

Close the loop

The testing practice is only as good as its feedback path:

Production monitoring flags a failure. See monitoring LLM quality in production.
A human listens to the call and categorizes what went wrong.
The scenario becomes a new persona or test case in the suite.
The suite runs on every change, so the failure cannot ship again [1].

Every production failure should make the next suite stronger. If your test suite has not grown in a month of live traffic, either nothing is going wrong or nobody is looking.

Common mistakes

Testing with scripted playback instead of adaptive simulated callers.
Evaluating only on clean audio, never on real telephony paths.
Leading with word error rate instead of task completion.
Treating all transcription errors as equivalent regardless of what they land on.
Grading transcripts only, so delivery problems stay invisible.
No adversarial testing on agents handling payments or identity.
Escalation tested only on the happy path, with no human-unavailable branch.
Skipping concurrency testing until the first busy day.
An automated grader nobody calibrates against human judgment.
A test suite that never grows from production failures.

Checklist

Test layers cover regression, adversarial, compliance, and production monitoring.
Simulated callers adapt in real time rather than replaying scripts.
A persona library covers interrupters, accents, noise, hostility, corrections, and out-of-scope requests.
Acoustic coverage includes real telephony, noise, packet loss, and device variation.
Scoring spans transcription, turn-taking, latency, task success, safety, and escalation.
Entity-level errors are measured separately from overall word error rate.
Any judge is validated against domain-expert labels and re-validated on change.
Adversarial and red-team scenarios run against agents handling sensitive actions.
Escalation is tested including the no-human-available path.
Concurrency testing reflects real peak load.
Human spot-checks cover tone and grader calibration.
Production failures are converted into regression cases.

Keep reading

CMW AI Labs builds production voice agents with evaluation suites from the first prototype: adaptive simulated callers, persona and acoustic coverage, and a loop that turns every production failure into a regression test. If your voice agent has never been tested against a hostile caller on a bad connection, talk to our team.

Sources

Coval, “Voice AI Agent Evaluation: The Complete Guide (2026).” https://www.coval.ai/blog/voice-ai-agent-evaluation-guide/
TestMu AI, “Voice Quality Testing: Complete Guide for VoIP and AI Voice Agents (2026),” August 2026. https://www.testmuai.com/blog/voice-quality-testing/
Cekura, “A Developer’s Guide to Voice AI Evaluation Metrics (2026),” May 2026. https://www.cekura.ai/blogs/voice-ai-evaluation-metrics
Famulor, “How to Test and Evaluate an AI Voice Agent in 2026,” July 2026. https://www.famulor.io/blog/how-to-test-and-evaluate-an-ai-voice-agent-in-2026
TestMu AI, “How to Test AI Calling Agents: The Practical Guide (2026),” June 2026. https://www.testmuai.com/blog/ai-calling-agent/