A voice agent should be evaluated as an interactive system, not as a transcript generator. Word error rate can reveal recognition failures, but it cannot tell whether the agent completed the task, waited appropriately, stopped when interrupted, preserved names and numbers, avoided unsafe actions, or gave the user control. Use a layered suite with annotated audio, task-state assertions, behavioral timing, and human review.
Define a failure taxonomy before scoring
Separate layers so one symptom is not blamed on the wrong component:
- Capture and transport: clipping, silence, loss, echo, reconnect failure.
- Recognition: deletion, insertion, substitution, speaker attribution, finalization.
- Dialogue: false endpoint, delayed response, bad interruption, lost context.
- Reasoning and tools: incorrect plan, bad arguments, duplicated side effect.
- Speech output: pronunciation, intelligibility, prosody, truncation.
- Product outcome: task completion, safety, accessibility, user control.
A wrong account number may begin as recognition failure and end as a harmful tool call. Record both causal layer and final impact.
Build a representative test corpus
Use audio that you are authorized to process under the applicable permission, contract, and licensing rules. Cover:
- supported languages, dialects, accents, speaking rates, and code-switching;
- names, addresses, dates, amounts, account identifiers, and domain terms;
- microphones, speakerphone, headset, mobile, and browser paths;
- quiet, vehicle, office, household, and competing-speech conditions;
- hesitations, corrections, pauses, backchannels, and interruptions;
- short commands and long explanations;
- users who use keyboard, captions, or push-to-talk alongside voice; and
- answerable, ambiguous, unauthorized, and impossible requests.
Do not create demographic labels by guessing from a voice. Collect attributes only for a documented evaluation purpose, through an explicit permission process where applicable, and after confirming the privacy and human-subject requirements for the deployment. Protect the data, and report uncertainty and sample sizes.
Keep speakers separated across development and held-out sets where speaker generalization matters. Otherwise the suite can reward adaptation to familiar voices.
Score recognition with task-aware slices
Word error rate is:
(substitutions + deletions + insertions) / reference words
NIST speech-recognition evaluation plans use WER with defined reference transcripts and scoring procedures; comparability depends on the data and normalization rules, not only the formula. See the NIST OpenASR20 evaluation plan.
Publish normalization rules for punctuation, casing, numbers, filled pauses, partial words, and overlapping speech. Add entity metrics for critical spans: exact account number, normalized currency value, date, name, or product code. One error in a critical identifier can matter more than several harmless function-word errors.
Evaluate partial and final transcripts separately. A correct final transcript arriving after the agent already acted does not repair the interaction.
Evaluate turn behavior directly
Annotate user speech onset and offset, intended interruptions, backchannels, and agent audible output. Measure:
- false endpoint rate;
- endpoint delay after reference turn end;
- false interruption rate;
- true interruption detection and audible stop delay;
- agent overlap with user speech;
- ignored backchannels and inappropriate responses to them; and
- work performed after a turn or response was canceled.
“Talking Turns” proposes evaluation of several turn-taking events and reports a human study alongside automatic methods. It supports evaluating interaction behavior beyond static transcripts, but its specific dataset and learned judge should not be assumed valid for every language or product. See Talking Turns, ICLR 2025.
Assert task state, not persuasive dialogue
For tool-using tasks, use a simulated or sandboxed backend with programmatic success predicates:
type VoiceScenario = {
initialState: unknown;
audioTurns: readonly string[];
expectedState: unknown;
forbiddenEffects: readonly string[];
requiresApproval: boolean;
};
Verify final state, tool arguments, number of writes, authorization, approval, and idempotency. A friendly spoken confirmation is not success if no reservation exists—or if two were created.
Inject ambiguous or conflicting speech and require clarification before consequential actions. Test network loss after the side effect but before the spoken confirmation.
Evaluate output audio separately
Use human listeners for intelligibility, pronunciation, naturalness, appropriateness, and fatigue, with randomized blinded samples. Include critical domain names and numbers. Automated audio metrics may help regression detection but should be validated against the judgment they are intended to proxy.
Check that captions match the actual spoken content and arrive soon enough to be useful. Test stop, mute, playback speed if offered, replay, device routing, and screen-reader interaction.
Include network and browser quality
Collect WebRTC statistics for packet loss, jitter, concealment, processing delay, and jitter-buffer behavior. The W3C specification defines these fields and their units; it does not define your product’s pass thresholds. See W3C WebRTC Statistics.
Run impairment experiments with declared profiles and preserve exact settings. Report sessions that fail to connect or produce no audio, not only quality among completed calls.
Use human review where meaning matters
Calibrate reviewers with examples and a rubric. Double-score a subset, measure agreement, and adjudicate important disagreements. Ask focused questions—“Did the agent stop when the user attempted to interrupt?”—rather than one overall impression that mixes many attributes.
Model-based judges can reduce cost after calibration, but voice-specific timing and audio defects may be absent from their text input. Never claim an audio system passed because a transcript-only judge liked the conversation.
Release gates and monitoring
Set thresholds by severity and cohort. A tiny improvement in average naturalness should not offset new unauthorized actions or a regression for one language. Gate on critical safety invariants, then task success, then interaction and performance budgets.
In production, monitor aggregate outcomes and opt-in reviewed samples with privacy controls. Convert real failure classes into redacted, reproducible fixtures. Re-run the suite after changing recognition, endpointing, dialogue policy, model, tools, speech synthesis, media transport, or browser code.