Voice-agent turn taking is a decision under uncertainty, not a silence timer. Acoustic activity can indicate speech, but it cannot by itself determine whether a user finished a thought, paused mid-sentence, offered a backchannel, spoke to someone else, or intended to interrupt the agent. A robust system combines acoustic, linguistic, dialogue-state, and explicit user controls, then measures both delay and social errors.
Separate the decisions
Do not collapse these into one VAD boolean:
- Speech activity: is speech-like audio present?
- End of turn: has the current speaker yielded the floor?
- Interruption intent: should current agent output stop?
- Response readiness: is there enough stable meaning to begin work?
- Floor ownership: who should be audible now?
Voice activity detection helps with the first question. The other decisions require context and policy.
Research benchmarks support treating turn behavior as multidimensional. “Talking Turns” evaluates prediction and performance of events such as turn transitions, interruption, and backchanneling rather than using only transcript accuracy. See the ICLR 2025 paper. Its results concern evaluated systems and datasets, not a universal threshold for production products.
Start with a conservative state machine
type FloorState =
| "user_speaking"
| "user_maybe_done"
| "agent_thinking"
| "agent_speaking"
| "overlap"
| "paused";
type TurnSignal = {
voiceActive: boolean;
partialTranscript: string;
transcriptStable: boolean;
silenceMs: number;
agentOutputActive: boolean;
explicitControl?: "send" | "interrupt" | "hold";
};
An explicit push-to-talk or send control should override inference. It is the accessibility and reliability baseline, not an embarrassing fallback.
When the user becomes quiet, move to user_maybe_done, not directly to agent_speaking. Wait for an endpoint decision based on silence, syntactic and semantic completeness, task state, and the cost of being wrong. A short command can end quickly; dictating an address or medical history deserves more patience.
Use layered endpointing
A practical endpoint policy can combine:
- acoustic evidence: speech probability and duration of quiet;
- linguistic evidence: whether the partial utterance appears incomplete;
- dialogue evidence: whether required fields are still missing;
- interaction evidence: user’s historical pace within this session; and
- explicit control: send, hold, push-to-talk, or keyboard shortcut.
Use two deadlines: an early boundary when evidence strongly indicates completion and a maximum wait that prevents indefinite silence. Tune them per language, acoustic environment, and task. Do not publish one global millisecond value as human conversation truth.
If partial transcription changes materially after work begins, cancel or revise downstream work before any side effect. Transcript finality is a service contract; confirm what “final” means for the selected recognizer.
Distinguish interruption from backchannel
While the agent speaks, detected user audio may be:
- a true correction: “No, cancel that”;
- a floor-taking request: “Wait”;
- a cooperative backchannel: “mm-hmm”;
- environmental or third-party speech; or
- echo from the agent itself.
Immediate cancellation minimizes interruption latency but produces false stops. Waiting for a semantic transcript reduces false stops but may make the agent talk over the user. Use a two-stage policy:
- attenuate or pause playback quickly when confident foreground speech begins;
- confirm interruption intent from duration, transcript, speaker evidence, or an explicit control;
- either cancel the response or resume if the event was a backchannel/noise.
Always provide an immediate manual interrupt control. For irreversible workflows, speech interruption should also cancel pending actions or return them to approval state; stopping audio alone is not cancellation.
Handle overlap as data, not a transport error
Full-duplex audio allows both sides to speak, but the product needs a policy for transcription, echo cancellation, playback attenuation, and which stream is presented as current. Record overlap intervals and attribute transcripts to speaker channels when technically and legally appropriate.
The browser Media Capture specification exposes echo-cancellation and noise-suppression constraints, but requested constraints are not guaranteed settings. Inspect actual track settings and test on speakers, headsets, mobile devices, and poor networks. See Media Capture and Streams.
Preserve turn identity through cancellation
Assign IDs to input turns, transcript revisions, agent responses, and tool work:
type Turn = {
turnId: string;
transcriptRevision: number;
responseId?: string;
state: "open" | "committed" | "canceled";
};
Every response event should name the turn and response it belongs to. After interruption, ignore late playback and transcript events for the canceled response. Propagate cancellation to model generation and safe tool work, but do not assume a canceled network request reversed a completed side effect.
Evaluate errors in both directions
Build a two-channel test set containing:
- short and long within-turn pauses;
- self-corrections and hesitations;
- backchannels;
- explicit interruptions at different response positions;
- background and third-party speech;
- accents, languages, code-switching, names, and numbers;
- device echo and packet impairment; and
- users who cannot or prefer not to use speech continuously.
Measure:
- false endpoints per user turn;
- endpoint delay after a true turn end;
- false interruptions per agent response;
- time from true interruption onset to audible stop;
- backchannel preservation;
- transcript revision after response start;
- overlap duration; and
- task success and user-rated control.
Report distributions and cohorts. Average endpoint latency can hide a tail where the system repeatedly cuts off slower speakers.
The 2026 TurnNat paper proposes a learned naturalness measure over two-speaker voice-activity states and validates controlled perturbations with human judgments. It is a useful research direction, but should complement—not replace—task-specific behavioral metrics and human review. See TurnNat.
Choose simpler interaction when it is safer
Use push-to-talk, explicit send, or half-duplex interaction for noisy environments, high-consequence data entry, accessibility preferences, or devices where echo makes automatic interruption unreliable. Naturalness is not more important than user control.