Measure Voice-Agent Latency Correctly: From Speech End to Audible Response

Voice-agent latency is not one duration. Measure a sequence of user-visible and system milestones: when speech actually ended, when endpointing committed the turn, when the transcript stabilized, when generation began, when first audio arrived, and when the browser made it audible. A single server timer hides endpointing delay, network buffering, canceled work, and playback gaps.

Define the timeline before collecting metrics

Use turn and response IDs plus explicit timestamps:

type VoiceTurnTiming = {
  sessionId: string;
  turnId: string;
  responseId: string;
  speechStart?: number;
  speechEndObserved?: number;
  endpointCommitted?: number;
  transcriptFinal?: number;
  responseRequested?: number;
  firstResponseToken?: number;
  firstAudioReceived?: number;
  firstAudioPlayed?: number;
  responseCanceled?: number;
};

Each field needs an owner and clock domain. Browser performance.now() is monotonic relative to performance.timeOrigin; server wall clocks require synchronization and still should not be subtracted blindly from browser monotonic timestamps. Either calculate durations within one process or propagate spans and clock-offset uncertainty.

Do not call a partial transcript timestamp “speech end.” It is an application event that depends on buffering and recognizer behavior.

Report several latency intervals

The intervals answer different questions:

The “observed speech end” itself is an estimate. Record whether it came from client VAD, server VAD, transcript events, or annotated evaluation audio. For controlled benchmarks, human-annotated boundaries can provide a reference; in production, label the operational estimate rather than presenting it as ground truth.

Instrument the browser playback boundary

Receiving bytes is not the same as hearing sound. Playback may wait for a jitter buffer, decoder, suspended audio context, muted element, or device route. Record when the remote track arrives and when application-visible playback begins, but document the precision the browser API actually provides. The WebRTC Recommendation defines the browser connection model; it does not expose the exact instant sound leaves a physical speaker.

WebRTC’s statistics model exposes cumulative metrics such as jitterBufferDelay, jitterBufferEmittedCount, totalProcessingDelay, packets lost, concealment events, and round-trip time where available. Average jitter-buffer delay is computed from cumulative delay divided by emitted count, so calculate deltas over the measurement interval rather than treating one cumulative value as a per-turn duration. See W3C WebRTC Statistics.

const report = await peer.getStats();
for (const stat of report.values()) {
  if (stat.type === "inbound-rtp" && stat.kind === "audio") {
    recordMediaSample({
      timestamp: stat.timestamp,
      packetsLost: stat.packetsLost,
      jitter: stat.jitter,
      jitterBufferDelay: stat.jitterBufferDelay,
      jitterBufferEmittedCount: stat.jitterBufferEmittedCount,
      concealedSamples: stat.concealedSamples,
    });
  }
}

Fields vary with browser support and whether the underlying object exists. Feature-detect them and never replace missing values with zero.

Correlate without recording conversation content

Propagate opaque session, turn, response, and trace IDs through browser events, signaling, transcription, model work, tool calls, audio generation, and cancellation. Record event type, timestamp, status, and carefully bounded dimensions such as client family or network class.

Raw transcripts and audio are not required to calculate latency. Keeping them by default increases privacy and security exposure and may trigger additional retention or notice requirements. If a reviewed debugging sample needs content, separate it from ordinary metrics, restrict access, set short retention, and record its purpose and the documented authorization for that use. Confirm applicable requirements for the deployment rather than inferring them from this engineering pattern.

Avoid high-cardinality metric labels such as user ID, session ID, full model response ID, or error text. Put per-run identifiers in traces or logs with controlled retention; aggregate metrics by stable, low-cardinality cohorts.

Design a controlled latency experiment

Use fixed audio clips with annotated speech boundaries and replay them through the supported input path. Vary one factor at a time:

• network delay, jitter, and packet loss;
• short versus long utterances;
• endpointing policy;
• no-tool versus tool-using responses;
• warm versus cold connection;
• concurrent sessions and queue depth;
• playback device and browser; and
• interruption point.

Measure at least p50, p95, p99, failure rate, canceled turns, false endpoints, and turns with no audible response. Separate successful turns from retries while still reporting retry incidence; dropping failed turns makes the latency distribution look better than the product.

For production, compare cohorts but do not infer causality from observational telemetry alone. Mobile users may have different networks, devices, tasks, and interruption behavior at the same time.

Build a budget from the user backward

Start with a measured user-experience target, then allocate provisional budgets to endpointing, orchestration, inference, audio generation, transport, and playback. Alert on both the whole interval and each component. A component budget is diagnostic, not permission to miss the end-to-end objective because every service remained under its local number.

Track wasted latency too:

• generation started before a transcript revision and then canceled;
• tool work completed after interruption;
• audio arrived after its response was superseded;
• reconnect replayed an obsolete request; or
• playback was generated while the client was muted or disconnected.

These consume capacity even when excluded from perceived-latency charts.

Avoid misleading comparisons

• Do not compare systems using different endpoint definitions.
• Do not mix first token, first audio byte, and first audible output.
• Do not omit cold starts or failed sessions.
• Do not report a mean without tail percentiles and sample count.
• Do not compare network regions without controlling client location.
• Do not call cumulative WebRTC stats per-turn values.
• Do not claim improvement without a confidence interval and unchanged workload.