Build a Real-Time Voice Agent over WebRTC

A browser voice agent over WebRTC needs three separate systems: your application authenticates the user and authorizes a short-lived session, signaling negotiates the connection, and the peer connection carries audio and optional application events. Treating those as one opaque “connect” call makes reconnection, permission handling, security review, and latency diagnosis unnecessarily difficult.

This guide is provider-neutral. Its state machine and browser examples are architectural; server signaling and model-specific session formats must follow the selected service’s documented protocol.

Draw the boundary first

browser ── authenticated HTTPS ──> application session endpoint
browser <── ephemeral session capability ── application
browser ── offer/answer + ICE signaling ──> realtime service
browser <════ audio + data channel over WebRTC ════> realtime service

The application backend may mint or broker a short-lived, narrowly scoped session capability. It should not send a durable provider credential to the browser. Bind the capability to the intended realtime resource, user, expiry, and allowed operations where the service supports those controls.

WebRTC defines RTCPeerConnection and an RTP media API for sending and receiving MediaStreamTrack objects. It does not define your application’s signaling transport; applications exchange offers, answers, and ICE information through their own protocol. See the W3C WebRTC Recommendation.

Use an explicit client state machine

Model the lifecycle instead of inferring it from button labels:

type VoiceState =
  | "idle"
  | "requesting_permission"
  | "authorizing"
  | "negotiating"
  | "connected"
  | "reconnecting"
  | "stopping"
  | "failed";

Each transition should have one owner and cleanup path. A stale asynchronous result must not revive a session after the user pressed stop. Keep a monotonically increasing attempt ID or an AbortController, and ignore results from superseded attempts.

Do not automatically request the microphone on page load. Capture is a user-visible capability with privacy consequences, and browsers require an express permission flow. The Media Capture specification also requires indicators for access and active recording. Browser permission is a technical prerequisite, not proof that every legal notice or consent requirement has been satisfied. See Media Capture and Streams.

Acquire the narrowest useful audio track

Start with few required constraints:

const localStream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true,
  },
  video: false,
});

const [microphone] = localStream.getAudioTracks();
if (!microphone) throw new Error("No microphone track was returned");

These values are requests, not proof of support or resulting quality. The W3C specification distinguishes capabilities, constraints, and current settings, and recommends using required constraints sparingly so the user agent has room to select a usable source. Inspect microphone.getSettings() when diagnostics require the actual settings.

Handle permission denial, missing devices, unsatisfied constraints, track ending, and a device being removed. Explain the recovery action without pressuring the user to grant permission.

Negotiate audio and application events

const peer = new RTCPeerConnection({ iceServers });
peer.addTrack(microphone, localStream);

const events = peer.createDataChannel("agent-events", { ordered: true });
peer.ontrack = ({ streams }) => {
  const [remoteStream] = streams;
  if (remoteStream) remoteAudio.srcObject = remoteStream;
};

const offer = await peer.createOffer();
await peer.setLocalDescription(offer);

const answer = await exchangeDescription(peer.localDescription);
await peer.setRemoteDescription(answer);

Some services create the data channel or require additional negotiation fields. Follow the service contract rather than assuming this exact direction.

Send typed events with a version and unique ID:

type ClientEvent =
  | { version: 1; id: string; type: "response.cancel" }
  | { version: 1; id: string; type: "session.stop" };

Validate inbound events before updating UI. Text received over a data channel is untrusted network input even when the media connection is encrypted.

Implement stop and interruption separately

Interruption cancels current agent output while keeping the conversation alive. Stop closes the session and releases the microphone. Conflating them causes accidental disconnects or continued capture after the UI appears stopped.

On stop:

1. prevent new events;
2. stop every local media track;
3. close data channels;
4. close the peer connection;
5. detach media elements;
6. expire application session state; and
7. make the UI visibly idle.

Stopping tracks turns off capture at the browser source; muting an element or setting a UI flag does not.

For interruption, send the service’s cancel event, stop queued local playback if applicable, and wait for an acknowledgment or authoritative new response state. Tag audio and transcripts with response IDs so late packets or events from a canceled response are not rendered as current.

Reconnect by creating a new connection

Observe connectionState, iceConnectionState, data-channel state, and track events. Brief network changes may recover through ICE behavior, but an application should put a ceiling on reconnect time and attempts.

After terminal failure, create a fresh RTCPeerConnection and obtain new short-lived authorization. Decide whether conversation state is server-resumable, client-replayed, or intentionally lost. Never replay an audio buffer or side-effecting instruction merely because signaling was retried.

Use exponential backoff with jitter for automatic retries, but require user action after permission denial or policy failure. Those conditions are not transient network errors.

Observe both application and media layers

Record timestamps for permission request, authorization, local description, answer receipt, connected state, first input audio, transcript milestones, response creation, first output audio, interruption, and close. Use response and turn IDs to correlate application events.

The WebRTC Stats specification defines browser-accessible metrics for packets, jitter, round-trip time, processing delay, concealment, and jitter-buffer delay. Its values are cumulative in many cases, so compute interval deltas and preserve units. See W3C WebRTC Statistics.

Do not record raw audio or transcripts merely for observability. Apply purpose limitation, access control, short retention, and explicit user disclosure.

Production checklist

• Durable service credentials never reach the browser.
• Session authorization is short-lived and resource-scoped.
• Microphone capture begins from a clear user action.
• Permission, device, signaling, and transport failures have distinct states.
• Stop always releases local tracks and closes the peer connection.
• Interruption cancels one response without silently ending capture.
• Inbound data events are versioned and validated.
• Reconnect attempts are bounded and do not duplicate side effects.
• Media and application timings share session and turn identifiers.
• The UI always shows when the microphone is live.