Voice agent architecture: speech-to-text, LLM, text-to-speech and the latency budget

Key points

• A cascade gives you text at every step and lets you swap each model; a speech-to-speech model removes hand-offs and hears the audio itself.
• Budget latency per stage: endpointing, final transcript, time to first token, time to first audio and the network. The caller hears the sum.
• Stream everything: partial transcripts in, tokens into text-to-speech as they arrive, audio out in small frames.
• Turn detection decides when to reply. A fixed silence window is simple but either cuts people off or adds delay; semantic turn detection also looks at the words.
• On barge-in, stop playback, cancel generation and trim the conversation history to what the caller actually heard.
• Measure word error rate on normalized text, plus task success and turn latency, on audio that sounds like your real calls.

Two architectures: cascaded and speech-to-speech

A cascaded voice agent chains three components. Speech-to-text (STT, also called automatic speech recognition) turns the caller's audio into text. A language model reads the transcript and the conversation so far, calls tools if it needs to, and writes a reply. Text-to-speech (TTS) turns the reply into audio. Each component can come from a different vendor or run on your own servers.

A speech-to-speech model takes audio in and produces audio out within one model and one session. OpenAI's Realtime API documentation describes a model that works directly with audio, maintains conversation state and can call tools, connected over WebRTC in the browser or WebSocket on the server. Google's Gemini Live API runs over a stateful WebSocket connection, takes 16-bit PCM audio at 16 kHz, returns audio at 24 kHz, lets users interrupt the model at any time, and supports function calling.

OpenAI's voice agents guide frames the choice clearly: a chained pipeline lets you inspect or transform the intermediate text and replace each component independently, while a speech-to-speech model interprets the audio, decides what to do and responds in speech by itself.

Mixed designs are common too: a speech-to-speech model for the conversation with text checks on its transcripts, or a cascade whose three components run in one region to cut network hops. Whether a voice agent is the right product in the first place is a separate question, covered in AI voice agents for business.

The latency budget and streaming at every stage

The number that matters is the silence between the moment the caller stops talking and the first sound of the reply. Human conversation sets a tight bar. Stivers and colleagues timed question-and-answer turns in 10 languages (PNAS 2009) and found that the most common gap in every language fell between 0 and 200 ms, with a mean of about 208 ms across their data; on average, responses were neither in overlap nor delayed by more than half a second. A voice agent will not match that, but every stage below adds to the gap.

The figures in that table are a planning example, not measurements of any product or provider. Set your own targets, then measure each stage separately in traces, at the median and the 95th percentile, because the slow turns are the ones callers remember. Tool calls come on top of this budget.

Streaming is what makes any such budget reachable. Each stage starts on partial input from the one before:

• STT streams partial transcripts while the caller speaks, so only the last word or two remain to finalize after the endpoint.
• The LLM streams tokens. Prompt processing grows with prompt length, so keep the system prompt and history lean and reuse a cached prefix where the provider supports it. LLM inference serving and batching explains time to first token and what affects it.
• TTS starts on the first sentence or clause while the LLM is still writing the rest. Cut at punctuation, not mid-word, or the intonation breaks.
• Audio goes out in small frames, tens of milliseconds each, and plays as it arrives.
• Some systems start the LLM speculatively on a likely-final transcript and discard the result if the caller keeps talking.

Voice activity detection and endpointing

Voice activity detection (VAD) classifies short frames of audio as speech or not speech. Endpointing, also called turn detection, decides when a run of speech has become a finished turn. The simplest endpointer waits for a fixed stretch of silence after speech. A short window makes the agent quick and also makes it cut people off when they pause to think; a long window makes it polite and slow.

OpenAI's Realtime API documentation describes both approaches. Server VAD, the default for speech-to-speech sessions, splits audio on silence. Its threshold sets how loud audio must be to count as speech (higher suits noisy environments), prefix_padding_ms keeps audio from just before speech was detected, and silence_duration_ms sets how much silence ends a turn, with shorter values detecting turns more quickly. Semantic VAD uses a classifier that scores how likely it is that the user has finished, based on their words, and waits longer after an utterance that trails off; its eagerness setting trades speed against patience.

• Tune on real audio from your channel. A threshold that works on a headset microphone can fail on a speakerphone in a moving car.
• Keep prefix padding. Without it the first syllable is clipped and STT mishears the first word.
• Different moments need different patience. After "What is your account number?" callers read digits with pauses, so widen the window for that turn.
• Plan for silence that goes on too long: prompt once, then end or transfer the call politely.

Barge-in: handling interruptions

Callers interrupt: to correct the agent, to answer before the question ends, or because the reply is running long. Supporting barge-in means the agent stops talking as soon as real speech starts and the conversation state reflects what the caller actually heard. Three things happen together: playback stops and queued audio is flushed, the in-flight LLM and TTS requests are cancelled, and the assistant message in history is cut to the words already played, so the model does not assume the caller heard a sentence that never reached them.

Not every sound is an interruption. Coughs, other voices in the room and backchannels such as "mm-hm" or "right" should not stop the agent. A common approach is a minimum length of voiced audio before a barge-in counts, plus echo cancellation so the agent never hears its own voice. The state machine below is library-agnostic: the VAD, the endpointer and the audio player send it events, and it returns the next state and triggers the side effects.

type State = 'listening' | 'userSpeaking' | 'thinking' | 'agentSpeaking';
type TurnEvent =
  | { type: 'voice'; voicedMs: number } // VAD: length of the current speech burst
  | { type: 'endOfTurn' }               // endpointer: the user is done
  | { type: 'firstAudio' } | { type: 'playbackDone' };
interface Effects { startReply(): void; cancelReply(): void; truncateHistory(): void }
const MIN_BARGE_IN_MS = 300; // illustrative: ignore coughs and "mm-hm"

function next(state: State, ev: TurnEvent, fx: Effects): State {
  switch (ev.type) {
    case 'voice':
      if (state === 'thinking' || state === 'agentSpeaking') {
        if (ev.voicedMs < MIN_BARGE_IN_MS) return state; // too short to count
        fx.cancelReply(); // abort LLM and TTS requests, flush queued audio
        if (state === 'agentSpeaking') fx.truncateHistory(); // keep what was heard
      }
      return 'userSpeaking';
    case 'endOfTurn':
      if (state !== 'userSpeaking') return state;
      fx.startReply(); // transcript to LLM, LLM tokens to streaming TTS
      return 'thinking';
    case 'firstAudio':
      return state === 'thinking' ? 'agentSpeaking' : state;
    case 'playbackDone':
      return state === 'agentSpeaking' ? 'listening' : state;
  }
}

The code compiles under TypeScript's strict mode. Running a scripted sequence through it: a caller turn, the reply starting, a 120 ms cough, then 500 ms of real speech. The cough left the agent speaking; the real speech cancelled the reply, truncated the history and moved to userSpeaking. Speech that arrives while the agent is still thinking cancels the pending reply too, so a caller who paused mid-sentence does not get an answer to half a question.

Truncation needs timing. Track how many milliseconds of each reply were played and map that to text, using the TTS provider's word timestamps where it returns them and an estimate where it does not. With a speech-to-speech API, check how the provider records an interrupted response in the session, and whether you have to tell it how much was played.

Telephony: WebRTC, SIP and the phone network

Where the audio comes from shapes the architecture. In a browser or mobile app, WebRTC is the usual transport. The W3C WebRTC specification defines the JavaScript APIs for sending media to, and receiving it from, another browser or device, and the IETF overview in RFC 8825 describes the protocols underneath: ICE to find a path through NATs, TURN relays when no direct path works, RTP for media, and SRTP encryption that every implementation must support. WebRTC media normally travels over UDP, so a late packet is dropped instead of resent, which suits live audio.

Phone calls arrive from the public switched telephone network (PSTN). A carrier or SIP trunking provider bridges the call into SIP, the signaling protocol that RFC 3261 defines for creating, modifying and terminating sessions such as internet telephone calls, with the audio carried over RTP. Your side is a media server or the provider's media stream, which forwards audio to the agent, often over a WebSocket. RFC 8825 notes that a signaling gateway between SIP and WebRTC can be built because both can express the same SDP offer and answer semantics.

• Phone audio is narrowband and compressed, so STT accuracy drops compared with a good browser microphone. Test with real phone recordings, and resample to the rate your model expects; the Gemini Live API, for example, takes 16 kHz input.
• The phone network adds delay you cannot remove. Run the agent close to the region where the carrier hands you the media.
• Callers press keys. Handle DTMF tones for menus and for digits a caller would rather not say aloud.
• Transfers to a person, voicemail detection on outbound calls and hold audio live in the telephony layer, not in the model.

Tool calls mid-conversation

A voice agent that books, looks up or changes something calls tools the same way a text agent does; how LLM tool calling works covers the protocol. Voice adds one constraint: silence while a tool runs sounds like a dropped call.

• Say something first. Let the model produce a short acknowledgement such as "Let me check that" before the tool call, or play a brief sound when a call runs past a set threshold.
• Keep tools fast and their results small. A large tool result adds prompt tokens to the next step and delays the reply.
• Decide what barge-in does to a running tool. Cancel a read-only lookup; let a write finish, then tell the caller what happened.
• Read back anything irreversible. Names, dates, amounts and addresses are where transcription errors hide, so confirm them before acting.
• Apply the same checks as in text. In a cascade you can check the reply before TTS speaks it; see LLM guardrails architecture.

Evaluating a voice agent: word error rate and task success

Speech recognition is usually scored with word error rate (WER). Align the recognized words with a reference transcript using the fewest edits, count the substitutions (S), deletions (D) and insertions (I), and divide by the number of words in the reference (N): WER = (S + D + I) / N. Because insertions count, WER can exceed 100 percent.

Normalization changes the score a lot. The Whisper authors (Radford and colleagues, 2022) point out that WER penalizes all differences between output and reference, including innocuous differences in transcript style, so a transcript a person would judge correct can still score badly. They built a text normalizer for that reason, which among other steps rewrites spoken numbers as digits, and report WER drops of up to 50 percent on some datasets, usually from quirks such as how a dataset's reference transcripts split contractions. Here is WER with and without a simple normalization step; the outputs in the comments are what it prints.

def normalize(text):
    return ''.join(c for c in text.lower() if c.isalnum() or c.isspace()).split()

def wer(ref, hyp):
    # Each cell: (edits, substitutions, deletions, insertions)
    prev = [(j, 0, 0, j) for j in range(len(hyp) + 1)]
    for i in range(1, len(ref) + 1):
        row = [(i, 0, i, 0)]
        for j in range(1, len(hyp) + 1):
            if ref[i - 1] == hyp[j - 1]:
                row.append(prev[j - 1])
                continue
            e, s, d, n = prev[j - 1]; sub = (e + 1, s + 1, d, n)
            e, s, d, n = prev[j];     dele = (e + 1, s, d + 1, n)
            e, s, d, n = row[j - 1];  ins = (e + 1, s, d, n + 1)
            row.append(min(sub, dele, ins))
        prev = row
    edits, s, d, n = prev[-1]
    return round(edits / len(ref), 3), {'S': s, 'D': d, 'I': n}

ref = 'Book a table for two at seven tonight.'
hyp = 'book a table for to at 7 tonight please'
print(wer(ref.split(), hyp.split()))       # (0.625, {'S': 4, 'D': 0, 'I': 1})
print(wer(normalize(ref), normalize(hyp)))  # (0.375, {'S': 2, 'D': 0, 'I': 1})

Lowercasing and stripping punctuation removes two false errors, "Book" against "book" and "tonight." against "tonight", and takes WER from 0.625 to 0.375. "two" heard as "to" is a real error. "seven" against "7" is a formatting difference that a fuller normalizer, like Whisper's, maps to one form.

WER is a component metric. What callers notice is whether the call worked:

Test on audio that sounds like production: phone-quality recordings, your callers' accents, background noise, and people who talk over the agent. Simulated callers, a second model speaking through TTS, are useful for regression tests but sound cleaner than real people, so keep a set of real recordings, collected with consent, as well.

Failure modes in real calls

• Crosstalk and echo: on a speakerphone the agent's own voice comes back through the caller's microphone, and without echo cancellation the agent interrupts itself. Other voices in the room can trigger barge-in or end up in the transcript.
• Background noise: traffic, a television or wind can keep a silence-based endpointer from ever hearing silence, so the agent never replies, or can be mistaken for speech. Raise the VAD threshold for noisy channels and set a maximum turn length.
• Accents, dialects and mixed languages: error rates rise for speech that is rare in the STT's training data. The Whisper authors found a strong correlation between a language's word error rate and the amount of training data for it. Measure WER per group of callers, not only overall.
• Domain words: product, drug and street names get mistranscribed. Use custom vocabulary or prompt hints where the STT offers them, and confirm critical values by reading them back.
• Pauses mid-thought: a caller who stops to find a card in their wallet gets cut off by an eager endpointer. Use semantic turn detection, or a longer window when you ask for something people have to look up.
• Mispronunciation: TTS reads numbers, abbreviations, dates and email addresses wrongly. Rewrite text for speech before TTS, for example reading an order ID digit by digit.
• Dropped connections: keep conversation state outside the media connection so a reconnect can resume, and end calls cleanly when audio stops arriving.

Consent, recording and AI disclosure

Voice agents record and process people's voices, and the rules depend on where the caller and the business are. This is not legal advice; treat the points below as questions to raise with counsel for each region you serve.

• Call recording: some places let one party to a call consent to recording it, others require every party to agree. Announcing the recording at the start of the call is common practice.
• AI disclosure: under the EU AI Act, people must be told they are interacting with an AI system unless that is obvious; see the EU AI Act for AI apps. Telling callers up front is good practice everywhere.
• Outbound calls in the United States: in a February 2024 declaratory ruling, the FCC found that the Telephone Consumer Protection Act's restrictions on "artificial or prerecorded voice" cover current AI technologies that generate human voices, so such calls need the called party's prior express consent unless an emergency purpose or an exemption applies.
• Voice data: recordings and transcripts can contain health, payment or identity details. Decide how long to keep them, redact sensitive fields from transcripts and logs, and check whether voiceprints count as biometric data where you operate.

If you are planning a voice agent for phone calls or inside your app, see our AI agent development service: the code lives in your GitHub from day one, AI-assisted work is reviewed by an engineer, and the founders are on the calls.