Voice AI and Realtime Agents

A voice agent is not a chatbot with a microphone attached. It is a soft realtime system that happens to contain language models, and the parts that make it hard are the parts that have nothing to do with the models.

Consider what has to be true for one exchange. Audio arrives continuously as small frames, tens of milliseconds each, over a connection that drops packets and delivers them out of order. Something has to decide, while the user is still speaking, whether they have finished. Something has to convert the audio to text or tokens fast enough that the decision is not stale. A language model has to start producing a reply before it has finished thinking. A speech synthesizer has to start speaking before the reply is complete. Audio has to flow back out through the same lossy connection, paced correctly, while the system continues listening in case the user interrupts. And if the user does interrupt, everything downstream has to be canceled and the record of the conversation corrected to reflect what was actually heard rather than what was generated.

None of that is a modeling problem. It is a streaming systems problem with a hard deadline, and the deadline is set by human conversational expectation rather than by any technical requirement.

The gap between this and batch speech work

People arrive at voice agents from two directions and both bring wrong instincts.

From the text agent side, the wrong instinct is that voice is an input and output format. In text, latency is annoying; in voice, latency is a conversational signal, and a two-second pause means something to a listener whether or not you intended it. In text, you can revise before sending. In voice, once a token has been synthesized and played, it has happened. In text, the user reads the whole reply. In voice, they interrupt halfway through, and the second half never existed as far as they are concerned even though your conversation history says it did.

From the audio production side, the wrong instinct is that quality is the objective. In batch narration you optimize the waveform, and you have unlimited time. In a conversation, time to first audio dominates perceived quality almost completely. A slightly less natural voice that begins speaking quickly is preferred to a beautiful one that pauses first, and the same is true of recognition: a fast model with a modestly higher error rate usually produces a better conversation than a slow, accurate one, because the errors are recoverable and the delay is not.

Every design decision in a voice agent is downstream of one number: how long the user waits between finishing their sentence and hearing the first syllable of the reply.

Human conversation sets the reference. A cross-linguistic study of turn-taking published in PNAS by Stivers and colleagues found that the gap between one speaker finishing and the next beginning clusters tightly around a fifth of a second across the languages sampled. No pipeline built from current components hits that. What matters is knowing how far over you are, and where the time went.

Where the time actually goes

Decompose the round trip into stages, because the intuitive answer is usually wrong.

Endpoint detection. The system has to conclude that the user stopped talking. If the rule is "silence for N milliseconds", then N is added to every single turn, unconditionally, before any processing begins. This is very often the largest single item in the budget, and it is the one least likely to be measured, because it does not appear in any model's benchmark and no vendor reports it.

Recognition finalization. Streaming recognizers emit partial hypotheses continuously and a final one after the endpoint. The final pass costs time, and some architectures cost much more of it than others.

Language model time to first token. Not total generation time. Once the first token is out, generation typically outruns speech, so the metric that matters is how long the model thinks before saying anything. Prompt length, cache state and whether the request had to establish a connection all show up here.

Speech synthesis time to first audio. Same principle. Total synthesis speed is nearly irrelevant if you stream at clause boundaries; time to the first chunk is everything.

Transport. Network round trips, jitter buffering, and the framing delay of whatever codec is in use. On a phone call this is not negligible, and it is not under your control.

What follows from the decomposition

Three consequences shape real systems. First, optimization effort belongs at the endpoint detector before anywhere else, because a smarter endpointing decision can remove more milliseconds than swapping every model in the stack. Second, streaming has to be end to end: a single stage that waits for complete input destroys the benefit of streaming everywhere else, and it is usually the synthesis stage buffering a full sentence. Third, you must instrument per stage with a shared correlation identifier and timestamps at every boundary, because without that you cannot tell whether last night's slow calls were recognition, the model, synthesis or the network, and a plausible-sounding guess is exactly the kind of answer that sends a week of work at the wrong stage.

Report percentiles rather than averages. Mean latency hides the tail, and it is the tail that users describe as "it keeps freezing".

Two architectures are in production use and the choice determines almost everything else you can and cannot do.

The cascaded pipeline chains three components: recognition converts audio to text, a language model produces a text reply, synthesis converts it back to audio. This is the older approach and it remains the default for good reasons. Every stage boundary produces text, and text is loggable, greppable, redactable, testable and cheap to store. You can put a guardrail between recognition and the model, and another between the model and synthesis. You can swap any component without touching the others. You can reuse the prompts, tools and evaluation harness from an existing text agent. When something goes wrong you can read what happened.

The speech-to-speech model takes audio in and produces audio out, with no text bottleneck in the middle. The commercial realtime APIs from the large providers work this way, and open work such as Kyutai's Moshi demonstrated genuine full-duplex conversation with the model able to listen and speak simultaneously. The advantages are real: lower latency because there are fewer stages, and preservation of paralinguistic information — tone, hesitation, emphasis, emotion — that a text bottleneck destroys entirely. A cascaded pipeline literally cannot know that the user sounded upset, because "I'm fine" is what reached the model.

The costs are equally real. Observability is much harder when the intermediate representation is not text. Constraining behavior is harder, because your guardrails were written for text. Tool calling is generally less mature. Provider lock-in is stronger, because the component boundaries you would swap at do not exist. And self-hosting options, while improving, are narrower.

How the decision usually resolves

A common production compromise is a cascaded pipeline with an emotion or acoustic-signal classifier running in parallel with recognition, so that some paralinguistic information reaches the model as structured metadata even though the main path is text. It is less elegant than a native model and considerably easier to operate.

This is where most voice agents are judged, and where most of them fail. Users rarely complain that the recognition was wrong. They complain that it talked over them, or that it cut them off, or that it just sat there.

Endpointing is a judgment call, not a threshold

The naive implementation waits for silence. Voice activity detection — energy-based like the WebRTC detector, or neural like Silero — tells you when speech is present, and a timer decides when enough silence has elapsed to call the turn over. Then you set the threshold and discover it cannot be right.

Short thresholds cut people off mid-thought, because humans pause inside sentences constantly: while thinking, before a name, after "um", and in the middle of reading a number off a card. Long thresholds make the agent feel slow on every short exchange, because the same delay applies to "yes" as to a rambling explanation.

The fixes are contextual rather than numerical. Adapt the threshold to what you asked: after a yes-or-no question, be aggressive; after "read me the reference number", be patient. Use semantic endpointing, where a small fast model judges whether the partial transcript is a complete thought, so that a trailing "and I also wanted to ask about" holds the turn regardless of silence. Use prosodic cues where available, since falling pitch is a strong turn-yielding signal in most languages. And treat the false-cut-off rate as a first-class metric with its own dashboard, because it is invisible in aggregate latency numbers and enormously visible to users.

Barge-in is harder than it sounds, in a specific way

Barge-in means the user starts talking while the agent is speaking, and the agent stops. Three separate things have to work.

The audio path must be genuinely full duplex, capturing while playing. On a browser or phone that means the platform's echo cancellation has to be doing its job, otherwise the agent hears its own output through the speaker, transcribes it, and interrupts itself. Debugging an agent that appears to be arguing with itself is a rite of passage, and the cause is almost always echo cancellation being disabled or defeated by the audio routing.

The pipeline must support cancellation. When interruption is detected, in-flight synthesis has to stop, queued audio has to be flushed, and the language model request has to be aborted. A pipeline built on non-cancelable calls will keep generating and keep speaking for seconds after the user began talking.

And — the part that most implementations get wrong — the conversation history has to be truncated to what the user actually heard. If the agent generated three sentences, played one and a half, and was then interrupted, the model's context must record one and a half sentences. Otherwise the agent believes it said things the user never heard, and every subsequent turn is built on a false premise. Users experience this as the agent referring to information it never gave them, which is far more damaging to trust than a slow response.

False triggers

Backchannel noises — "mm-hm", "right", "yeah" — are not interruptions. A system that stops speaking every time the user acknowledges it is exhausting to talk to. The usual approach is a short grace window plus a minimum energy and duration threshold before an interruption counts, tuned differently for noisy environments. Background speech from a television or an open-plan office is the same problem in a harder form, and is one of the strongest arguments for keeping a wake-condition or push-to-talk mode available.

An agent that only talks is a demo. An agent that looks up an order, books an appointment or files a ticket is a product, and adding actions to a conversation introduces problems that text agents never face.

Latency becomes conversational

In a chat interface a three-second database query is a spinner. In a conversation it is three seconds of silence, which a human interprets as the line dropping. The pattern that works is to acknowledge before acting: emit a short spoken filler at the moment the tool call starts, then stream the result when it arrives. This has to be genuine rather than decorative — a filler that plays before every response, including instant ones, reads as a verbal tic within three turns.

Set hard timeouts on every tool and design the spoken fallback for each. "I am not able to reach the booking system right now, would you like me to take a message" is a better outcome than a thirty-second silence followed by a correct answer.

Confirmation is expensive in a low-bandwidth channel

Voice carries far less information per second than a screen, and it is linear: the user cannot skim back. That changes what you can confirm. Reading back a full address, a date, a reference number and a price before every action is technically thorough and unbearable to sit through. The workable compromise is confirming only what is both consequential and error-prone, using the recognition confidence and the semantic risk of the field together, and letting the rest ride on the assumption that the user will correct you.

Numbers, names and alphanumeric identifiers deserve particular attention, because they are the highest-error and highest-consequence category simultaneously. Constrained recognition, phonetic alphabets and structured re-prompting all help more than a better general-purpose model does.

State and memory

Within a call, state is straightforward and mostly a matter of not losing it when a component restarts. Across calls, personalization runs into identity: on the phone you may have a number, on the web a session, and neither reliably identifies a person. Building durable preferences on a weak identifier is how one caller ends up with another caller's history.

The other memory question is what to carry forward. Voice conversations produce long transcripts full of repair, backtracking and filler, and stuffing them into the next call's prompt degrades both latency and quality. Summarize at the end of a call into a small structured record, and keep the raw transcript for audit rather than for context.

Voice-specific security

Prompt injection arrives through the microphone. Anything the caller says reaches the model, and so does anything else audible: hold music with lyrics, a television, another automated system on a transferred call, a person coaching the caller in the background. Instructions delivered in speech are indistinguishable from user intent by the time they are text.

Treat everything from the audio path as untrusted input, keep tool permissions scoped to what the caller has been authenticated to do, and separate authentication from conversation rather than letting the model decide who somebody is. Voice makes impersonation cheap in both directions, so any high-value action should be gated on something other than the sound of a voice.

Most commercially valuable voice agents end up on a phone line, and the phone network imposes constraints that no amount of model quality overcomes.

The audio is narrowband. Traditional telephony carries speech through G.711 at an 8 kHz sample rate, roughly half the bandwidth that speech models are typically trained on and a fraction of what a web microphone delivers. Information above the cutoff is simply gone, and the consonant distinctions that live up there are exactly the ones recognition depends on. Expect materially worse accuracy on a call than in a browser tab with the same model, plan for it in your prompts and confirmation strategy, and evaluate on telephony-band audio rather than clean recordings. Wideband codecs exist on some paths and cannot be assumed.

The plumbing is unfamiliar. SIP trunking, RTP streams, jitter buffers, DTMF tones carried out of band, call transfer semantics, and answering machine detection for outbound work. None of it is conceptually difficult and all of it is unlike HTTP. A platform provider hides most of it; the parts that leak are the ones that break, particularly transfer, hold and conference behavior.

The regulatory surface is real. Call recording consent rules differ by jurisdiction and several US states require all parties to consent. In the United States the FCC ruled in 2024 that calls using AI-generated voices fall within the Telephone Consumer Protection Act's restrictions on artificial and prerecorded voice, which has direct consequences for outbound campaigns. Disclosure requirements for automated callers exist in several jurisdictions, and the EU AI Act includes transparency obligations for systems that interact with people. If you handle card payments, PCI requirements mean pausing recording and masking DTMF during capture rather than recording everything and redacting later.

Contact center integration is where projects actually stall. The agent is rarely the hard part. Routing rules, queue behavior, warm transfer with context handed to the human, wrap-up codes, CRM writeback and reporting that reconciles with what the existing platform reports are all work, and all of it involves systems that predate the project by a decade. Budget accordingly, and design the escalation path to a human first rather than last, because it is the safety net for every failure mode in this document.

A voice agent that works in a demo and fails in production usually fails for reasons nobody measured. Voice needs its own evaluation vocabulary and its own operational model.

What to measure

Word error rate is necessary and badly insufficient. It weights a missed article the same as a missed account number, and it says nothing about whether the conversation succeeded. Build the metric set in layers.

At the task layer: completion rate, containment rate — the proportion of calls resolved without escalation — and, most useful of all, a sampled human rating of transcripts against the outcome the caller actually wanted.

At the conversation layer: false endpoint rate, interruption handling success, average turns to completion, and the frequency of repair sequences where the user says a variant of "no, I said". Repair frequency is the single most sensitive proxy for whether the agent is pleasant to use.

At the stage layer: per-component latency percentiles, tool call success and timeout rates, and recognition confidence distributions segmented by audio path, since telephony and browser calls should be tracked separately or the telephony problems disappear into an average.

Testing without a room full of people

Synthetic callers are the practical answer. Drive the agent with a scripted or model-driven caller over the same audio path real users take, with speech synthesis producing the input, and you get a regression suite that can run on every deployment. Build the scenario set from real failures: the caller who interrupts constantly, the one with heavy background noise, the one who supplies information in the wrong order, the one who changes their mind, the one reading a long alphanumeric string, the one speaking a second language. Adversarial scenarios belong here too, including a caller who tries to instruct the agent directly.

Synthetic callers do not sound like real ones and will not surface every problem, so the suite is a floor and not a substitute for listening to real calls.

Observability that is specific to audio

Store, for every session, the audio, the transcript with timestamps, the model messages, the tool calls and the per-stage timings, all joined by one identifier. Without the audio you cannot tell a recognition error from a user who genuinely said something odd. With it, most investigations take minutes. This creates a data protection obligation immediately: recordings contain biometric and personal data, retention periods need to be short and enforced, redaction needs to happen before storage rather than at query time, and access needs to be logged.

Scaling is not like scaling a web service

Sessions are stateful, long-lived and cannot be load balanced mid-call, so sticky routing is mandatory. Capacity is bounded by concurrent calls rather than requests per second, and the binding resource is usually speech processing rather than language model tokens. Cold starts are visible to users, so warm pools matter more than autoscaling responsiveness. And deployment needs draining rather than rolling restarts, because terminating an instance drops live conversations — a class of outage that is entirely invisible in ordinary web metrics and immediately obvious to the person who was talking.