What Is an AI Voice Agent?

A demo that sounds human is not a voice agent.

That product is a realtime control loop. It listens, decides, calls tools, speaks, and keeps state when the caller interrupts. One speech model does none of that alone.

We evaluate the assembled loop, not the voice. Identity, permissions, interruptions, and a human exit are part of the system.

The voice is the interface. The agent is everything behind it.

The loop has five jobs, then it has to remember

Every voice turn requires the system to:

1. detect that the user is speaking and when the turn ends,
2. convert speech into model input, either as audio or text,
3. reason over the request, conversation, policy, and retrieved context,
4. call tools or form an answer, and
5. generate audio that can be interrupted safely.

A sixth job spans every turn: preserve state without repeating an action. If a caller interrupts while an address change is running, the system must know whether the tool completed before trying again.

That is why a voice agent sits closer to a distributed application than to a talking chatbot. Five of those jobs can each be excellent while the assembled product is unusable, because the failures compound across boundaries rather than staying inside one component.

Turn detection is the job that looks easiest and behaves worst. Detectors that wait for certainty make an agent feel slow and stilted. Ones that fire early clip the last digits of an account number, and the caller starts over. Neither failure shows up in a model evaluation, because neither is the model's doing. Teams that tune only the model and never the detector usually plateau at an agent that is fast, fluent, and mildly infuriating.

Cascaded stacks keep components replaceable

Cascaded architectures connect speech-to-text, a text language model, and text-to-speech. A telephony or realtime transport layer carries audio, while an orchestrator handles turn state and tools.

Table 1 Download CSV

Layer	Job	Failure to test
Turn detector	Decide when speech starts and ends	Cuts the caller off or waits through silence
Speech recognition	Produce the transcript	Names, numbers, accents, noise, and code-switching
Language model	Decide and call tools	Wrong action, long answers, policy failure
Tool layer	Read or change external systems	Duplicate actions, stale state, permission errors
Speech generation	Speak the response	Pronunciation, delay, tone, cancellation
Transport	Carry live audio	Jitter, reconnects, regional delay

Replaceability is the benefit. Teams can choose a specialist transcriber, a fast text model, a particular voice, and a telephony vendor, route languages differently, and swap out one weak component without rebuilding the product.

Integration is the cost. Every boundary adds buffering, tracing, error handling, and one more place where a cancellation signal can arrive too late to matter.

Native audio makes the chain shorter

Native audio models accept speech and generate speech directly. GPT-Realtime 2.1 and Gemini 3.1 Flash Live Preview are current examples. Both support function calling inside a realtime session.

Native models retain acoustic information a transcript discards: hesitation, cadence, pronunciation, overlapping speech, and emotion-like cues. They also cut the number of separate model calls in the path.

Coupling is the trade. One provider now controls the model, the voices, session behavior, billing, rate limits, and your migration path. You may still need a transcript for audit or downstream systems even when it is no longer the reasoning input, which quietly returns part of the cascaded complexity you were trying to remove.

Choose between them by asking which risk you would rather carry. Cascaded stacks fail in ways you can isolate and replace, at the price of owning more seams. Native models fail as one opaque unit, and when a provider changes session behavior your options are to adapt or leave. Neither is safer in general. Cascaded rewards teams with operational capacity. Native rewards teams shipping quickly against a narrow workflow.

Tools are what make a voice interface an agent

Without tools, the system answers questions but cannot check an order, book an appointment, update an address, or transfer a call with context. Tools should be narrow, typed, permissioned, and observable.

A safe address-change flow might:

1. authenticate the caller,
2. read the current address,
3. capture and normalize the proposed address,
4. repeat the change in plain language,
5. require explicit confirmation,
6. call one idempotent update tool, and
7. read back the confirmed result.

Language models propose the tool call. Applications enforce identity, authorization, schema validation, confirmation, rate limits, and idempotency.

Natural language is not a permission system.

Voice raises the stakes on every tool error. Chat users see a wrong result and correct it in the next message. Callers hear a confident sentence, believe it, and hang up. They have no visible form to re-read, no undo button in earshot, and no transcript in front of them. That asymmetry is why confirmation and idempotency belong in the application rather than the prompt, and why read-only tools should ship well before writes do.

The turn budget is the whole loop

Callers do not experience model latency. They experience the gap between the moment they stop talking and the moment they hear a voice, and every stage in the loop spends part of it.

Table 2 Download CSV

Stage	What it adds to the gap	Who controls it
End-of-turn detection	Wait before the system believes the caller finished	You, via tuning
Speech recognition	Final transcript after last audio	Vendor and audio quality
Model first answer	Prompt processing plus first generated chunk	Model and prompt size
Speech synthesis	First audio chunk after first text	Vendor and voice settings
Transport and playback	Network, jitter buffer, device	Region and connection

Only the middle row is the model, and it is the row most teams try to fix first. Current first-answer measurements across scored models look like this:

Table 3 Download CSV

Model	Latency (first answer)	Output speed	Type	Overall score
Gemini 3 Flash	0.75s	199 t/s	Non-Reasoning	60
GPT-5 (high)	1.33s	66 t/s	Reasoning	59
Claude Opus 4.5	1.42s	52 t/s	Non-Reasoning	63
GLM-4.5	1.45s	51 t/s	Non-Reasoning	58
Claude Sonnet 4.6	1.60s	47 t/s	Non-Reasoning	64
Claude Opus 4.6	2.37s	44 t/s	Non-Reasoning	68
Kimi K2.6	2.88s	40 t/s	Reasoning	60
GPT-5.4 nano	3.64s	191 t/s	Reasoning	66

Two things in that table matter for voice. The spread between the quickest and slowest rows is large enough to consume an entire turn budget on its own. And first-answer order does not track the quality column, so the model that would win a latency contest is frequently not the model you want deciding whether to cancel someone's order.

We time text completions, not calls. Those rows screen candidates and cannot rank them for voice, because nothing in them includes turn detection, transcription, synthesis, or transport. Our TTFT explainer sets out why the two clocks differ, and Best LLMs for Voice AI Agents turns the screen into a shortlist that a call replay then decides.

Speed also has a false positive. Systems look fast when they guess early that the caller finished, and that guess truncates account numbers.

A platform is the loop you would otherwise build

Most teams asking what a voice agent is are really asking whether to buy one. Vendors such as Vapi, Retell, ElevenLabs, Twilio, and Aircall package telephony, turn detection, barge-in handling, session state, recording, and dashboards behind one API.

That bundle is most of the engineering and rarely the differentiator. Turn detection in particular is a long tail of tuning that looks trivial in a demo and consumes weeks in production.

Build the loop yourself when routing is unusual, when a compliance boundary requires specific components in specific regions, or when swapping one model or voice independently is the point. Buy it when the workflow, tools, and escalation path are where your product actually lives.

The division survives either choice. No platform owns your tools, your permission checks, your escalation design, or your replay set, and those four are what separate an agent that resolves calls from one that merely answers them.

One warning on evaluating platforms: demos get recorded in quiet rooms by people who know exactly what to say. Ask for a trial on your own numbers, in your own region, with a caller who mumbles a real order reference. Most platform disappointment lives in the gap between a demo and a Tuesday afternoon.

Escalation is a product path

Agents need a clear boundary for uncertainty, policy exceptions, distress, accessibility needs, identity failures, and requests outside their tools. "I can't help" is not a finished escalation design.

Transfer the conversation state, verified identity, requested action, tool results, and a concise summary to the human queue. Tell the caller what is happening and whether they need to repeat information. If no human is available, offer a bounded next step with a reference number.

Escalation rate is not automatically a failure. For a high-risk workflow, transferring early can be the correct behavior, and a pilot that drove escalations to zero has usually widened its permissions rather than improved its judgment.

Track what the human had to redo after each transfer. That number tells you whether the handoff carried real context or merely moved the call.

Conversations are the unit, not isolated prompts

Build a replay set from consented and redacted calls, or from synthetic scripts that reproduce the same turn shapes. Include interruptions, silence, background noise, spelling, numbers, corrections, tool failures, unsupported requests, and malicious instructions embedded in retrieved content.

Score the whole conversation:

• task completion and correct tool arguments,
• unauthorized or duplicate actions,
• end-to-end latency distribution,
• transcript and pronunciation errors that change meaning,
• recovery after interruptions and tool failures,
• escalation quality, and
• model, speech, telephony, and tool cost per completed task.

Prompt-level testing misses all of the interesting failures, because the interesting failures live in the seams: a cancellation that arrived after synthesis started, a retry that fired a second write, a transcript that turned a postcode into a phone number.

Size the replay set by failure variety rather than call count. One hundred calls covering twenty distinct break conditions catch more regressions than a thousand runs of the happy path with different names. Listen to a sample of passes as well as failures, because a scored pass can still contain an agent that guessed, sounded certain, and happened to be right.

The agent is ready when the complete loop passes, not when one demo sounds human.