Notes
🔍

Voice Agent

Calling a company and being told to press 1 for billing is the old way of letting software answer the phone. A voice agent replaces that menu: you say what you want in your own words, and it works out what to do, does it, and tells you.

Speech at both ends of the loop
A voice agent is an agent whose input and output are spoken. Audio arrives, speech recognition converts it to text, the model plans and calls tools, and speech synthesis speaks the reply. Newer systems drop the text round-trip and use a single speech-to-speech model to cut the delay. The agent layer underneath is familiar — plan, call a booking API, read the result, decide the next step — but every step now runs against a clock, because a caller hears three seconds of silence as a dropped call rather than as thinking.

What the voice channel adds
Design pressure shifts from accuracy alone to timing:

  • Turn detection — judging when the caller has actually finished, instead of cutting them off mid-sentence.
  • Barge-in — stopping mid-word when the caller talks over the agent.
  • Streaming, so the agent can say it is checking while the lookup is still running.
  • Spoken confirmation before anything irreversible, since there is no screen showing what is about to be charged.

Where it breaks
A misheard account number propagates silently into a tool call that succeeds on the wrong record. Frameworks such as the OpenAI Agents SDK and Pipecat handle the audio pipeline, but the hard part stays agentic: a caller who changes their mind halfway through leaves the agent holding a half-finished plan it has to abandon rather than complete.

A voice agent is an agent that takes spoken input and replies in speech: audio is transcribed (or fed to a speech-to-speech model), the agent plans and calls tools, and the response is spoken back. It matters because the voice channel puts the plan–act–observe loop on a latency budget — turn-taking, interruption and spoken confirmation become part of the control flow, and silence while a tool runs reads as failure.

Think of phoning a helpful shop assistant instead of poking at a touch-tone menu. You explain what you want in ordinary words, they ask one or two questions, go and sort it out, and tell you what they did. A voice agent is software that handles a call like that.

It listens, works out what you are asking for, takes the steps needed — looking up your order, rebooking a flight — and answers out loud. The tricky part is timing. People expect a reply straight away, and they interrupt. So a voice agent has to know when you have finished speaking, and when to stop talking because you have started.