Shipping a Jarvis-like voice interface without the Jarvis-sized bill: lessons learned

Building a voice agent is a solved problem if all you want is speech in, speech out, and three function calls in between. Microphone, transcription, a small prompt, text-to-speech, voice modulation, capability to handle barge in. That part is commodity now, so we won't dwell on it.

The hard part is voicing an agent that thinks. Ours does. The Transilience security agent runs on Claude Sonnet, on top of the Vercel AI SDK, and before it answers you it reasons for several seconds and fans out across 60+ tools. REST calls to run scans and pull findings. Document skills that produce PDFs and slide decks. Dashboard and scheduling actions. File reads. One spoken question can set off a chain of tool calls and thousands of tokens of hidden reasoning.

So the question was never how to make it talk. It was how to voice something that thinks this hard and does this much, without ten seconds of silence, and without sounding like a machine reading a log.

Here's what we learned building it.

We didn't change the agent. We listened to it.

The agent stays exactly as it is in text. Same model, same prompt, same personality, same 60+ tools, same deep reasoning. We didn't fork it into a chatty voice twin. Voice is a layer that listens in on the agent the AI SDK is already running and decides what to say out loud.

There's a lot to listen to. As Sonnet works, the SDK streams its live reasoning, every tool call it fires, every result that comes back, and finally the answer. Three kinds of moments, all happening at once:

• the thinking: "I should pull their IAM findings, then cross-reference the last scan"
• the tool calls: runApp, readFileContent, getSessionDetail, a document skill
• the answer itself

Voicing the agent means connecting those dots in real time. You take a stream of internal machinery and turn it into calm narration that sounds like someone working on your problem.

A second model, running in parallel, just to sound human

The main agent is busy thinking and calling tools. You can't ask it to narrate itself too, because that slows it down and pollutes its reasoning. So we run a smaller, faster model alongside it (Claude Haiku) with one job: watch what the big agent is doing and say one short, human line about it.

We call it the Voice Director. It reads the agent's live reasoning and the tool it just fired, then says something grounded:

"Pulling your IAM findings now." "Cross-referencing that against the last scan."

Not a canned "please wait." We shipped canned filler first. It sounded like hold music, so we deleted it the same week. Each line is one breath, capped to about sixteen words, and there's a hard limit on how many it says per turn so it narrates instead of rambling. If it can't say something honest in time, it stays quiet. Silence beats a robot.

It runs fire-and-forget, in parallel. The Director never sits in the agent's path, never blocks a tool call, never touches the answer. The agent doesn't even know it's there. It's a commentator watching the game, not a player.

Saying the answer like a person, not reading it like a file

When the answer comes, the obvious move is to read it aloud. We tried. It was awful. The agent's real answers are rich markdown: bullet lists, links, file names, UUIDs, tables. Read out loud, that becomes a breathless run-on that recites identifiers like a license plate.

So what you hear is never the raw answer. The screen still shows the full, linked, formatted version. The voice gets a distillation, split in two so you're not waiting on the whole thing.

The moment the first couple of sentences exist, a fast Haiku pass distills the opening into one calm line. The agent is prompted to lead with the bottom line, so that opening line is the answer. You hear the verdict almost immediately, while the rest is still being written. Then it distills the remainder, told what it already said so it doesn't repeat itself, and we kick that off early, mid- stream, so it's ready to play the instant the opening finishes. No gap.

Same human voice for the acks, the narration, and the answer. From the outside it's one person talking you through their work. Underneath, it's the big agent thinking, a small model narrating, and a distiller turning conclusions into speech, all overlapping.

you ask · speech to text, Deepgram Flux

on screen · claude sonnet, unchanged

out loud · claude haiku, in parallel

1. thinking

   “I should pull their IAM findings, then cross-reference the last scan”

   voice director

   “Pulling your IAM findings now.”

2. tool calls

   runApp · getSessionDetail

   voice director

   “Cross-referencing that against the last scan.”

3. answer, opening

   The bottom line, written first

   distiller

   One calm line. You hear the verdict almost immediately.

4. answer, the rest

   Lists, links, file names, UUIDs, tables

   distiller

   The remainder, told what was already said

you hear one voice · text to speech, ElevenLabs Flash v2.5

The AI SDK plumbing that made it real-time

One unglamorous fight earned the whole thing. The natural way to feed answer text into speech on the Vercel AI SDK is to consume result.textStream. We did, and our low-latency pipeline stalled until the model had finished the entire answer. The SDK buffers that view when you're also consuming the UI stream.

The fix was to stop using it. We tap the raw onChunk text deltas, push them into our own queue, and feed that to speech, so audio starts at the first token instead of the last. Small change, hard to find, and it's the difference between an agent that feels alive and one that feels like it's buffering. That's the layer the demo never shows.

The bill: we only pay for audio at the edges

This is cheaper than the obvious approach by an order of magnitude, and that falls straight out of the architecture.

A realtime voice API like OpenAI's gpt-realtime bills you in audio tokens, and it makes one model do everything inside that audio session: listen, reason, call your tools, and talk. Audio output runs about $64 per million tokens, and the assistant's speech is metered at one token per 50 ms. You pay premium rates for every second it talks, plus every reasoning token and tool result that rides along in the same session. For a light chatbot that's roughly $0.18 to $0.46 a minute. But our agent thinks for seconds and round-trips through 60+ tools with big JSON results. For us the orchestration is the expensive part, and a realtime API would run all of it at audio-session prices.

So we unbundled it. The expensive work, the deep reasoning and the tool, REST, and document-skill orchestration, runs on a plain text model (Claude Sonnet, $3 and $15 per million text tokens), exactly as it already does for the text product, at no premium. Speech only ever touches the edges:

And because the voice only ever speaks the distilled answer, not the full one, even the text-to-speech bill is a fraction of reading the whole thing aloud. The reasoning we already pay for in text stays at text prices. Voice adds a few cents at the rim instead of a premium on the whole turn. Route this agent through a realtime voice API and you'd pay far more, for weaker tool-calling and skills you'd have to rebuild anyway.

One agent, thinking as hard as ever, now out loud

What we're proud of isn't speech recognition or a nice voice. Anyone can buy those. It's that an agent doing real work, reasoning deeply, orchestrating dozens of REST tools, generating documents, running skills, now talks you through that work the way a sharp colleague would. It tells you what it's checking, what it found, and what to do about it, in real time, with no dead-air pause and without sounding like a machine.

We didn't wrap a voice API. We taught a thinking agent to think out loud.