Create a fast phone agent with live voice streaming
They assembled a call-answering agent with live transcription, generation, and speech output, then moved it closer to the services to get about 400ms latency.
- What they did
- They put together a FastAPI app that accepts Twilio call audio through a WebSocket and passes it to Deepgram Flux for transcription and turn detection. When a speaker stops, the transcript is sent to an LLM, its tokens stream into ElevenLabs TTS, and the resulting audio is returned to Twilio right away. They also added interruption handling so a fresh utterance can stop both the model and the audio stream, and they kept TTS connections open so they were already warmed up.
- How it went
- On a local machine, the full loop averaged about 1.7s. Hosting it in the EU brought that down to around 790ms, a bit faster than Vapi’s estimate of roughly 840ms. Replacing gpt-4o-mini with Groq’s llama-3.3-70b got the average to about 400ms, although the first exchange still lagged. The project took about a day and cost around $100 in API credits.
- Tips
- If you can, co-locate the server with Twilio, Deepgram, and ElevenLabs to avoid extra delay. Measure the first response separately, since it may behave differently from later turns.
Try it yourself with Claude Code
Set up a [language] phone agent for [use case] that listens to Twilio call audio, turns speech into text with [transcription provider], sends each completed user turn to [LLM provider], and streams the answer through [TTS provider] back to the caller. Optimize for a response time under 500ms, add logs for each stage, and support barge-in so a new user utterance can stop the current reply. Before any irreversible step such as placing live calls, purchasing numbers, or submitting data externally, stop and ask me to confirm.
Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.
Get 3 agent uses like this every morning
With the day's top AI news. Free, and one click to unsubscribe.
Related agent uses
Use agents to benchmark and tune Rust performance
Have an agent measure Rust code, make one optimization at a time, and keep only changes that improve runtime.
Connect a coding agent to an Excalidraw board
They hooked an installed coding agent to a live Excalidraw canvas so it could read the scene and edit the drawing directly.
Review a feature and pass the findings to an engineer
One developer had an agent inspect a feature for problems, then handed the issue list to an engineer whose agent fixed most of them.
Build a terminal calling tool with human backup
They wired a CLI agent to place calls and loop in a person only when it needed an answer.
Revive old apps and sketch new ones with a coding agent
Terence Tao used a coding agent to port old Java applets to JavaScript and to build two new math visualizations in conversation.
Run bug fixes through a staged agent team
Use separate agent roles to reproduce, diagnose, review, and audit bugs before any irreversible step.