Build a phone voice agent that answers in under half a second
This post made it to the Hacker News front page! Task: The author built a custom voice agent orchestration layer to replace off-the-shelf platforms like Vapi. The workflow involved connecting Twilio for telephony, Deepgram Flux for streaming transcription and turn detection, Groq for low-latency LLM inference, and ElevenLabs for text-to-speech. The system uses a streaming pipeline where LLM tokens are immediately converted to audio, with logic to cancel generation and flush buffers when the user interrupts. The author iterated from a basic VAD prototype to a full production-ready stack, optimizing for geographic proximity and model selection.; tools: custom-built or unnamed agent. Human role: The original post does not say how a person stayed involved.; limitations: Building a custom orchestration layer requires significant engineering effort to handle edge cases like barge-ins, network latency, and state management. Off-the-shelf platforms offer observability, reliability, and configuration options that are difficult to reproduce. The performance gains depend heavily on geographic co-location of services and specific model choices (e.g., Groq for low TTFT). This approach is suitable for teams with specific latency requirements or those needing deep control over the voice pipeline..
Tools
custom-built or unnamed agent
Published on Agent Work Atlas:
Overview
This post made it to the Hacker News front page!
Workflow
The author built a custom voice agent orchestration layer to replace off-the-shelf platforms like Vapi. The workflow involved connecting Twilio for telephony, Deepgram Flux for streaming transcription and turn detection, Groq for low-latency LLM inference, and ElevenLabs for text-to-speech. The system uses a streaming pipeline where LLM tokens are immediately converted to audio, with logic to cancel generation and flush buffers when the user interrupts. The author iterated from a basic VAD prototype to a full production-ready stack, optimizing for geographic proximity and model selection.
Reported result
The custom implementation achieved approximately 400ms end-to-end latency, which was twice as fast as the equivalent Vapi configuration (~840ms). The author reported that the conversation felt more natural and responsive, with clean interruption handling. The build took about one day and cost roughly $100 in API credits. The source code was published on GitHub.
Human role
The original post does not say how a person stayed involved.
Limitations
Building a custom orchestration layer requires significant engineering effort to handle edge cases like barge-ins, network latency, and state management. Off-the-shelf platforms offer observability, reliability, and configuration options that are difficult to reproduce. The performance gains depend heavily on geographic co-location of services and specific model choices (e.g., Groq for low TTFT). This approach is suitable for teams with specific latency requirements or those needing deep control over the voice pipeline.
Try it
I want to build a custom voice agent prototype to test latency improvements over existing platforms. **Inputs:** - Telephony provider: Twilio (use [replaceable Twilio Account SID and Auth Token]) - STT/Turn Detection: Deepgram Flux (use [replaceable Deepgram API Key]) - LLM: Groq llama-3.3-70b (use [replaceable Groq API Key]) - TTS: ElevenLabs (use [replaceable ElevenLabs API Key]) - Target: Achieve sub-500ms end-to-end latency. **Steps:** 1. Set up a FastAPI server with a WebSocket endpoint for Twilio. 2. Implement a streaming pipeline: Twilio audio -> Deepgram Flux -> Groq LLM -> ElevenLabs TTS -> Twilio audio. 3. Add logic to cancel LLM generation and TTS synthesis immediately when Deepgram detects 'start of turn' (user interruption). 4. Deploy the server in the same region as the API providers (e.g., EU or US East) to minimize network hops. 5. Run a dry test with a single sample conversation to measure latency and verify interruption handling. **Deliverables:** - A working prototype codebase. - A latency report comparing the custom build against a baseline Vapi setup. **Safeguards:** - Do not deploy to production without human review of the interruption logic. - Use a test phone number for initial trials. - Monitor API costs closely during the first 100 interactions.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.