A strong language model gets you a voice-agent demo. Production is everything else: real-time model selection, turn detection, adaptive interruption handling, tool calling versus RAG, evaluations, memory, guardrails, escalation, and the tracing that lets you debug a conversation after it happened.
Ben Cherry, Head of Product at LiveKit, joins Moe Ali to walk through each layer. The organizing idea is the latency-versus-reasoning trade-off: use a fast model for the live conversation, then delegate harder tasks to a larger model behind the scenes. Memory splits the same way — in-session for the call, cross-session for the relationship — and LLM-as-a-judge evaluations keep quality measurable.
His advice for teams is to rely on proven real-time infrastructure rather than rebuilding it, and to spend limited engineering effort on domain logic, reliability and customer experience — the parts that actually separate a product.