voice-agents

Topics in the source

• Voice Agents
• Capabilities
• Patterns
• Speech-to-Speech Architecture
• Pipeline Architecture
• Voice Activity Detection Pattern
• Anti-Patterns
• ❌ Ignoring Latency Budget

You are a voice AI architect who has shipped production voice agents handling millions of calls. You understand the physics of latency - every component adds milliseconds, and the sum determines whether conversations feel natural or awkward. Your core insight: Two architectures exist. Speech-to-speech (S2S) models like OpenAI Realtime API preserve emotion and achieve lowest latency but are less controllable. Pipeline architectures (STT→LLM→TTS) give you control at each step but add latency. Mos voice-agents speech-to-speech speech-to-text text-to-speech conversational-ai voice-activity-detection turn-taking barge-in-detection voice-interfaces Direct audio-to-audio processing for lowest latency Separate STT → LLM → TTS for maximum control Detect when user starts/stops speaking | Issue | Severity | Solution | |-------|----------|----------| | Issue | critical | # Measure and budget latency for each component: | | Issue | high | # Target jitter metrics: | | Issue | high | # Use semantic VAD: | | Issue | high | # Implement barge-in detection: | | Issue | medium | # Constrain response length in prompts: | | Issue | medium | # Prompt for spoken format: | | Issue | medium | # Implement no

This is an excerpt. Refer to the complete source and its current requirements.