The Rise of Low-Latency Speech-to-Speech Models

Executive Summary

Voice is the most natural form of human communication. However, for years, voice-based human-computer interfaces (such as Siri, Alexa, or legacy phone menu IVR systems) felt frustratingly robotic, slow, and rigid. They were unable to grasp sarcasm, detect emotional inflections, or handle interruptions.

The introduction of Large Voice Models (LVMs) and end-to-end speech-to-speech models (like GPT-4o voice and Gemini Live) represents a paradigm shift. By processing audio signals directly as neural tokens rather than cascading separate transcription and text-to-speech programs, voice AI agents can now converse with human-like latency (sub-200 milliseconds) and detect vocal expressions. This article details the structural evolution, technical mechanics, WebRTC network requirements, and enterprise applications of modern voice AI agents.

1. The Evolution of Voice AI Architectures

To understand why modern voice agents represent a massive leap, we must compare the two primary engineering approaches to voice processing:

A. Cascaded Pipeline (Legacy):[ Audio Input ] ──► [ Speech-to-Text (ASR) ] ──► [ Text LLM ] ──► [ Text-to-Speech (TTS) ] ──► [ Audio Output ]   (Whisper)            (Transcribed text)       (Generates reply)      (ElevenLabs)   * Latency: 1.5 - 3.5 seconds (Rigid, breaks conversation flow)B. End-to-End Speech-to-Speech (Modern):[ Audio Input ] ───────────────────────────► [ Audio LLM ] ───────────────────────────► [ Audio Output ]   (Raw waveform)                             (Processes tokens)                       (Direct audio stream)   * Latency: 150 - 250 milliseconds (Human-like, supports interruptions)

The Cascaded Pipeline (ASR + LLM + TTS)

Historically, voice apps ran three sequential systems.

An Automatic Speech Recognition (ASR) model (e.g., Whisper) converted user audio to text.
The text was sent to a standard LLM to generate a text response.
A Text-to-Speech (TTS) engine (e.g., ElevenLabs) synthesized the text into output audio.

Limitations: Every step added latency. Pitch, sarcasm, and emotional inflections were completely lost during the transcription stage. Users could not interrupt the agent while it was speaking because the output audio was pre-rendered.

The End-to-End Speech-to-Speech Model

Modern voice models process audio natively. The neural network takes raw audio tokens directly, processes them in a single forward pass, and immediately outputs raw audio tokens. This preserves emotional nuance (whispering, laughter, stress) and cuts latency down to human-like conversation speeds.

2. Technical Mechanics: Audio Tokenization

To process sound inside a transformer network, continuous sound waves must be converted into discrete tokens that a model can read. This is achieved using neural audio codecs (such as Meta’s EnCodec or Google’s SoundStream):

Quantization: The codec compresses the continuous analog sound wave (spectrogram) into discrete, digital audio tokens.
Transformer Processing: The LLM receives these audio tokens alongside text tokens. It outputs next-token predictions that represent both the textual content and the acoustic properties (voice model, pitch, speed, and volume).
De-quantization (Decoding): The output tokens are converted back into continuous audio waveforms, streaming the sound instantly to the user.

3. The Latency Challenge: WebRTC & Streaming

For a voice agent to feel natural, it must match the human response latency of 200 milliseconds. Delivering this speed across the internet requires deep network optimization:

WebRTC Protocol: Traditional HTTP/REST requests are too slow for real-time audio. Voice agents utilize WebRTC (Web Real-Time Communication), which uses UDP (User Datagram Protocol) to stream audio packets back and forth with minimal latency, bypassing heavy TCP handshake overheads.
Speculative Decoding and Chunking: The server streams audio in tiny chunks (e.g., 20ms of audio). The model speculatively predicts the next phoneme before finishing the prior sentence, ensuring continuous sound delivery.
Acoustic Echo Cancellation (AEC) & VAD: Voice agents run highly sensitive Voice Activity Detection (VAD) models locally on the user’s device. The millisecond the user starts speaking, the VAD sends a interrupt signal to the server, instantly halting the agent’s current audio stream so the human can speak.

4. Acoustic Features and Tone Modeling

Modern end-to-end models represent audio on a two-dimensional token stream:

Semantic Tokens: Capture the textual meaning (words spoken).
Acoustic Tokens: Capture non-verbal voice cues (pitch, rhythm, tone, background noise, and pacing).

Because the model natively maps acoustic parameters, the voice agent can adjust its speech patterns dynamically. If a caller is speaking quickly and sounding stressed, the agent can adjust its own tone to be calm, soft, and spoken at a measured pace. If the caller whispers, the agent can reply in a matching whisper, creating an empathetic customer experience.

5. Conclusion

Voice AI Agents are transitioning digital communication from mechanical interfaces to natural human dialogue. By bypassing slow, multi-stage pipelines and utilizing end-to-end audio models streaming over WebRTC, voice AI achieves response times that feel human. While managing network latency, background noise, and turn-taking logic presents major technical challenges, the convenience of voice-controlled operations makes voice agents a vital element in customer support, field services, and next-generation translation platforms.

Voice AI Agents