Eleven Realtime vs OpenAI Realtime API: Inside ElevenLabs' All-in-One Speech-to-Speech Bet
ElevenLabs’ Eleven Realtime pitches one speech-to-speech model on an OpenAI-compatible API—what that means for latency, voice quality, and vendor switching.
Posted by
Related reading
Composio Instant: The Agent Tool Wallet, Revisited After Its Product Hunt Resurgence
Composio Instant gives agents a prepaid wallet and 1,500+ MCP tools—we revisit the Product Hunt resurgence, wallet-vs-virtual-card trade-offs, and spending guardrails.
T3 Code: The Open-Source Coding Agent Control Plane, and What Its Latest Update Changes
T3 Code turns scattered coding agents into one control plane—what the late-October update adds for remote iOS simulators, connect speed, and shared threads.
tstack: Theo Browne's Open-Source Claude Code Skills, Explained
Theo Browne’s tstack bundles nine Claude Code skills for daily work—inside the PR-merge pipeline, the curation debate, and how to vet third-party skills safely.
What speech-to-speech agents actually do
Most production voice agents are a relay: automatic speech recognition transcribes the caller, an LLM drafts the reply, and text-to-speech speaks it. Each hop adds latency and throws away signal—pauses, tone, interruption timing. Speech-to-speech models compress the loop into audio in, reasoning, audio out, within one model session. ElevenLabs’ Eleven Realtime, announced at elevenlabs.io/realtime, is positioned as exactly this kind of unified agent model, and our ElevenLabs realtime voice agent hub tracks how the rollout has progressed.
The OpenAI-compatible-API strategy
As announced and reported, Eleven Realtime is compatible with the OpenAI Realtime API surface. If that holds in practice, a team that built on OpenAI’s websocket events protocol can point its client at a different endpoint with minimal code changes—the URL and the key change, the event handling mostly does not. The strategic read is that API contracts, not model weights, are what lock teams in, and OpenAI-compatible has become the de facto interchange format for realtime voice. Compatibility claims also drift, so diff the event schemas yourself before migrating anything customer-facing.
What all-in-one actually bundles
A single session covers recognition, the reasoning turn, and synthesis. Fewer network hops sit between the caller and the first word of the response, interruption handling becomes a property of the model rather than a state machine you wrote, and the voice identity stays consistent across turns. That last point matters more than teams expect—stitched pipelines tend to drift in prosody on long calls, which callers read as the bot not listening. For character-facing use cases rather than phone calls, the same logic shows up in talking-character stacks.
Trade-offs against assembling your own pipeline
A stitched pipeline lets you pick best-of-breed per stage: domain-tuned recognition for medical or legal jargon, a proprietary voice, a cheap model for small talk and a frontier model for escalations, plus per-stage observability. All-in-one wins time-to-ship and coherence; stitched wins cost tuning and control. If your constraint is self-hosting or on-device latency, the comparison changes again—see our coverage of local voice AI stacks and open speech-to-speech work.
An evaluation checklist before you switch
- Latency: time-to-first-audio at p50 and p95 on your real telephony route, not on localhost.
- Interruptions: does it recover cleanly when a caller cuts in mid-word, twice in a row?
- Voice quality: consistency over twenty-plus turns, including numbers, dates, and spell-outs.
- Cost: price per conversation minute at your traffic shape, including failed or abandoned calls.
Teams we have spoken with who run this checklist in a day find vendor switching far cheaper than they assumed—mostly because the OpenAI-compatible surface makes the swap a config change rather than a rewrite. The realtime voice agents hub collects the current provider landscape if you want a second candidate for the same test script.