Voice agents, bought and bundled

A voice agent that answers a company’s phone is not one product but a chain. One part listens and turns speech into text, one understands the request and acts on it, one speaks the answer, and one carries the call over the phone network or the web. Until now many builders assembled that chain from different vendors. In the second half of October the links started to arrive in one box, and the company behind Pipecat, the open framework many builders use to join them, is being bought.

Deepgram buys the plumbing

On 29 October Deepgram, which makes speech recognition and voice models, said it is acquiring Daily. Daily runs real-time infrastructure for audio and video calls and created Pipecat, an open source framework that connects the models in a voice agent and handles hard parts such as turn detection and interruptions. The price was not disclosed. “The Pipecat framework will remain fully open source and free to all developers under its existing BSD V2 license,” Deepgram wrote. Daily’s co-founder Kwindla Hultman Kramer added that an independent Pipecat Foundation will be run by a steering committee “that represents the entire realtime AI ecosystem, including companies that compete with Deepgram.”

The same day Deepgram’s product chief, Ed Anuff, previewed a Voice Agent Platform built on Pipecat, to run “hosted by Deepgram or in your own cloud”. It has not launched: “The platform itself is still ahead. We’ll share timing soon.” Most voice agents, he wrote, are “three models in a row”, and at every handoff almost everything except the words is lost. Deepgram’s answer is one company accountable for the whole chain, on a framework that also works with other vendors’ models, plus an open protocol, the Real-Time Context Standard, that carries a whisper or an unclear word from one model to the next.

The whole chain in one box

On 20 October Mistral released Voxtral 3, new models for transcription and speech with a voice agent kit built on LiveKit “that brings listening, thinking, tool calling, and speaking into a single real-time streaming pipeline.” Mistral says each turn completes in under one second and that “the agent stops talking when the caller cuts in”. The agent takes and makes phone calls and works in browsers and apps. It runs on Mistral’s EU-hosted API or on a company’s own infrastructure, for 4 cents a minute with Mistral Small 4 doing the reasoning.

ElevenLabs released Eleven Realtime the same day, in a Mini and a Max version, and calls it “a new class of Speech to Speech models”: audio in, audio out, one model name. Underneath it is still a chain, “built on a cascaded architecture of the components described above, tuned end to end.” ElevenLabs says both versions speak more than 90 languages and that, in tests by Artificial Analysis, Max starts speaking after 0.57 seconds on average.

What to test before you choose

Every launch promises natural turn taking. The test that counts is your own calls: a caller who pauses to think, changes the day halfway through a sentence, talks over the agent or rings from a noisy car. Measure how long the agent takes to answer on your own phone line, not on a benchmark, and listen for whether it really stops when interrupted.

Languages need the same care. None of the announcements we read names Swiss German. Mistral lists more than 50 dialects for speech output and Deepgram’s new voices speak German, but a caller from Bern does not speak Standard German. A preprint published in May found that the test set behind the best published Swiss German results “primarily measures convention matching rather than dialectal comprehension.” A good score on paper proves less than recordings of your own callers.

What does not come in the box

Disclosure is not optional. Since 2 August the EU’s AI Act has required that “Users must be clearly informed when they are not interacting with a real person, but an AI system”, with fines of up to 15 million euros or 3 percent of global annual turnover. In Switzerland the Federal Data Protection and Information Commissioner holds that under existing law “users have a legal right to know whether they are speaking or corresponding with a machine.” The simplest way to meet both is the first sentence of every call.

A cloned voice needs more. The commissioner writes that using programs which can falsify the voice of an identifiable person must always be clearly indicated, and may in some cases be unlawful under criminal law. Mistral offers voice cloning, and its announcement says nothing about how the speaker’s consent is checked. That part stays with the company that uses the voice: written consent from the speaker, and a record of it.

Then ask where the audio goes. Deepgram offers regional endpoints in the US, the EU, India and Australia that keep processing and storage in the region. Mistral offers its EU-hosted API or your own servers. Call recordings and transcripts usually contain personal data, so where they are kept, and for how long, belongs in the contract.

Keep the chain replaceable

Consolidation is not bad in itself. One company accountable for the whole chain can fix problems that sit between the links. The risk is the other side of it: the fewer independent suppliers there are, the harder it is to leave one.

That is why the open pieces matter. Pipecat stays under a permissive licence, and its foundation is meant to keep it neutral, although neither announcement names a committee member or a date. Mistral says each stage of its agent is a separate model, so the language model, the voice or the language can change “without rewriting the agent”. ElevenLabs follows OpenAI’s Realtime API, so switching can mean changing an address and a model name.

Our view: keep the parts that are yours outside any vendor. Your phone number, your knowledge, prices and rules, your call flows and your tests should survive a change of supplier. Then a better model, or a supplier bought by a competitor, is a change of settings and not a new project.

Where Timbre stands

Timbre is our product for this, and it is live. It clones a voice a client chooses, only with the speaker’s written consent, and puts it to work on the phone, at reception, in booking and on the website. Callers are told they are speaking with an assistant, and whenever a person matters more, the call goes to one with a written summary. We build it with voice and telephony partners such as ElevenLabs and run it ourselves. As the market consolidates, we think that is the right order: the partners underneath may change, but the consent, the disclosure and the client’s rules should not.