A voice agent that handles English cleanly will not survive its first thousand calls in India. Not because the models are weak, but because the assumptions underneath them — one language per call, clean audio, speakers who finish their sentences — describe a conversation that does not happen here.
This is a note on the failure modes we keep meeting in production, and what it takes to design around them. It is written for people evaluating voice agents, not just buying them.
Nobody speaks one language per call
The single biggest gap between a demo and a deployment is code-switching. A retailer confirming an order does not speak Hindi, then English. They speak both inside one sentence, and the switch lands on exactly the words that matter — the quantity, the SKU, the price.
Systems that detect language once at the start of a call get this wrong in a specific and expensive way: they lock to Hindi, then mis-transcribe the English product name, and the order goes in wrong. The failure is silent. Nobody hears an error; they hear a confident, incorrect number.
Haan bhai, do carton bhej do — uhm wahi wala, five star model, aur rate thoda adjust karna.
One sentence, two languages, a product name in English, a quantity in Hindi and a hedge in the middle. Detect-once language routing mis-handles at least one of these.
Handling this means treating language as a property of the utterance rather than the call, and accepting that Hinglish is not a degraded form of either language — it is the actual language being spoken. Our speech recognition runs English and Hindi today, with further languages in training; on the synthesis side the same voice identity carries across all 14 Indian and 15 international languages, so an agent that greets in Hindi and switches to English still sounds like one person rather than two.
Telephony audio is not the audio you tested on
Most model evaluation happens on clean, wideband recordings. A production call arrives narrowband, compressed by a codec built in the 1970s, over a mobile connection, from a shop floor with a ceiling fan and a second conversation in the background.
This is worth stating plainly because it explains a common procurement mistake: benchmark numbers published on clean corpora tell you very little about performance on a real trunk. The only number that means anything is the one measured on your own call recordings, in your own languages, through your own telephony path.
- Ask for word error rate on narrowband telephony audio, not on read speech. The gap between the two is often larger than the gap between vendors.
- Ask what happens on barge-in — when the caller talks over the agent, which in practice they always do.
- Ask about time to first token, not just total latency. A caller judges responsiveness by the pause before the agent starts, not by how long it speaks.
- Ask what the agent does when it is unsure. An agent that guesses confidently is worse than one that hands over.
Tone carries the information the transcript loses
A transcript records what was said. On a collections call, a support escalation, or a patient asking about a test result, the useful signal is often how it was said — the hesitation before a commitment, the sharpening of a voice that is about to become a complaint.
We tag that signal inline rather than discarding it, using a fixed emotion vocabulary the downstream flow can branch on: [whisper] [laugh] [angry] [sigh] [shout] [silence] [neutral] [uhm]. A pause is not nothing. In a payment conversation it is frequently the most informative event in the call.
Haan haan, pata hai... sigh is baar thoda late ho jayega, silence salary abhi tak aayi nahi hai.
The words commit to paying. The sigh and the pause are what tell you this account needs a different follow-up than one that simply says yes.
Volume changes which problems matter
At a hundred calls a day, edge cases are anecdotes. At twenty thousand, they are a line item. The failure modes that dominate at scale are rarely the ones that dominate in a pilot.
Across deployments we run, a few patterns hold. Connection rate governs everything downstream — a campaign is capped by how many people pick up long before it is capped by how persuasive the agent is. Escalation paths get used far more than anyone forecasts. And the calls that go wrong are overwhelmingly the ones where the agent should have stopped and did not.
The agent triages but never advises clinically — anything critical goes to a human, not to a better answer.
That rule is worth generalising. The question to ask of any voice agent is not how much it can handle, but how cleanly it gives up. In our inbound healthcare deployment, roughly 82% of calls resolve without a human — the number that matters more is that a critical call reaches a supervisor in under ten seconds.
What to actually evaluate
If you are comparing platforms, most public comparisons rank on language count. Language count is close to meaningless on its own — it says nothing about whether those languages work on a real call, or whether the voice stays consistent when the caller switches mid-sentence.
A more useful checklist:
- Run your own audio through the recogniser — your trunk, your accents, your background noise.
- Test a deliberately code-switched utterance with a product name or number in the second language.
- Interrupt the agent mid-sentence and see whether it yields.
- Force an escalation and time it end to end.
- Ask where the models are hosted and trained, and whether call audio leaves your jurisdiction.
- Ask what happens on the hundredth concurrent call, not the first.
The last two are where most evaluations stop short. Data residency is a procurement question with a technical answer, and concurrency behaviour is invisible until the day it is not.
If you want to try these against our stack, the documentation covers the speech APIs, or you can get in touch and we will run your own recordings through it.