Zoom has opened the speech recognition behind its My Notes tool to other companies, and is in conversation with some AI note-taking firms that are considering switching to it, chief technology officer XD Huang told TNW. He said Zoom wants AI that finishes the work discussed in a meeting before the meeting ends.
Huang leads Zoom’s AI strategy. Before joining Zoom, he spent 30 years at Microsoft, where he was a technical fellow and CTO of Azure AI and helped launch the Azure OpenAI Service. His teams reached human parity in speech recognition, machine translation, natural language understanding and computer vision. He holds more than 170 US patents and is a member of the US National Academy of Engineering.
Huang spoke to Cristian Dina for the second episode of the newly launched TNW Podcast, recorded on 22 September in Amsterdam at a Zoom event ahead of HumanX. The first episode featured Yahoo’s Jim Lanzone. The full conversation is in the video below.
“Just imagine the possibility. You can afford to bring Steve Jobs to every meeting you are attending,” Huang said.
Huang calls the shift “C2C”, from conversation to completion. The first C is the meeting business Zoom built, above all in the pandemic. The second is AI that completes tasks on a user’s behalf, with their authorisation. He said 2026 would be “the year of completion”.
My Notes captures conversations in Zoom, Microsoft and Google meetings, as well as in person, Huang said. Most people do not realise it works outside Zoom, he added.
Opening up the Scribe API
Startups built bots that join calls and send summaries, and Dina asked whether Zoom had crushed them with My Notes.
“I wouldn’t say we crashed. And actually, we open this up,” Huang said.
The technology behind My Notes is available to anyone as the Scribe API, part of Zoom AI Services. Huang said Scribe has stayed on top of the Open ASR Leaderboard on Hugging Face, which benchmarks more than 60 speech recognition systems, since its first version. Version two is now out.
He said Scribe also powers ZoomMate, Zoom’s AI assistant, and its customer support tools. Search across Zoom, Google Drive and Microsoft documents adds the context, with connectors to Slack, Jira and Salesforce.
A panel of judges
Dina asked about calls to slow AI development. Huang said Zoom focuses on helping people safely, securely and with compliance, and that it does not rely on one model. He calls this a federated approach: several models check each other to cut hallucinations and improve accuracy.
“When you actually play gymnastics, you know why there are multiple judges,” he said.
On Humanity’s Last Exam, a test of expert knowledge, combining several frontier models beat each of them alone, Huang said. Zoom uses models from OpenAI and Anthropic together with its own small language model. Simple tasks go to smaller models, and only the hardest go to frontier models.
Zoom built its own speech recognition and translation, he said. Its meeting summarisation model is as good as ChatGPT at far lower cost, because it is trained for that one task.
Forty years on speech
Born in China and educated in Scotland, Xuedong Huang started work on speech recognition in 1982, as a graduate student in China. Keyboards were built for Western languages, which made Chinese hard to type, he said. He joined Microsoft before he was 30 and founded its Speech Technology Group. Zoom’s main client has been downloaded more than a billion times, he said. That figure covers the client as a whole, not ZoomMate or its language features.
He uses Codex, Claude Code and ZoomMate every day, and turns on My Notes in every meeting, lecture and workshop. He then asks ZoomMate what three big things he missed.
He does not think the language barrier is solved. The Scribe API’s error rate rises from about 3% to five or six per cent on the Speech Accessibility Project at the University of Illinois Urbana-Champaign. That project records people who have had a stroke or cannot speak clearly. Zoom offers its best transcription in about ten languages, and Huang wants to cover all 24 official EU languages.
Asked if Zoom would build a wearable pin to record conversations, he said Zoom does not announce products it has not launched. Zoom does not manufacture hardware, but holds partnerships with the likes of Neat, HP Poly, Logitech and Yealink, the company said.
He said humans still own the final decision, and AI only helps them make it.
“The more AI slop you have to deal with, the more you would like to talk to people like me,” Huang said.
Get the TNW newsletter
Get the most important tech news in your inbox each week.
