Nuance Labs bags $50M in funding to fix the awkwardness of AI avatars

Nuance Labs, a Seattle-based startup that’s trying to make artificial intelligence models better at face-to-face conversations, said today it has closed on a $50 million early-stage funding round to make that happen.

Today’s Series A round was led by Lightspeed Venture Partners, which previously backed the company’s seed funding round. Existing investors Accel and South Park Commons also returned, alongside new investors Nvidia Corp. and Define Ventures.

Nuance is trying to build a “human foundation model” to power avatars that can engage in conversation with humans without the lag and awkwardness associated with existing chatbots. It’s doing this by giving them more emotional intelligence, co-founder and Chief Executive Fangchang Ma said in an interview with Business Insider.

Ma, formerly an AI researcher at Apple Inc., explained that existing voicebots and AI avatars are built in a roundabout way, with developers effectively duct-taping together various models, like voice-to-text generation model, a large language model that responds to those texts, and then a text-to-voice model that transforms those responses into audio. That’s why existing chatbots tend to have delayed responses, because they have to process everything step–by-step. In the case of AI avatars, there’s a fourth step, because the model also needs to animate a face to match what the avatar is saying.

According to Ma, these handoffs strip away any feeling of engaging with a human, resulting in avatars that appear frozen when someone is talking to them. “If you use existing AI avatars, when they’re listening, they’re not reacting, or they’re just doing random things,” Ma said.

Nuance Labs wants to change this and make AI avatars more realistic, so it has developed a model that can perceive and generate responses and reactions via one full-duplex system. The system incorporates audiovisual perception of the user’s stream to understand what the person it’s engaged with is doing and saying, and at the same time, streams back an audiovisual response. This allows the model to react to verbal and non-verbal cues from the human as they’re speaking.

Ma said Nuance’s model can perceive words, gaze, gestures, tone and timing, and respond to these inputs in real time with facial and vocal expressions. It learns to do this better over time by studying how people behave during its conversations. As a result, it can demonstrate understanding in the moment, even while the user is still speaking, just as regular humans do. “We decided to build this on one system, on one model,” Ma said. “There’s audio/video in and audio/video out.”

The company has posted a demo of its model, and while it’s still a work in progress, that progress appears to have come quite far:

According to Ma, Nuance’s models enable AI avatars to engage in conversations with humans that feel more natural and productive, with better outcomes in scenarios where expressions can help drive results. He’s targeting use cases such as sales and customer service, coaching, professional training and education. So someone would be able to hop onto a video call to learn a new language, or practice an interview.

Nuance has not yet launched any product as it’s still tinkering with its model to make conversations feel even more fluid, but it hopes to have its first research preview ready to roll out to the public later this year. In the meantime, the funds from today’s round will help to accelerate those development efforts, Ma said. Nuance will also hire more researchers.

Lightspeed Venture Partners’ Nnamdi Iregbulem said there’s a “massive opportunity” to fix the way people interact with AI systems. “The founding team has a rare mix of technical talent and operating excellence, and they’ve turned that into a single model that follows you in real time,” he said. “We can see this becoming a foundational layer for AI products everywhere, and this is the team to build it.”

Photo: Nuance Labs