Speech-to-Speech Model Research at Google DeepMind — Valeria Wu Fon & Tom Ouyang, Google DeepMind

>> Hi everyone. Thanks for coming to our >> Hi everyone. Thanks for coming to our talk. We're going to talk a lot about talk. We're going to talk a lot about talk. We're going to talk a lot about how we are planning on voiceifying the how we are planning on voiceifying the how we are planning on voiceifying the agentic future with speech-to-speech agentic future with speech-to-speech agentic future with speech-to-speech research here at Google DeepMind. My research here at Google DeepMind. My research here at Google DeepMind. My name is Valeria. I'm the product lead name is Valeria. I'm the product lead name is Valeria. I'm the product lead for the speech-to-speech model in Gemini for the speech-to-speech model in Gemini for the speech-to-speech model in Gemini and Tom. and Tom. and Tom. >> Yeah, my name is Tom. I work on >> Yeah, my name is Tom. I work on >> Yeah, my name is Tom. I work on speech-to-speech as an engineer and speech-to-speech as an engineer and speech-to-speech as an engineer and Gemini. So, yeah, great to be presenting Gemini. So, yeah, great to be presenting Gemini. So, yeah, great to be presenting our work. our work. our work. >> Cool. >> Cool. >> Cool. So, before we start diving into the So, before we start diving into the So, before we start diving into the history of Google research and audio and history of Google research and audio and history of Google research and audio and the latest research we've been doing, I the latest research we've been doing, I the latest research we've been doing, I just want to take a step back and really just want to take a step back and really just want to take a step back and really kind of like think about why this is kind of like think about why this is kind of like think about why this is important to us in the team. And the important to us in the team. And the important to us in the team. And the main reason is that voice is the most main reason is that voice is the most main reason is that voice is the most natural way for humans to interact with natural way for humans to interact with natural way for humans to interact with both the physical and the virtual world. both the physical and the virtual world. both the physical and the virtual world. And we're already seeing so many And we're already seeing so many And we're already seeing so many applications that are starting even applications that are starting even applications that are starting even within the own Google products. We see within the own Google products. We see within the own Google products. We see voice being used to ask questions and voice being used to ask questions and voice being used to ask questions and you know, do homework or info seeking or you know, do homework or info seeking or you know, do homework or info seeking or EDU in search live and Gemini live. All EDU in search live and Gemini live. All EDU in search live and Gemini live. All the way to these same models being the way to these same models being the way to these same models being deployed in cloud and the API for deployed in cloud and the API for deployed in cloud and the API for enterprise or voice agent use cases. enterprise or voice agent use cases. enterprise or voice agent use cases. Should I use this as a Should I use this as a Should I use this as a microphone?

microphone? microphone? Hello. Okay. Hello. Okay. Hello. Okay. Um Um Um So, because of that, we only think that So, because of that, we only think that So, because of that, we only think that the number of applications is going to, the number of applications is going to, the number of applications is going to, you know, exponentially increase over you know, exponentially increase over you know, exponentially increase over the next few years and we believe that the next few years and we believe that the next few years and we believe that speech-to-speech models are the way to speech-to-speech models are the way to speech-to-speech models are the way to go when we want to build robust go when we want to build robust go when we want to build robust universal voice agents. And with that, universal voice agents. And with that, universal voice agents. And with that, I'll hand it off to Tom. I'll hand it off to Tom. I'll hand it off to Tom. >> All right. So, of course, like one way >> All right. So, of course, like one way >> All right. So, of course, like one way to create a voice agent is to just add to create a voice agent is to just add to create a voice agent is to just add an ASR like speech-to-text model in the an ASR like speech-to-text model in the an ASR like speech-to-text model in the pipeline and then a text-to-speech model pipeline and then a text-to-speech model pipeline and then a text-to-speech model on the other end, right? So, of course, on the other end, right? So, of course, on the other end, right? So, of course, that's where a lot of the history of that's where a lot of the history of that's where a lot of the history of speech come from. Of course, Google has speech come from. Of course, Google has speech come from. Of course, Google has been working on this for a long time. been working on this for a long time. been working on this for a long time. I'm going to give you a little bit of a I'm going to give you a little bit of a I'm going to give you a little bit of a historical overview of what speech historical overview of what speech historical overview of what speech modeling, especially speech-to-text modeling, especially speech-to-text modeling, especially speech-to-text automatic automatic automatic speech recognition, typically looks speech recognition, typically looks speech recognition, typically looks like, right? So, up until around 2018, like, right? So, up until around 2018, like, right? So, up until around 2018, this usually involved a lot of different this usually involved a lot of different this usually involved a lot of different components. You have um you know of components. You have um you know of components. You have um you know of course feature extraction that's fairly course feature extraction that's fairly course feature extraction that's fairly general and then you have all these general and then you have all these general and then you have all these different pieces like acoustic modeling, different pieces like acoustic modeling, different pieces like acoustic modeling, pronunciation modeling, language pronunciation modeling, language pronunciation modeling, language modeling, a second pass rescoring that modeling, a second pass rescoring that modeling, a second pass rescoring that allows you to go from the audio input to allows you to go from the audio input to allows you to go from the audio input to a text transcription. Right? And of a text transcription. Right? And of a text transcription. Right? And of course like around 2018, these moved course like around 2018, these moved course like around 2018, these moved more and more towards end-to-end more and more towards end-to-end more and more towards end-to-end systems. You don't have to have so much systems. You don't have to have so much systems. You don't have to have so much domain knowledge. You can actually have domain knowledge. You can actually have domain knowledge. You can actually have mostly the the neural model learn this mostly the the neural model learn this mostly the the neural model learn this pattern and mapping between acoustic pattern and mapping between acoustic pattern and mapping between acoustic inputs and text. But these aren't really inputs and text. But these aren't really inputs and text. But these aren't really the end-to-end models that we think the end-to-end models that we think the end-to-end models that we think about when we think about LLMs. They're about when we think about LLMs. They're about when we think about LLMs. They're only doing kind of one thing, which is only doing kind of one thing, which is only doing kind of one thing, which is speech to a transcription of that speech to a transcription of that speech to a transcription of that speech. They're not responding, they're speech. They're not responding, they're speech. They're not responding, they're not translating. If you wanted to have not translating. If you wanted to have not translating. If you wanted to have the model tell you about the tone or the the model tell you about the tone or the the model tell you about the tone or the emotion or the speed of the speaker. If emotion or the speed of the speaker. If emotion or the speed of the speaker. If you wanted to bias it towards words or you wanted to bias it towards words or you wanted to bias it towards words or much less like images, those are all much less like images, those are all much less like images, those are all things you have to build yourself as things you have to build yourself as things you have to build yourself as part of the system and there's really part of the system and there's really part of the system and there's really really a barrier to how easily you can

really a barrier to how easily you can really a barrier to how easily you can scale these systems. So, scale these systems. So, scale these systems. So, fast forward to now, which is kind of of fast forward to now, which is kind of of fast forward to now, which is kind of of course the LLM era, right? Like you course the LLM era, right? Like you course the LLM era, right? Like you know, the first LLMs were mostly text, know, the first LLMs were mostly text, know, the first LLMs were mostly text, but even there I think you could kind of but even there I think you could kind of but even there I think you could kind of hack audio embeddings into these text hack audio embeddings into these text hack audio embeddings into these text models and it kind of worked, but now of models and it kind of worked, but now of models and it kind of worked, but now of course for a long time now, Gemini course for a long time now, Gemini course for a long time now, Gemini models have been very natively models have been very natively models have been very natively multimodal. So, what does that mean? It multimodal. So, what does that mean? It multimodal. So, what does that mean? It means that when we train these models in means that when we train these models in means that when we train these models in pre-training, which is where the bulk of pre-training, which is where the bulk of pre-training, which is where the bulk of the data comes from, these are the data comes from, these are the data comes from, these are multimodal interleaved examples, right? multimodal interleaved examples, right? multimodal interleaved examples, right? So, the bottom uh diagram here gives you So, the bottom uh diagram here gives you So, the bottom uh diagram here gives you kind of one example of what that might kind of one example of what that might kind of one example of what that might look like. So, this is a task where look like. So, this is a task where look like. So, this is a task where you're asking the model to summarize a you're asking the model to summarize a you're asking the model to summarize a bedtime story and there's a text prompt bedtime story and there's a text prompt bedtime story and there's a text prompt in the beginning, but then there's this in the beginning, but then there's this in the beginning, but then there's this sequence of video and audio inputs that sequence of video and audio inputs that sequence of video and audio inputs that the model gets the model gets the model gets and then of course like what you expect and then of course like what you expect and then of course like what you expect the model to do here is produce um both the model to do here is produce um both the model to do here is produce um both the summary and also annotate timestamps the summary and also annotate timestamps the summary and also annotate timestamps for where interesting things happen and for where interesting things happen and for where interesting things happen and so forth. So, this example is teaching so forth. So, this example is teaching so forth. So, this example is teaching the model to translate its understanding the model to translate its understanding the model to translate its understanding of the audio and the video into text. of the audio and the video into text. of the audio and the video into text. You might have other examples in You might have other examples in You might have other examples in pre-training that ask you to caption a pre-training that ask you to caption a pre-training that ask you to caption a video. So, you might have a video that video. So, you might have a video that video. So, you might have a video that has audio and the model is learning to has audio and the model is learning to has audio and the model is learning to bias towards both the video and the bias towards both the video and the bias towards both the video and the audio signal to caption this well. And audio signal to caption this well. And audio signal to caption this well. And of course like there's limitless like of course like there's limitless like of course like there's limitless like YouTube videos with captions that you YouTube videos with captions that you YouTube videos with captions that you can train these models on. Other models can train these models on. Other models can train these models on. Other models might actually try to generate audio might actually try to generate audio might actually try to generate audio from the video from the text, right? So, from the video from the text, right? So, from the video from the text, right? So, you can have ASR, TTS, or any you can have ASR, TTS, or any you can have ASR, TTS, or any combination of these plus all of these combination of these plus all of these combination of these plus all of these sort of agentic tasks all kind of sort of agentic tasks all kind of sort of agentic tasks all kind of learned under one unified token learned under one unified token learned under one unified token embedding space. So, this becomes a embedding space. So, this becomes a embedding space. So, this becomes a foundation for a lot of what we want to foundation for a lot of what we want to foundation for a lot of what we want to do in audio because we already have a do in audio because we already have a do in audio because we already have a model that understands audio, video, model that understands audio, video, model that understands audio, video, text, and how these things relate and text, and how these things relate and text, and how these things relate and transition from one to the next.

transition from one to the next. transition from one to the next. So, very quickly, like one of the So, very quickly, like one of the So, very quickly, like one of the applications that this enables that applications that this enables that applications that this enables that we've launched recently is live we've launched recently is live we've launched recently is live translation, right? And this is a kind translation, right? And this is a kind translation, right? And this is a kind of application that kind of only works of application that kind of only works of application that kind of only works when you have all of these capabilities when you have all of these capabilities when you have all of these capabilities working within the same model. You have working within the same model. You have working within the same model. You have basically state-of-the-art translation basically state-of-the-art translation basically state-of-the-art translation quality. Even though this model is quality. Even though this model is quality. Even though this model is translating basically as the user or translating basically as the user or translating basically as the user or speakers are speaking, you kind of ask, speakers are speaking, you kind of ask, speakers are speaking, you kind of ask, "Hey, I I speak English. There's maybe "Hey, I I speak English. There's maybe "Hey, I I speak English. There's maybe friends who are talking in Spanish and friends who are talking in Spanish and friends who are talking in Spanish and Italian and Chinese." And it's Italian and Chinese." And it's Italian and Chinese." And it's translating all of them to your language translating all of them to your language translating all of them to your language as they talk. Um and then we're we're as they talk. Um and then we're we're as they talk. Um and then we're we're finding is the translation quality for finding is the translation quality for finding is the translation quality for this like streaming real-time this like streaming real-time this like streaming real-time translation is about as good as you translation is about as good as you translation is about as good as you would get with offline systems, right? would get with offline systems, right? would get with offline systems, right? Where you kind of know the full Where you kind of know the full Where you kind of know the full utterance utterance utterance um from the very beginning. So, that's um from the very beginning. So, that's um from the very beginning. So, that's something that has been classically very something that has been classically very something that has been classically very hard to do with these cascaded systems, hard to do with these cascaded systems, hard to do with these cascaded systems, but with LLMs, it actually just but with LLMs, it actually just but with LLMs, it actually just a lot of it comes out of the a lot of it comes out of the a lot of it comes out of the pre-training. So, of course, to do this pre-training. So, of course, to do this pre-training. So, of course, to do this task it needs to do multilingual task it needs to do multilingual task it needs to do multilingual switching because you could be switching because you could be switching because you could be translating across different languages. translating across different languages. translating across different languages. You don't know what those languages are You don't know what those languages are You don't know what those languages are beforehand. It needs to preserve the beforehand. It needs to preserve the beforehand. It needs to preserve the speaker voice of the source speaker and speaker voice of the source speaker and speaker voice of the source speaker and be able to understand multiple speakers, be able to understand multiple speakers, be able to understand multiple speakers, be robust to noise, and of course, like be robust to noise, and of course, like be robust to noise, and of course, like do all this in real time, right? So, do all this in real time, right? So, do all this in real time, right? So, again, it would be very hard to try to again, it would be very hard to try to again, it would be very hard to try to engineer this, but then with the LLM and engineer this, but then with the LLM and engineer this, but then with the LLM and Gemini models, this almost becomes a Gemini models, this almost becomes a Gemini models, this almost becomes a prompting task.

prompting task. prompting task. And on that, like, I know, in this And on that, like, I know, in this And on that, like, I know, in this diagram we're saying, "Hey, at the top diagram we're saying, "Hey, at the top diagram we're saying, "Hey, at the top with these models, if you prompt it to with these models, if you prompt it to with these models, if you prompt it to do the speaking like this streaming do the speaking like this streaming do the speaking like this streaming translation task, translation task, translation task, and you give it the audio, it will and you give it the audio, it will and you give it the audio, it will produce the streaming translation produce the streaming translation produce the streaming translation output, right? The same model, if you output, right? The same model, if you output, right? The same model, if you ask it to act like an agent and respond ask it to act like an agent and respond ask it to act like an agent and respond to maybe image and audio input, maybe to maybe image and audio input, maybe to maybe image and audio input, maybe asking questions about that image, it asking questions about that image, it asking questions about that image, it will give you an answer. And finally, will give you an answer. And finally, will give you an answer. And finally, like very well we'll show examples of like very well we'll show examples of like very well we'll show examples of this, you can also have it create this this, you can also have it create this this, you can also have it create this embodied, you know, virtual agent that embodied, you know, virtual agent that embodied, you know, virtual agent that has a face, that has, you know, things has a face, that has, you know, things has a face, that has, you know, things and tools that it can show you, and it and tools that it can show you, and it and tools that it can show you, and it will produce this sort of embodied agent will produce this sort of embodied agent will produce this sort of embodied agent experience. So, with that, I'm going to experience. So, with that, I'm going to experience. So, with that, I'm going to give it to Valeria to talk more about give it to Valeria to talk more about give it to Valeria to talk more about the North Star and some of the key demo the North Star and some of the key demo the North Star and some of the key demo products that we built. products that we built. products that we built. >> Yeah, so to create this type of kind of >> Yeah, so to create this type of kind of >> Yeah, so to create this type of kind of universal, versatile, uh kind of universal, versatile, uh kind of universal, versatile, uh kind of model/product, model/product, model/product, there's like three vectors that we think there's like three vectors that we think there's like three vectors that we think about when we do research and product about when we do research and product about when we do research and product for these models. Um and they also come for these models. Um and they also come for these models. Um and they also come with some challenges, so I'll like walk with some challenges, so I'll like walk with some challenges, so I'll like walk you through some of them. So, at the you through some of them. So, at the you through some of them. So, at the core of a speech-to-speech model, the core of a speech-to-speech model, the core of a speech-to-speech model, the first thing that people usually think first thing that people usually think first thing that people usually think about is that it has very about is that it has very about is that it has very conversational, right? It's low latency, conversational, right? It's low latency, conversational, right? It's low latency, it's very conversational, very snappy, it's very conversational, very snappy, it's very conversational, very snappy, very natural. But, I think within our very natural. But, I think within our very natural. But, I think within our team, we really don't only want this team, we really don't only want this team, we really don't only want this model to sound nice. We also have two model to sound nice. We also have two model to sound nice. We also have two pillars at the top that we also really pillars at the top that we also really pillars at the top that we also really care about like pulling all together care about like pulling all together care about like pulling all together into one model, which is intelligence into one model, which is intelligence into one model, which is intelligence and it being multimodal. So, when we and it being multimodal. So, when we and it being multimodal. So, when we talk about intelligence, we talk about, talk about intelligence, we talk about, talk about intelligence, we talk about, you know, task completion, instruction you know, task completion, instruction you know, task completion, instruction following, reasoning, like capabilities following, reasoning, like capabilities following, reasoning, like capabilities that the model needs to have natively in that the model needs to have natively in that the model needs to have natively in order to complete tasks and to like do order to complete tasks and to like do order to complete tasks and to like do things uh that have high customer things uh that have high customer things uh that have high customer satisfaction, for example. And on the

satisfaction, for example. And on the satisfaction, for example. And on the other side of the Venn diagram, we also other side of the Venn diagram, we also other side of the Venn diagram, we also have the idea that these models should have the idea that these models should have the idea that these models should be very multimodal, both in audio in and be very multimodal, both in audio in and be very multimodal, both in audio in and audio out, right? So, sometimes a user audio out, right? So, sometimes a user audio out, right? So, sometimes a user doesn't only want to input audio in and doesn't only want to input audio in and doesn't only want to input audio in and have that be the start of the have that be the start of the have that be the start of the conversation. We need video, your screen conversation. We need video, your screen conversation. We need video, your screen sharing, PDFs, whatever you would want sharing, PDFs, whatever you would want sharing, PDFs, whatever you would want the model to interpret and understand, the model to interpret and understand, the model to interpret and understand, we should be able to stream it in and we should be able to stream it in and we should be able to stream it in and also produce output out of it. So, also produce output out of it. So, also produce output out of it. So, that's kind of like the trifecta of that's kind of like the trifecta of that's kind of like the trifecta of which we think about speech-to-speech which we think about speech-to-speech which we think about speech-to-speech models. Um and I want to add a caveat models. Um and I want to add a caveat models. Um and I want to add a caveat about ITNN. I think actually the about ITNN. I think actually the about ITNN. I think actually the majority of our Gemini users are majority of our Gemini users are majority of our Gemini users are non-English speakers. Um so, we put a non-English speakers. Um so, we put a non-English speakers. Um so, we put a big focus on having and making sure that big focus on having and making sure that big focus on having and making sure that all these capabilities work within not all these capabilities work within not all these capabilities work within not only, you know, EN-US, but all the only, you know, EN-US, but all the only, you know, EN-US, but all the languages that our customers care about. languages that our customers care about. languages that our customers care about. Um Um Um of course, this also our North Star also of course, this also our North Star also of course, this also our North Star also becomes one of the biggest challenges in becomes one of the biggest challenges in becomes one of the biggest challenges in our research because our research because our research because once you move one of the knobs, it's once you move one of the knobs, it's once you move one of the knobs, it's very easy for the other knobs to kind of very easy for the other knobs to kind of very easy for the other knobs to kind of like mess up, right? Like a very quick like mess up, right? Like a very quick like mess up, right? Like a very quick example, oh, how do we increase example, oh, how do we increase example, oh, how do we increase intelligence in the model? Well, you can intelligence in the model? Well, you can intelligence in the model? Well, you can turn thinking high or like the thinking turn thinking high or like the thinking turn thinking high or like the thinking is high as possible to have the model is high as possible to have the model is high as possible to have the model think a lot before calling a tool or think a lot before calling a tool or think a lot before calling a tool or answering a question, which in eval's it answering a question, which in eval's it answering a question, which in eval's it does show that it does improve the does show that it does improve the does show that it does improve the model's intelligence. But when what does model's intelligence. But when what does model's intelligence. But when what does that do to latency, right? And time to that do to latency, right? And time to that do to latency, right? And time to first audio and the naturalness of the first audio and the naturalness of the first audio and the naturalness of the conversation? So, within the Gemini conversation? So, within the Gemini conversation? So, within the Gemini team, we're really trying to push team, we're really trying to push team, we're really trying to push forward research initiatives that can forward research initiatives that can forward research initiatives that can kind of blend in the three of them kind of blend in the three of them kind of blend in the three of them without really sacrificing any of those without really sacrificing any of those without really sacrificing any of those by a lot. Um, by a lot. Um, by a lot. Um, but in the meantime, we're going to show but in the meantime, we're going to show but in the meantime, we're going to show you some of the demos that we think are you some of the demos that we think are you some of the demos that we think are hinting at how our speech-to-speech hinting at how our speech-to-speech hinting at how our speech-to-speech model can combine all these three into

model can combine all these three into model can combine all these three into really cool application. So, really cool application. So, really cool application. So, the first one is you kind of saw this as the first one is you kind of saw this as the first one is you kind of saw this as a preview, but our live model, as you a preview, but our live model, as you a preview, but our live model, as you know, uh, powers streaming translation know, uh, powers streaming translation know, uh, powers streaming translation that supports 70+ languages. So, this is that supports 70+ languages. So, this is that supports 70+ languages. So, this is a little bit around the core of a little bit around the core of a little bit around the core of conversation quality and IT&N efforts conversation quality and IT&N efforts conversation quality and IT&N efforts that we have in the team. I'll play a that we have in the team. I'll play a that we have in the team. I'll play a quick video on how this works on Google quick video on how this works on Google quick video on how this works on Google Meets to help two people or maybe Meets to help two people or maybe Meets to help two people or maybe multiple people that are speaking multiple people that are speaking multiple people that are speaking different languages still have a live different languages still have a live different languages still have a live conversation. conversation. conversation. Oh. Oh. Oh. One sec. Okay. One sec. Okay. One sec. Okay. >> Okay, let's turn on speech translation. >> So, here the user can simply select the >> So, here the user can simply select the language that they want the translation language that they want the translation language that they want the translation to happen in and then the rest will be to happen in and then the rest will be to happen in and then the rest will be done in real time in multiple speakers done in real time in multiple speakers done in real time in multiple speakers kind of having a conversation back and kind of having a conversation back and kind of having a conversation back and forth. So, I'll show you a snippet of forth. So, I'll show you a snippet of forth. So, I'll show you a snippet of this video of what happens. >> It's great to see you both. Cassie, >> It's great to see you both. Cassie, how's the weather in Shanghai? how's the weather in Shanghai? how's the weather in Shanghai? >> It's nice to meet you. The weather here >> It's nice to meet you. The weather here >> It's nice to meet you. The weather here is really nice, sunny and bright. I is really nice, sunny and bright. I is really nice, sunny and bright. I spent all day Saturday in the park with spent all day Saturday in the park with spent all day Saturday in the park with my family.

>> That sounds wonderful. >> That sounds wonderful. Anna, you mentioned last week that you Anna, you mentioned last week that you Anna, you mentioned last week that you were celebrating your birthday. How was were celebrating your birthday. How was were celebrating your birthday. How was it? it? it? >> Oh, that was fantastic. I had dinner at >> Oh, that was fantastic. I had dinner at >> Oh, that was fantastic. I had dinner at my favorite restaurant with some my favorite restaurant with some my favorite restaurant with some friends. If you visit Sweden, you must friends. If you visit Sweden, you must friends. If you visit Sweden, you must try this restaurant. It was try this restaurant. It was try this restaurant. It was >> So, um a lot of things happening. Not >> So, um a lot of things happening. Not >> So, um a lot of things happening. Not only there's, you know, real-time only there's, you know, real-time only there's, you know, real-time translation. Whoops. translation. Whoops. translation. Whoops. How do I How do I How do I back to Okay. back to Okay. back to Okay. Um not only there's real-time Um not only there's real-time Um not only there's real-time translation that is happening, uh but we translation that is happening, uh but we translation that is happening, uh but we see that it's in a multi-speaker see that it's in a multi-speaker see that it's in a multi-speaker setting, you know, low latency, like setting, you know, low latency, like setting, you know, low latency, like right after the user starts speaking, right after the user starts speaking, right after the user starts speaking, the translation kicks off, so that it the translation kicks off, so that it the translation kicks off, so that it doesn't feel like it's really turn by doesn't feel like it's really turn by doesn't feel like it's really turn by turn and robotic and you're afraid to turn and robotic and you're afraid to turn and robotic and you're afraid to interrupt because, you know, once you interrupt because, you know, once you interrupt because, you know, once you start speaking, the model will start start speaking, the model will start start speaking, the model will start catching your translation directly. So, catching your translation directly. So, catching your translation directly. So, now I'll show some other applications now I'll show some other applications now I'll show some other applications where we see that we have one single where we see that we have one single where we see that we have one single speech-to-speech model for many speech-to-speech model for many speech-to-speech model for many conversational frontiers. So, two conversational frontiers. So, two conversational frontiers. So, two products that I want to highlight here products that I want to highlight here products that I want to highlight here are the same model that we power search are the same model that we power search are the same model that we power search live for everyday conversations in any live for everyday conversations in any live for everyday conversations in any language. It's the same model that we language. It's the same model that we language. It's the same model that we use to power developer experiences in use to power developer experiences in use to power developer experiences in our live API. As you'll see in the our live API. As you'll see in the our live API. As you'll see in the videos, both use cases are extremely videos, both use cases are extremely videos, both use cases are extremely different, but because we try and make different, but because we try and make different, but because we try and make this this model or this speech-to-speech this this model or this speech-to-speech this this model or this speech-to-speech model very promptable and universal, you model very promptable and universal, you model very promptable and universal, you can see how it adapts to the different can see how it adapts to the different can see how it adapts to the different use cases. So, let me just play the use cases. So, let me just play the use cases. So, let me just play the first video.

So, to highlight in this video, many So, to highlight in this video, many things are happening. First, the things are happening. First, the things are happening. First, the multimodal, you know, it's real-time uh multimodal, you know, it's real-time uh multimodal, you know, it's real-time uh video and audio streaming in, so that video and audio streaming in, so that video and audio streaming in, so that you don't have to actually describe what you don't have to actually describe what you don't have to actually describe what the sofa looks like. But then the the sofa looks like. But then the the sofa looks like. But then the multimodal out, you see that there was a multimodal out, you see that there was a multimodal out, you see that there was a tool called made to pull up the right tool called made to pull up the right tool called made to pull up the right search cards if the user wanted to know search cards if the user wanted to know search cards if the user wanted to know more information. And then finally, the more information. And then finally, the more information. And then finally, the ITN component, you know, we're ITN component, you know, we're ITN component, you know, we're localizing the speaker's Spanish to localizing the speaker's Spanish to localizing the speaker's Spanish to Spain Spanish. And also, when we do the Spain Spanish. And also, when we do the Spain Spanish. And also, when we do the kind of responding in Spanish, we're kind of responding in Spanish, we're kind of responding in Spanish, we're also smart enough to know that some also smart enough to know that some also smart enough to know that some words are better to be left in English, words are better to be left in English, words are better to be left in English, like the mid-century type of word. It like the mid-century type of word. It like the mid-century type of word. It didn't translate it directly to Spanish didn't translate it directly to Spanish didn't translate it directly to Spanish cuz it knew that the term was commonly cuz it knew that the term was commonly cuz it knew that the term was commonly known in English. Sorry, in Spanish. So, known in English. Sorry, in Spanish. So, known in English. Sorry, in Spanish. So, this is one example of how users are this is one example of how users are this is one example of how users are using search live for like info-seeking using search live for like info-seeking using search live for like info-seeking questions. Now, let me show a very questions. Now, let me show a very questions. Now, let me show a very different demo on how we do this for different demo on how we do this for different demo on how we do this for conversational voice agents. >> Hi, I've blown a tire and had to pull >> Hi, I've blown a tire and had to pull over to the side of the road. Can you over to the side of the road. Can you over to the side of the road. Can you help?

help? help? >> Hello, you've reached Auto Transform >> Hello, you've reached Auto Transform >> Hello, you've reached Auto Transform Insurance Roadside Assistance. My name Insurance Roadside Assistance. My name Insurance Roadside Assistance. My name is Ace. I understand this is stressful, is Ace. I understand this is stressful, is Ace. I understand this is stressful, but we'll take care of everything. To but we'll take care of everything. To but we'll take care of everything. To get started, could I please have your get started, could I please have your get started, could I please have your name and your policy number? name and your policy number? name and your policy number? >> Uh my name is Richard Jenkins. I I don't >> Uh my name is Richard Jenkins. I I don't >> Uh my name is Richard Jenkins. I I don't actually have my policy number to hand. actually have my policy number to hand. actually have my policy number to hand. >> No worries, Richard. I can look you up >> No worries, Richard. I can look you up >> No worries, Richard. I can look you up with your registration plate and with your registration plate and with your registration plate and postcode instead. Could you provide postcode instead. Could you provide postcode instead. Could you provide those for me, please? those for me, please? those for me, please? >> Whoa, that was a bit close. Um yeah, my >> Whoa, that was a bit close. Um yeah, my >> Whoa, that was a bit close. Um yeah, my registration plate is BD21 registration plate is BD21 registration plate is BD21 XYA XYA XYA and uh my postcode is SN48ZX. >> Policy details BD21 >> Policy details BD21 XYA. Thank you. For your safety, please XYA. Thank you. For your safety, please XYA. Thank you. For your safety, please stay clear of the road. I've found your stay clear of the road. I've found your stay clear of the road. I've found your details and I see you're in a blue Mini details and I see you're in a blue Mini details and I see you're in a blue Mini Cooper F-Series. Is that the vehicle Cooper F-Series. Is that the vehicle Cooper F-Series. Is that the vehicle you're in? you're in? you're in? >> Yeah, yeah, that that's the vehicle. >> Got it.

>> Got it. >> Okay, so I'll pause it here, but as you >> Okay, so I'll pause it here, but as you >> Okay, so I'll pause it here, but as you can see, very different things are can see, very different things are can see, very different things are happening under the hood with the same happening under the hood with the same happening under the hood with the same model. You know, it's more about model. You know, it's more about model. You know, it's more about alphanumeric accuracy for complex kind alphanumeric accuracy for complex kind alphanumeric accuracy for complex kind of like postcode numbers or addresses. of like postcode numbers or addresses. of like postcode numbers or addresses. There's like this feature that we have There's like this feature that we have There's like this feature that we have called proactive audio, which is the called proactive audio, which is the called proactive audio, which is the idea that the LLM knows when or when not idea that the LLM knows when or when not idea that the LLM knows when or when not to respond to an external input. So, for to respond to an external input. So, for to respond to an external input. So, for example, if you're in a conversation and example, if you're in a conversation and example, if you're in a conversation and someone else is talking or there's someone else is talking or there's someone else is talking or there's background noise happening in the back, background noise happening in the back, background noise happening in the back, the model knows to not stop or cut the model knows to not stop or cut the model knows to not stop or cut itself short because there's external itself short because there's external itself short because there's external noise happening. Because we realized noise happening. Because we realized noise happening. Because we realized that the majority of these conversations that the majority of these conversations that the majority of these conversations are not happening in an office room with are not happening in an office room with are not happening in an office room with like no noise, you know, in a sealed like no noise, you know, in a sealed like no noise, you know, in a sealed environment. They're happening on the environment. They're happening on the environment. They're happening on the go, on the train, while you're on a go, on the train, while you're on a go, on the train, while you're on a walk. Um and that's the type of walk. Um and that's the type of walk. Um and that's the type of experiences that we want to facilitate experiences that we want to facilitate experiences that we want to facilitate with the speech-to-speech model. with the speech-to-speech model. with the speech-to-speech model. Um lastly, um another thing we're really Um lastly, um another thing we're really Um lastly, um another thing we're really excited about is not only voice out, but excited about is not only voice out, but excited about is not only voice out, but also multimodal out. We believe that, also multimodal out. We believe that, also multimodal out. We believe that, you know, the true AGI conversational you know, the true AGI conversational you know, the true AGI conversational frontier will also require visual frontier will also require visual frontier will also require visual presence. So, we were excited to launch presence. So, we were excited to launch presence. So, we were excited to launch our first kind of pilot demo with City our first kind of pilot demo with City our first kind of pilot demo with City um in Cloud Next, which supports um in Cloud Next, which supports um in Cloud Next, which supports customized real-time avatars.

customized real-time avatars. customized real-time avatars. Um and you can kind of like Um and you can kind of like Um and you can kind of like personalize anything from a personalize anything from a personalize anything from a hyperrealistic human to a cartoon hyperrealistic human to a cartoon hyperrealistic human to a cartoon appearance and everything in the middle. appearance and everything in the middle. appearance and everything in the middle. And powered by the same speech-to-speech And powered by the same speech-to-speech And powered by the same speech-to-speech model we've been showing, it allows to model we've been showing, it allows to model we've been showing, it allows to have, you know, low-latency, have, you know, low-latency, have, you know, low-latency, multilingual lip-syncing, and a really multilingual lip-syncing, and a really multilingual lip-syncing, and a really kind of like uh ongoing fluid kind of like uh ongoing fluid kind of like uh ongoing fluid conversation that has visual presence. conversation that has visual presence. conversation that has visual presence. So, let me show you a demo that also So, let me show you a demo that also So, let me show you a demo that also brings our Venn diagram together and brings our Venn diagram together and brings our Venn diagram together and what we're excited about. what we're excited about. what we're excited about. >> All this is reminding me of my >> All this is reminding me of my >> All this is reminding me of my daughter's college fund. How are we daughter's college fund. How are we daughter's college fund. How are we tracking on that? >> You're tracking well, Jackson. I've also >> You're tracking well, Jackson. I've also identified a new opportunity that may identified a new opportunity that may identified a new opportunity that may get you there even sooner. get you there even sooner. get you there even sooner. Oh, and I can see Lisa just joined you. Oh, and I can see Lisa just joined you. Oh, and I can see Lisa just joined you. Hi, Lisa. She must be so excited about Hi, Lisa. She must be so excited about Hi, Lisa. She must be so excited about her college acceptance. Congratulations. her college acceptance. Congratulations. her college acceptance. Congratulations. It's wonderful to see your savings goals It's wonderful to see your savings goals It's wonderful to see your savings goals coming to life. coming to life. coming to life. >> Excelente. Las cosas han estado tan >> Excelente. Las cosas han estado tan >> Excelente. Las cosas han estado tan inestables últimamente. >> Sí. Los servicios de tecnología y >> Sí. Los servicios de tecnología y comunicaciones han mostrado un desempeño comunicaciones han mostrado un desempeño comunicaciones han mostrado un desempeño sólido en lo que va del año.

sólido en lo que va del año. sólido en lo que va del año. >> That's good news. >> That's good news. >> That's good news. >> I always joke that the user's audio in >> I always joke that the user's audio in >> I always joke that the user's audio in Spanish is worse than the audio model Spanish is worse than the audio model Spanish is worse than the audio model speaking back, but um this is kind of speaking back, but um this is kind of speaking back, but um this is kind of just to show how like our Venn diagram just to show how like our Venn diagram just to show how like our Venn diagram of combining, you know, multimodality in of combining, you know, multimodality in of combining, you know, multimodality in and out, you know, tool calling to like and out, you know, tool calling to like and out, you know, tool calling to like pull up the relevant examples from the pull up the relevant examples from the pull up the relevant examples from the user, and also conversational fluidity user, and also conversational fluidity user, and also conversational fluidity with ITNN are starting slowly to come with ITNN are starting slowly to come with ITNN are starting slowly to come together in these types of demos that together in these types of demos that together in these types of demos that we're excited to keep pushing the we're excited to keep pushing the we're excited to keep pushing the frontier of. frontier of. frontier of. Um so with this parting thought, I guess Um so with this parting thought, I guess Um so with this parting thought, I guess last last last kind of thought that we have for you is kind of thought that we have for you is kind of thought that we have for you is that we believe that AGI will not be that we believe that AGI will not be that we believe that AGI will not be typed, that it will be spoken. Um and typed, that it will be spoken. Um and typed, that it will be spoken. Um and for it to be spoken, there's a lot of for it to be spoken, there's a lot of for it to be spoken, there's a lot of things that need to work together in a things that need to work together in a things that need to work together in a single promptable, versatile model that single promptable, versatile model that single promptable, versatile model that allows a user to switch between all the allows a user to switch between all the allows a user to switch between all the sorts of conversation modes that we're sorts of conversation modes that we're sorts of conversation modes that we're looking at, right? From translation to looking at, right? From translation to looking at, right? From translation to taking action to brainstorming to taking action to brainstorming to taking action to brainstorming to rambling, and we truly believe in the rambling, and we truly believe in the rambling, and we truly believe in the power of these speech-to-speech models power of these speech-to-speech models power of these speech-to-speech models to achieve that like seamless switching. to achieve that like seamless switching. to achieve that like seamless switching. Um so we're excited to push the frontier Um so we're excited to push the frontier Um so we're excited to push the frontier on that. So if you're excited or want to on that. So if you're excited or want to on that. So if you're excited or want to learn more, please come talk to us, and learn more, please come talk to us, and learn more, please come talk to us, and thank you so much for coming.