Voice Agents Can Just Do Things — Charlie Guo, OpenAI

So, my name is Charlie uh and I work on So, my name is Charlie uh and I work on the developer experience team at OpenAI. the developer experience team at OpenAI. the developer experience team at OpenAI. And part of my job is talking to And part of my job is talking to And part of my job is talking to developers to understand and see, you developers to understand and see, you developers to understand and see, you know, what and how they're building with know, what and how they're building with know, what and how they're building with our models. um whether that's text, our models. um whether that's text, our models. um whether that's text, image or audio. And lately I've been image or audio. And lately I've been image or audio. And lately I've been thinking about a misconception that I thinking about a misconception that I thinking about a misconception that I have seen or maybe it's just a have seen or maybe it's just a have seen or maybe it's just a misunderstanding misunderstanding misunderstanding and it's the idea that voice agents and it's the idea that voice agents and it's the idea that voice agents have to talk back. >> And to some of you that might sound, you >> And to some of you that might sound, you know, absurd. It's a voice agent. What know, absurd. It's a voice agent. What know, absurd. It's a voice agent. What do you mean it's not supposed to talk? do you mean it's not supposed to talk? do you mean it's not supposed to talk? Uh, but I think if there's one thing Uh, but I think if there's one thing Uh, but I think if there's one thing that you take away from this that you take away from this that you take away from this presentation, I would like it to be the presentation, I would like it to be the presentation, I would like it to be the idea that speech is not the only way idea that speech is not the only way idea that speech is not the only way that a voice model has to respond. And I think models are getting And I think models are getting intelligent enough and capable enough intelligent enough and capable enough intelligent enough and capable enough that they're starting to open up some uh that they're starting to open up some uh that they're starting to open up some uh new modes of design. I mean there's new modes of design. I mean there's new modes of design. I mean there's actually three kind of modes that that actually three kind of modes that that actually three kind of modes that that you know I kind of see emerging these you know I kind of see emerging these you know I kind of see emerging these days right uh speech to speech speech to days right uh speech to speech speech to days right uh speech to speech speech to action and event to speech and there's a action and event to speech and there's a action and event to speech and there's a couple of things I think worth pointing couple of things I think worth pointing couple of things I think worth pointing out about um these three categories. The out about um these three categories. The out about um these three categories. The first is that they're not new, right? I first is that they're not new, right? I first is that they're not new, right? I think as we've seen from from previous think as we've seen from from previous think as we've seen from from previous talks, even just today, um there's a talks, even just today, um there's a talks, even just today, um there's a long history of building these types of

long history of building these types of long history of building these types of systems in and around voice. Um if you systems in and around voice. Um if you systems in and around voice. Um if you squint, you could make the argument that squint, you could make the argument that squint, you could make the argument that the movie phone hotline where you called the movie phone hotline where you called the movie phone hotline where you called in to get showtimes was an example of a in to get showtimes was an example of a in to get showtimes was an example of a speech-to-pech system. Uh and I think speech-to-pech system. Uh and I think speech-to-pech system. Uh and I think you could pretty reasonably make the you could pretty reasonably make the you could pretty reasonably make the argument that you know GPS navigation in argument that you know GPS navigation in argument that you know GPS navigation in your car which has existed since I was a your car which has existed since I was a your car which has existed since I was a kid is an example of an event to speech kid is an example of an event to speech kid is an example of an event to speech system. So, it's not that they are brand system. So, it's not that they are brand system. So, it's not that they are brand new, but I think it is that we are able new, but I think it is that we are able new, but I think it is that we are able to do some some much more interesting to do some some much more interesting to do some some much more interesting things with them uh now that uh we're in things with them uh now that uh we're in things with them uh now that uh we're in this era, right? And I think, you know, this era, right? And I think, you know, this era, right? And I think, you know, the other thing I would mention here is the other thing I would mention here is the other thing I would mention here is that um they're they're remixable, that um they're they're remixable, that um they're they're remixable, right? They're not meant to be mutually right? They're not meant to be mutually right? They're not meant to be mutually exclusive. Um and I think as we'll see exclusive. Um and I think as we'll see exclusive. Um and I think as we'll see in a little bit, the best products uh in a little bit, the best products uh in a little bit, the best products uh exist in a way that combines all of exist in a way that combines all of exist in a way that combines all of these modes. So, speechto um everybody these modes. So, speechto um everybody these modes. So, speechto um everybody knows it. Hopefully, everybody loves it. knows it. Hopefully, everybody loves it. knows it. Hopefully, everybody loves it. the user talks uh and then the model the user talks uh and then the model the user talks uh and then the model talks back and I think there are a few talks back and I think there are a few talks back and I think there are a few examples that that I can give um for examples that that I can give um for examples that that I can give um for this type of use case right you've got this type of use case right you've got this type of use case right you've got things like live practice and coaching things like live practice and coaching things like live practice and coaching especially around language learning especially around language learning especially around language learning right I think the ability to um hear a right I think the ability to um hear a right I think the ability to um hear a lot of emphasis or emotion and give lot of emphasis or emotion and give lot of emphasis or emotion and give people that feedback is really powerful people that feedback is really powerful people that feedback is really powerful um I think you can have you know what I um I think you can have you know what I um I think you can have you know what I am sort of cheekily calling concierge am sort of cheekily calling concierge am sort of cheekily calling concierge experiences which I think is just experiences which I think is just experiences which I think is just another way to say customer support another way to say customer support another way to say customer support plus+ Um the first voice tutorial that plus+ Um the first voice tutorial that plus+ Um the first voice tutorial that most people you know try to build when most people you know try to build when most people you know try to build when they have access to this technology is they have access to this technology is they have access to this technology is some sort of customer support chatbot some sort of customer support chatbot some sort of customer support chatbot for very good reasons. But I think with for very good reasons. But I think with for very good reasons. But I think with you know when we add a richness and a

you know when we add a richness and a you know when we add a richness and a depth to the voice models as has been depth to the voice models as has been depth to the voice models as has been happening in recent months in recent happening in recent months in recent happening in recent months in recent years um we can build something that is years um we can build something that is years um we can build something that is like much more enjoyable to use than like much more enjoyable to use than like much more enjoyable to use than like talking your way through a phone like talking your way through a phone like talking your way through a phone tree. Um, and so, you know, I have the tree. Um, and so, you know, I have the tree. Um, and so, you know, I have the hope that like soon, if not like, you hope that like soon, if not like, you hope that like soon, if not like, you know, now, we're capable of building know, now, we're capable of building know, now, we're capable of building support experiences with agents that support experiences with agents that support experiences with agents that actually feel much more enjoyable to actually feel much more enjoyable to actually feel much more enjoyable to talk to than like arguably like the talk to than like arguably like the talk to than like arguably like the median human support agent. median human support agent. median human support agent. Um, and as we just saw if you hear the Um, and as we just saw if you hear the Um, and as we just saw if you hear the last talk, uh, live translation, right? last talk, uh, live translation, right? last talk, uh, live translation, right? The models have gotten good enough and The models have gotten good enough and The models have gotten good enough and fast enough that we can just dynamically fast enough that we can just dynamically fast enough that we can just dynamically translate content on the fly, um, with translate content on the fly, um, with translate content on the fly, um, with like little to no latency. It wouldn't like little to no latency. It wouldn't like little to no latency. It wouldn't shock me if at next year's keynote um shock me if at next year's keynote um shock me if at next year's keynote um you know they live streamed it from the you know they live streamed it from the you know they live streamed it from the main stage but also uh dubbed it in real main stage but also uh dubbed it in real main stage but also uh dubbed it in real time across multiple languages. The second category is speech to action. The second category is speech to action. Uh these are talks and the model uses Uh these are talks and the model uses Uh these are talks and the model uses tools and I think this is one of the tools and I think this is one of the tools and I think this is one of the most underexplored areas that we have. most underexplored areas that we have. most underexplored areas that we have. Um I actually almost titled this talk uh Um I actually almost titled this talk uh Um I actually almost titled this talk uh voice is the next capability overhang voice is the next capability overhang voice is the next capability overhang because I think there is just a vast because I think there is just a vast because I think there is just a vast vast amount of stuff um that we could be vast amount of stuff um that we could be vast amount of stuff um that we could be doing in this category that we are not doing in this category that we are not doing in this category that we are not currently doing. Uh for example um currently doing. Uh for example um currently doing. Uh for example um there's a broad spectrum I don't have there's a broad spectrum I don't have there's a broad spectrum I don't have you know there's way too many examples you know there's way too many examples you know there's way too many examples even fit on this slide but three even fit on this slide but three even fit on this slide but three categories that that I find particularly categories that that I find particularly categories that that I find particularly interesting. Uh first is form filling interesting. Uh first is form filling interesting. Uh first is form filling right um so much of the internet is just right um so much of the internet is just right um so much of the internet is just filling out forms. Um, and there is, you filling out forms. Um, and there is, you filling out forms. Um, and there is, you know, today no reason why you shouldn't know, today no reason why you shouldn't know, today no reason why you shouldn't be able to just talk. You know, I would be able to just talk. You know, I would be able to just talk. You know, I would love it if instead of spending an hour

love it if instead of spending an hour love it if instead of spending an hour filling out a government document, I filling out a government document, I filling out a government document, I could just talk for five minutes and it could just talk for five minutes and it could just talk for five minutes and it would get 90% of it for me and I would would get 90% of it for me and I would would get 90% of it for me and I would do a quick check, you know, just to make do a quick check, you know, just to make do a quick check, you know, just to make sure that everything looked good, right? sure that everything looked good, right? sure that everything looked good, right? That is a vastly superior experience That is a vastly superior experience That is a vastly superior experience than like having to type in every single than like having to type in every single than like having to type in every single name and address that I've lived in the name and address that I've lived in the name and address that I've lived in the last 5 years and, you know, all of my last 5 years and, you know, all of my last 5 years and, you know, all of my previous identities. Um and so I think I previous identities. Um and so I think I previous identities. Um and so I think I think that one is though it may seem think that one is though it may seem think that one is though it may seem boring you know affects a a significant boring you know affects a a significant boring you know affects a a significant GDP of the internet right the next GDP of the internet right the next GDP of the internet right the next category is creative tools uh where I am category is creative tools uh where I am category is creative tools uh where I am privileged enough that I can speak the privileged enough that I can speak the privileged enough that I can speak the language of software and so I can tell language of software and so I can tell language of software and so I can tell codeex you know here's exactly what I codeex you know here's exactly what I codeex you know here's exactly what I want you to build and I can articulate want you to build and I can articulate want you to build and I can articulate it in a way that um I get much more it in a way that um I get much more it in a way that um I get much more leverage than sort of just like cludily leverage than sort of just like cludily leverage than sort of just like cludily trying to iterate one thing at a time trying to iterate one thing at a time trying to iterate one thing at a time but I can't do that when it comes to you but I can't do that when it comes to you but I can't do that when it comes to you know using making music or painting um know using making music or painting um know using making music or painting um and so if I don't have the ability to and so if I don't have the ability to and so if I don't have the ability to articulate um the exact aesthetic that articulate um the exact aesthetic that articulate um the exact aesthetic that I'm looking for. Um and if I don't know I'm looking for. Um and if I don't know I'm looking for. Um and if I don't know how to use Photoshop or Ableton, I'm how to use Photoshop or Ableton, I'm how to use Photoshop or Ableton, I'm left in this state where, you know, my left in this state where, you know, my left in this state where, you know, my my taste exceeds my capability. Um and my taste exceeds my capability. Um and my taste exceeds my capability. Um and so I'm really looking forward to so I'm really looking forward to so I'm really looking forward to integrating voice into creative tools so integrating voice into creative tools so integrating voice into creative tools so that I can just sort of cludgy go along that I can just sort of cludgy go along that I can just sort of cludgy go along and, you know, vibe create, vibe and, you know, vibe create, vibe and, you know, vibe create, vibe compose, vibe paint, um and make compose, vibe paint, um and make compose, vibe paint, um and make something that that's really beautiful something that that's really beautiful something that that's really beautiful to me. And I think the generalizable um to me. And I think the generalizable um to me. And I think the generalizable um category here, right, then just starts category here, right, then just starts category here, right, then just starts to become computer use. And we've to become computer use. And we've to become computer use. And we've already seen some companies start to do already seen some companies start to do already seen some companies start to do this. Um, you know, it raises the this. Um, you know, it raises the this. Um, you know, it raises the question of like, look, if the models question of like, look, if the models question of like, look, if the models are just getting good enough to do are just getting good enough to do are just getting good enough to do everything on a computer that a human everything on a computer that a human everything on a computer that a human can do, like why am I talking to an app?

can do, like why am I talking to an app? can do, like why am I talking to an app? Why am I talking to a terminal? Why am I Why am I talking to a terminal? Why am I Why am I talking to a terminal? Why am I not just talking to the entire computer? not just talking to the entire computer? not just talking to the entire computer? Uh, and so I think that's sort of a Uh, and so I think that's sort of a Uh, and so I think that's sort of a really interesting uh way to start really interesting uh way to start really interesting uh way to start exploring. But if you're a developer exploring. But if you're a developer exploring. But if you're a developer today, right? Whoops. If you're a today, right? Whoops. If you're a today, right? Whoops. If you're a developer today, um, what does that mean developer today, um, what does that mean developer today, um, what does that mean for building your own software, right? for building your own software, right? for building your own software, right? And I think it is like much easier than And I think it is like much easier than And I think it is like much easier than you think to start adding audio as an you think to start adding audio as an you think to start adding audio as an intelligence layer to the intelligence intelligence layer to the intelligence intelligence layer to the intelligence layer to the apps that you already have. layer to the apps that you already have. layer to the apps that you already have. Um, if you're building a modern web Um, if you're building a modern web Um, if you're building a modern web application, you already expose so much application, you already expose so much application, you already expose so much of it as like action as nouns and verbs, of it as like action as nouns and verbs, of it as like action as nouns and verbs, right? And if you think about all the right? And if you think about all the right? And if you think about all the verbs that you have, you have uh API verbs that you have, you have uh API verbs that you have, you have uh API endpoints, you have, you know, React endpoints, you have, you know, React endpoints, you have, you know, React hooks. Each of those things can like hooks. Each of those things can like hooks. Each of those things can like pretty relatively easily be converted pretty relatively easily be converted pretty relatively easily be converted into a tool that you expose to a model into a tool that you expose to a model into a tool that you expose to a model and then you can give the user the and then you can give the user the and then you can give the user the ability to just drive your existing ability to just drive your existing ability to just drive your existing software um with their voice, right? And software um with their voice, right? And software um with their voice, right? And yes, you still need guardrails, you yes, you still need guardrails, you yes, you still need guardrails, you still need safety checks. Like many of still need safety checks. Like many of still need safety checks. Like many of the talks today are going to talk about the talks today are going to talk about the talks today are going to talk about securing and you know productizing this, securing and you know productizing this, securing and you know productizing this, but um for this I just want you to think but um for this I just want you to think but um for this I just want you to think about you know what would it mean to about you know what would it mean to about you know what would it mean to take your existing software and just take your existing software and just take your existing software and just talk to it. talk to it. talk to it. Um and to go back to that misconception, Um and to go back to that misconception, Um and to go back to that misconception, right? I think there are a lot of um you right? I think there are a lot of um you right? I think there are a lot of um you know like if you're talking to the know like if you're talking to the know like if you're talking to the software maybe it can talk back but software maybe it can talk back but software maybe it can talk back but we've been developing other ways of we've been developing other ways of we've been developing other ways of communicating with the user for decades communicating with the user for decades communicating with the user for decades right we know these things we know we right we know these things we know we right we know these things we know we can show notifications and popups we can can show notifications and popups we can can show notifications and popups we can change state like the color of a button change state like the color of a button change state like the color of a button or a drop shadow we can highlight text or a drop shadow we can highlight text or a drop shadow we can highlight text um if you've used computer use in the um if you've used computer use in the um if you've used computer use in the codeex app you know there's this amazing codeex app you know there's this amazing codeex app you know there's this amazing like little ghost cursor animation that like little ghost cursor animation that like little ghost cursor animation that goes around and clicks things for you so goes around and clicks things for you so goes around and clicks things for you so we don't have to use words to actually we don't have to use words to actually we don't have to use words to actually tell the user what is happening on tell the user what is happening on tell the user what is happening on screen with their software

Uh and then the last bucket here is Uh and then the last bucket here is event to speech, right? Um the model event to speech, right? Um the model event to speech, right? Um the model receives an event and talks to the user. receives an event and talks to the user. receives an event and talks to the user. Um and sort of the counterpoint from Um and sort of the counterpoint from Um and sort of the counterpoint from speech to action. I think this one is speech to action. I think this one is speech to action. I think this one is still very very exploratory, right? Um still very very exploratory, right? Um still very very exploratory, right? Um you know, if you saw Quinn's talk, I you know, if you saw Quinn's talk, I you know, if you saw Quinn's talk, I think there's a lot of uh space here of think there's a lot of uh space here of think there's a lot of uh space here of like things we can do. Um and to me, we like things we can do. Um and to me, we like things we can do. Um and to me, we haven't quite seen what AI native really haven't quite seen what AI native really haven't quite seen what AI native really looks like in this vein yet. But um of looks like in this vein yet. But um of looks like in this vein yet. But um of the things that I've seen, I think the things that I've seen, I think the things that I've seen, I think there's a couple of through lines that I there's a couple of through lines that I there's a couple of through lines that I tend to notice, right? The first is tend to notice, right? The first is tend to notice, right? The first is hands-free or screen-free experiences. hands-free or screen-free experiences. hands-free or screen-free experiences. There might be times where uh I need to There might be times where uh I need to There might be times where uh I need to interact with software, interact with interact with software, interact with interact with software, interact with objects and I can't use my hands or more objects and I can't use my hands or more objects and I can't use my hands or more importantly my attention is diverted importantly my attention is diverted importantly my attention is diverted elsewhere. Um that might be something elsewhere. Um that might be something elsewhere. Um that might be something you know like uh a recipe app. Maybe I'm you know like uh a recipe app. Maybe I'm you know like uh a recipe app. Maybe I'm cooking and I need to just say like cooking and I need to just say like cooking and I need to just say like what's going on and and have something what's going on and and have something what's going on and and have something else have something happen. Um the other else have something happen. Um the other else have something happen. Um the other category is proactive outreach, right? category is proactive outreach, right? category is proactive outreach, right? Where you the model needs to be able to Where you the model needs to be able to Where you the model needs to be able to tell you something or get your attention tell you something or get your attention tell you something or get your attention in a way um that you might not be in a way um that you might not be in a way um that you might not be looking at, right? I think every looking at, right? I think every looking at, right? I think every developer um has uh an endless amount of developer um has uh an endless amount of developer um has uh an endless amount of notifications and events happening in notifications and events happening in notifications and events happening in their software. But um no developer in their software. But um no developer in their software. But um no developer in their right mind would sort of say I their right mind would sort of say I their right mind would sort of say I should show all of these logs. Nor you should show all of these logs. Nor you should show all of these logs. Nor you know would they say I should speak all know would they say I should speak all know would they say I should speak all of these logs. But we can start to of these logs. But we can start to of these logs. But we can start to conceive of voice as this like upper conceive of voice as this like upper conceive of voice as this like upper level in this escalatory path of like level in this escalatory path of like level in this escalatory path of like okay maybe you animate something and okay maybe you animate something and okay maybe you animate something and then maybe you pop something up and then then maybe you pop something up and then then maybe you pop something up and then if that doesn't work maybe you talk to if that doesn't work maybe you talk to if that doesn't work maybe you talk to the user to get their attention.

the user to get their attention. the user to get their attention. And underlying both of these categories And underlying both of these categories And underlying both of these categories and I think this this you know this and I think this this you know this and I think this this you know this whole presentation is this broader theme whole presentation is this broader theme whole presentation is this broader theme of accessibility. Um on a personal of accessibility. Um on a personal of accessibility. Um on a personal personal note, I know like multiple personal note, I know like multiple personal note, I know like multiple developers who um over the course of developers who um over the course of developers who um over the course of their careers lost mobility in their their careers lost mobility in their their careers lost mobility in their hands, lost dexterity in their fingers hands, lost dexterity in their fingers hands, lost dexterity in their fingers and for many of them, they thought their and for many of them, they thought their and for many of them, they thought their career as a programmer was more or less career as a programmer was more or less career as a programmer was more or less over. Um and then came large language over. Um and then came large language over. Um and then came large language models, right? Then came coding agents models, right? Then came coding agents models, right? Then came coding agents and voice agents and now they generate and voice agents and now they generate and voice agents and now they generate orders of magnitude more code than they orders of magnitude more code than they orders of magnitude more code than they like previously did um you know on a like previously did um you know on a like previously did um you know on a given given day or month. Um, and so I given given day or month. Um, and so I given given day or month. Um, and so I think there's there's a lot that we can think there's there's a lot that we can think there's there's a lot that we can unlock here uh for the broader world as unlock here uh for the broader world as unlock here uh for the broader world as well. well. well. Um, to go back to like I said, you know, Um, to go back to like I said, you know, Um, to go back to like I said, you know, I think like when it comes to these I think like when it comes to these I think like when it comes to these three modalities, you can mix and match three modalities, you can mix and match three modalities, you can mix and match them and we already have some, you know, them and we already have some, you know, them and we already have some, you know, rudimentary ways that we're seeing this. rudimentary ways that we're seeing this. rudimentary ways that we're seeing this. I think there's things like, you know, I think there's things like, you know, I think there's things like, you know, all of these pieces for in-car all of these pieces for in-car all of these pieces for in-car assistants exist, though nothing has assistants exist, though nothing has assistants exist, though nothing has quite like combined them into this quite like combined them into this quite like combined them into this seamless way. You can talk to like the seamless way. You can talk to like the seamless way. You can talk to like the CarPlay dashboard. Um, you can tell it, CarPlay dashboard. Um, you can tell it, CarPlay dashboard. Um, you can tell it, "Hey, go play some Spotify music for "Hey, go play some Spotify music for "Hey, go play some Spotify music for me." Um, and then it can come back and me." Um, and then it can come back and me." Um, and then it can come back and tell you, Google Maps can come back and tell you, Google Maps can come back and tell you, Google Maps can come back and tell you, hey, like, you know, there's tell you, hey, like, you know, there's tell you, hey, like, you know, there's traffic on this route. We're going to traffic on this route. We're going to traffic on this route. We're going to reroute you. But like we can now start reroute you. But like we can now start reroute you. But like we can now start to think about what does it mean to to think about what does it mean to to think about what does it mean to combine that into like a single voice combine that into like a single voice combine that into like a single voice agent across multiple modes. [snorts] agent across multiple modes. [snorts] agent across multiple modes. [snorts] Um, similarly, you know, there's a lot Um, similarly, you know, there's a lot Um, similarly, you know, there's a lot of experimentation in the game space of experimentation in the game space of experimentation in the game space with multimodality. Um, you can think with multimodality. Um, you can think with multimodality. Um, you can think about a real life character where you're about a real life character where you're about a real life character where you're talking to it to, you know, mine talking to it to, you know, mine talking to it to, you know, mine information about the game, about the information about the game, about the information about the game, about the world. um you can talk to it to execute world. um you can talk to it to execute world. um you can talk to it to execute actions on your behalf and then it can actions on your behalf and then it can actions on your behalf and then it can react to like world events right that react to like world events right that react to like world events right that are happening and then give that are happening and then give that are happening and then give that information to you rather than just like information to you rather than just like information to you rather than just like a simple notification

um and I think the question you know um and I think the question you know behind the question here right is like I behind the question here right is like I behind the question here right is like I mentioned we've had all these things for mentioned we've had all these things for mentioned we've had all these things for a while people have been prototyping a while people have been prototyping a while people have been prototyping them for a while why focus on them now them for a while why focus on them now them for a while why focus on them now why think about building with them now why think about building with them now why think about building with them now um and I think that brings me to uh a um and I think that brings me to uh a um and I think that brings me to uh a little bit of context here right as as little bit of context here right as as little bit of context here right as as I'm hopefully most of you know I'm hopefully most of you know I'm hopefully most of you know traditionally voice agents are built in traditionally voice agents are built in traditionally voice agents are built in this chain Ed model, right? You uh talk, this chain Ed model, right? You uh talk, this chain Ed model, right? You uh talk, you transcribe, you send that to a you transcribe, you send that to a you transcribe, you send that to a language model. It calls tools. language model. It calls tools. language model. It calls tools. Hopefully, it doesn't take too long to Hopefully, it doesn't take too long to Hopefully, it doesn't take too long to respond. Um it then generates text respond. Um it then generates text respond. Um it then generates text output. You make that into audio and output. You make that into audio and output. You make that into audio and then you play that back to the user. then you play that back to the user. then you play that back to the user. And some time ago, um OpenAI, you know, And some time ago, um OpenAI, you know, And some time ago, um OpenAI, you know, decided on a different approach, right? decided on a different approach, right? decided on a different approach, right? The real-time model family does not do The real-time model family does not do The real-time model family does not do any transcription behind the scenes. It any transcription behind the scenes. It any transcription behind the scenes. It is trained on native audio as tokens. is trained on native audio as tokens. is trained on native audio as tokens. So, you send audio in and you get audio So, you send audio in and you get audio So, you send audio in and you get audio back out. And the industry I think in back out. And the industry I think in back out. And the industry I think in general has been, you know, trending general has been, you know, trending general has been, you know, trending more in this direction. Um, and not just more in this direction. Um, and not just more in this direction. Um, and not just making it native audio, but even just making it native audio, but even just making it native audio, but even just letting go of the turnbased abstraction letting go of the turnbased abstraction letting go of the turnbased abstraction that we've had, right? Um, and so making that we've had, right? Um, and so making that we've had, right? Um, and so making it that it's just continuous streaming it that it's just continuous streaming it that it's just continuous streaming audio in and out. audio in and out. audio in and out. And the reason that opening I did this And the reason that opening I did this And the reason that opening I did this was, you know, turns out there's a lot was, you know, turns out there's a lot was, you know, turns out there's a lot of stuff that you lose when you of stuff that you lose when you of stuff that you lose when you transcribe speech and when you transcribe speech and when you transcribe speech and when you transcribe audio, right? Um there's the transcribe audio, right? Um there's the transcribe audio, right? Um there's the old saying that when humans communicate old saying that when humans communicate old saying that when humans communicate face to face, 55% of the information is face to face, 55% of the information is face to face, 55% of the information is in body language, another 38% is in your in body language, another 38% is in your in body language, another 38% is in your tone of voice and the last like 7% is tone of voice and the last like 7% is tone of voice and the last like 7% is the actual words you are saying. Um and the actual words you are saying. Um and the actual words you are saying. Um and so when you transcribe, you lose tone so when you transcribe, you lose tone so when you transcribe, you lose tone and cadence and emotional uh you know and cadence and emotional uh you know and cadence and emotional uh you know impact, you lose like whether they're impact, you lose like whether they're impact, you lose like whether they're trying to interrupt you, you lose trying to interrupt you, you lose trying to interrupt you, you lose background noise, all of this stuff background noise, all of this stuff background noise, all of this stuff which is really important context for which is really important context for which is really important context for the model to understand.

the model to understand. the model to understand. Um a much more quantitative reason to do Um a much more quantitative reason to do Um a much more quantitative reason to do it is that you know the first two uh it is that you know the first two uh it is that you know the first two uh voice modes in chat GBT were built with voice modes in chat GBT were built with voice modes in chat GBT were built with this chained approach. Um and as you can this chained approach. Um and as you can this chained approach. Um and as you can see had you know significantly higher see had you know significantly higher see had you know significantly higher latency than using uh the native latency than using uh the native latency than using uh the native approach with advanced voice mode. approach with advanced voice mode. approach with advanced voice mode. And that brings me to GPT realtime 2. Um And that brings me to GPT realtime 2. Um And that brings me to GPT realtime 2. Um and this is going to be the one part of and this is going to be the one part of and this is going to be the one part of the talk where you know I make my the talk where you know I make my the talk where you know I make my shameless plug. Um real time 2 is the shameless plug. Um real time 2 is the shameless plug. Um real time 2 is the the latest model in the real time the latest model in the real time the latest model in the real time family. We released it a couple of family. We released it a couple of family. We released it a couple of months ago. Um, and the the really cool months ago. Um, and the the really cool months ago. Um, and the the really cool thing about this model is that it brings thing about this model is that it brings thing about this model is that it brings reasoning to the audio medium. Um, and reasoning to the audio medium. Um, and reasoning to the audio medium. Um, and so much like our text models, it can now so much like our text models, it can now so much like our text models, it can now think before it speaks. Um, I'm sure think before it speaks. Um, I'm sure think before it speaks. Um, I'm sure many of us have seen some demos of voice many of us have seen some demos of voice many of us have seen some demos of voice models saying things that are a little models saying things that are a little models saying things that are a little bit less than intelligent. Um, and so bit less than intelligent. Um, and so bit less than intelligent. Um, and so you can now, you know, try to ensure you can now, you know, try to ensure you can now, you know, try to ensure that you give it more reasoning budget that you give it more reasoning budget that you give it more reasoning budget uh to come up with a good answer. Part uh to come up with a good answer. Part uh to come up with a good answer. Part of why that's also useful is that we of why that's also useful is that we of why that's also useful is that we introduced tool calling a little while introduced tool calling a little while introduced tool calling a little while ago. And so the model in addition to ago. And so the model in addition to ago. And so the model in addition to thinking it can also delegate parallel thinking it can also delegate parallel thinking it can also delegate parallel tool calls. Um you can start to bring tool calls. Um you can start to bring tool calls. Um you can start to bring these together. Um though of course that these together. Um though of course that these together. Um though of course that adds latency, right? And you know that's adds latency, right? And you know that's adds latency, right? And you know that's why we also added preamles. Um preamles why we also added preamles. Um preamles why we also added preamles. Um preamles are a way that you can prompt the model are a way that you can prompt the model are a way that you can prompt the model to uh give the user a heads up if it's to uh give the user a heads up if it's to uh give the user a heads up if it's going to be thinking or if it's going to going to be thinking or if it's going to going to be thinking or if it's going to be calling tools. Um you know if you be calling tools. Um you know if you be calling tools. Um you know if you think about the scenario of a travel think about the scenario of a travel think about the scenario of a travel agent, right? If I called the travel agent, right? If I called the travel agent, right? If I called the travel agent on the phone, uh you would want agent on the phone, uh you would want agent on the phone, uh you would want the travel agent to say to say, "Hey, the travel agent to say to say, "Hey, the travel agent to say to say, "Hey, like I'm going to go check flight like I'm going to go check flight like I'm going to go check flight prices, right? give me a couple seconds prices, right? give me a couple seconds prices, right? give me a couple seconds to do that. Um, and now with an AI to do that. Um, and now with an AI to do that. Um, and now with an AI travel agent, um, and preamles, you can travel agent, um, and preamles, you can travel agent, um, and preamles, you can actually have it communicate that to the

actually have it communicate that to the actually have it communicate that to the user while it's performing actions in user while it's performing actions in user while it's performing actions in the background. Uh, there's a few other the background. Uh, there's a few other the background. Uh, there's a few other things here, right? It's got longer things here, right? It's got longer things here, right? It's got longer context, better domain understanding, context, better domain understanding, context, better domain understanding, um, more natural voices, and it's much um, more natural voices, and it's much um, more natural voices, and it's much more steerable. Um, there's some really more steerable. Um, there's some really more steerable. Um, there's some really cool features, uh, that it can do when cool features, uh, that it can do when cool features, uh, that it can do when it comes to like wake words and just it comes to like wake words and just it comes to like wake words and just waiting for you to to tell it. You you waiting for you to to tell it. You you waiting for you to to tell it. You you can give it a name. Uh, you can say can give it a name. Uh, you can say can give it a name. Uh, you can say like, you know, hey, Marin, do you want like, you know, hey, Marin, do you want like, you know, hey, Marin, do you want to say hi to the room? Um, and if you've to say hi to the room? Um, and if you've to say hi to the room? Um, and if you've like prompted that into the model, then like prompted that into the model, then like prompted that into the model, then it'll, you know, go ahead and and it'll, you know, go ahead and and it'll, you know, go ahead and and respond to you, right? Uh, and of respond to you, right? Uh, and of respond to you, right? Uh, and of course, you know, uh, obligatory course, you know, uh, obligatory course, you know, uh, obligatory benchmark slide, uh, it does pretty well benchmark slide, uh, it does pretty well benchmark slide, uh, it does pretty well on the the latest audio benchmarks, too. on the the latest audio benchmarks, too. on the the latest audio benchmarks, too. So, TLDDR, uh, is a pretty good model. So, TLDDR, uh, is a pretty good model. So, TLDDR, uh, is a pretty good model. Um, Um, Um, but I think the the kind of final thing but I think the the kind of final thing but I think the the kind of final thing that you know I want to leave you with that you know I want to leave you with that you know I want to leave you with here is um when building voice agents uh here is um when building voice agents uh here is um when building voice agents uh not to not to not to start with the question of like what start with the question of like what start with the question of like what kind of voice agent am I trying to kind of voice agent am I trying to kind of voice agent am I trying to build, right? I think the thing I want build, right? I think the thing I want build, right? I think the thing I want to leave you with is start with the to leave you with is start with the to leave you with is start with the question of like what is the role of question of like what is the role of question of like what is the role of voice and audio in this interaction? Um voice and audio in this interaction? Um voice and audio in this interaction? Um and then how do I move forward from and then how do I move forward from and then how do I move forward from there, right? Right? And often when I there, right? Right? And often when I there, right? Right? And often when I ask that question, it leads to a bunch ask that question, it leads to a bunch ask that question, it leads to a bunch more questions after that. Things like more questions after that. Things like more questions after that. Things like what can the model perceive? What what can the model perceive? What what can the model perceive? What context does it have? Right? Um what context does it have? Right? Um what context does it have? Right? Um what tools are available to it and which of tools are available to it and which of tools are available to it and which of those tools should it be, you know, uh those tools should it be, you know, uh those tools should it be, you know, uh executing safely and correctly? Um executing safely and correctly? Um executing safely and correctly? Um should it communicate now? Should it should it communicate now? Should it should it communicate now? Should it wait? Uh you know, how should it wait? Uh you know, how should it wait? Uh you know, how should it communicate? Should it be sending visual communicate? Should it be sending visual communicate? Should it be sending visual notifications or using audio? Um, and so

notifications or using audio? Um, and so notifications or using audio? Um, and so taken together, yeah, I hope everybody taken together, yeah, I hope everybody taken together, yeah, I hope everybody in here can can start to build some much in here can can start to build some much in here can can start to build some much richer experiences with voice. Um, richer experiences with voice. Um, richer experiences with voice. Um, because like others have said, uh, I do because like others have said, uh, I do because like others have said, uh, I do believe that AGI will be spoken, not believe that AGI will be spoken, not believe that AGI will be spoken, not typed. typed. typed. Thank you very much. I'll be at the Thank you very much. I'll be at the Thank you very much. I'll be at the OpenAI booth, uh, for any Q&A after. Um, OpenAI booth, uh, for any Q&A after. Um, OpenAI booth, uh, for any Q&A after. Um, yeah, have a good event.