Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

>> My name is Bo. I'm going to be here >> My name is Bo. I'm going to be here presenting real-time voice agents with presenting real-time voice agents with presenting real-time voice agents with Frontier Intelligence. Effectively, Frontier Intelligence. Effectively, Frontier Intelligence. Effectively, going to be talking a little bit about going to be talking a little bit about going to be talking a little bit about how we at Xnor.ai architected our voice how we at Xnor.ai architected our voice how we at Xnor.ai architected our voice agent harness to get real-time voice agent harness to get real-time voice agent harness to get real-time voice with the Frontier level of intelligence with the Frontier level of intelligence with the Frontier level of intelligence that we need. that we need. that we need. Okay. So, before I start, Okay. So, before I start, Okay. So, before I start, I think I wanted to kind of draw some I think I wanted to kind of draw some I think I wanted to kind of draw some parallels about parallels about parallels about why we decided to go with cascaded voice why we decided to go with cascaded voice why we decided to go with cascaded voice agents and especially kind of agents and especially kind of agents and especially kind of comparing that to self-driving cars comparing that to self-driving cars comparing that to self-driving cars which I was working in before. So, to which I was working in before. So, to which I was working in before. So, to me, cascaded voice agents makes a lot of me, cascaded voice agents makes a lot of me, cascaded voice agents makes a lot of sense when you view it in lens of kind sense when you view it in lens of kind sense when you view it in lens of kind of breaking it down into perception of breaking it down into perception of breaking it down into perception which is which is which is for self-driving cars, it's you know, for self-driving cars, it's you know, for self-driving cars, it's you know, the bounding boxes, the camera, the the bounding boxes, the camera, the the bounding boxes, the camera, the lidar. lidar. lidar. For voice, it's going to be the For voice, it's going to be the For voice, it's going to be the transcription. Basically, effectively transcription. Basically, effectively transcription. Basically, effectively turning these like signals from the real turning these like signals from the real turning these like signals from the real world into world into world into elements of data that the language model elements of data that the language model elements of data that the language model or whatever brain you're working on or whatever brain you're working on or whatever brain you're working on can process.

can process. can process. Second one is the planning step which is Second one is the planning step which is Second one is the planning step which is pretty straightforward. pretty straightforward. pretty straightforward. This is where the language model This is where the language model This is where the language model will take in the outputs from the will take in the outputs from the will take in the outputs from the perception stage and produce the outputs perception stage and produce the outputs perception stage and produce the outputs that you want to produce out back and that you want to produce out back and that you want to produce out back and out into the real world. out into the real world. out into the real world. And finally, there's the controls layer And finally, there's the controls layer And finally, there's the controls layer where where where on self-driving, you'd be taking the on self-driving, you'd be taking the on self-driving, you'd be taking the trajectory that the planner would output trajectory that the planner would output trajectory that the planner would output and kind of turn it into the real and kind of turn it into the real and kind of turn it into the real controls to kind of build like drive the controls to kind of build like drive the controls to kind of build like drive the car. Here, we're turning the text into car. Here, we're turning the text into car. Here, we're turning the text into audio that we use to express our voice audio that we use to express our voice audio that we use to express our voice agent's thoughts. And um And um yeah, so here I'm going to be like going yeah, so here I'm going to be like going yeah, so here I'm going to be like going to diving into each one of these to diving into each one of these to diving into each one of these elements and we've made a few kind of elements and we've made a few kind of elements and we've made a few kind of interesting tricks on each of these interesting tricks on each of these interesting tricks on each of these areas to areas to areas to improve the speed of our voice agents improve the speed of our voice agents improve the speed of our voice agents without sacrificing the intelligence. without sacrificing the intelligence. without sacrificing the intelligence. So, the first one is going to be uh the So, the first one is going to be uh the So, the first one is going to be uh the transcriber layer. So, we came up with transcriber layer. So, we came up with transcriber layer. So, we came up with this concept called like the streaming this concept called like the streaming this concept called like the streaming speculative transcriber where speculative transcriber where speculative transcriber where effectively we are layering a fast effectively we are layering a fast effectively we are layering a fast streaming transcriber like Flux on top streaming transcriber like Flux on top streaming transcriber like Flux on top of or kind of below a of or kind of below a of or kind of below a uh scribe V2 or a accurate batch uh scribe V2 or a accurate batch uh scribe V2 or a accurate batch transcription which kind of takes in transcription which kind of takes in transcription which kind of takes in more context. It's a little bit slower, more context. It's a little bit slower, more context. It's a little bit slower, but it will give you more accurate but it will give you more accurate but it will give you more accurate detections.

detections. detections. So, we're going to walk through a So, we're going to walk through a So, we're going to walk through a scenario. So, in this in this case the scenario. So, in this in this case the scenario. So, in this in this case the agent just asked, you know, providing agent just asked, you know, providing agent just asked, you know, providing can you provide your name and date of can you provide your name and date of can you provide your name and date of birth and the user is going to say this birth and the user is going to say this birth and the user is going to say this and we'll see how that plays out um and we'll see how that plays out um and we'll see how that plays out um timing-wise. So, first we're going to timing-wise. So, first we're going to timing-wise. So, first we're going to get, you know, the short detection. Um get, you know, the short detection. Um get, you know, the short detection. Um we'll get it from we'll get it from the we'll get it from we'll get it from the we'll get it from we'll get it from the streaming layer. The accurate streaming layer. The accurate streaming layer. The accurate layer uh the corrective layer is not layer uh the corrective layer is not layer uh the corrective layer is not going to fire because it's the same going to fire because it's the same going to fire because it's the same text. text. text. Um we're going to get some more Um we're going to get some more Um we're going to get some more streaming text detections and in this streaming text detections and in this streaming text detections and in this case the corrective layer is actually case the corrective layer is actually case the corrective layer is actually canceled because we got new um new text. canceled because we got new um new text. canceled because we got new um new text. So, you know, more context, more audio So, you know, more context, more audio So, you know, more context, more audio is going to beat the old accurate one. is going to beat the old accurate one. is going to beat the old accurate one. And here's where kind of the first And here's where kind of the first And here's where kind of the first correction comes in. So, because the correction comes in. So, because the correction comes in. So, because the scribe V2 layer understands, you know, scribe V2 layer understands, you know, scribe V2 layer understands, you know, the the context of the question, it's the the context of the question, it's the the context of the question, it's able to understand that this is talking able to understand that this is talking able to understand that this is talking about name and this is a date of birth. about name and this is a date of birth. about name and this is a date of birth. Then a couple more detections, these are Then a couple more detections, these are Then a couple more detections, these are just punctuation, we don't care. just punctuation, we don't care. just punctuation, we don't care. And so, in the end we kind of release And so, in the end we kind of release And so, in the end we kind of release this text over to the agent.

this text over to the agent. this text over to the agent. And moving on um to the language model And moving on um to the language model And moving on um to the language model layer. layer. layer. So, here since we're kind of using these So, here since we're kind of using these So, here since we're kind of using these slow but intelligent LLMs, we really slow but intelligent LLMs, we really slow but intelligent LLMs, we really want to reduce the number of round trips want to reduce the number of round trips want to reduce the number of round trips and the thing that causes us to do a lot and the thing that causes us to do a lot and the thing that causes us to do a lot of inferences is tool calling. So, one of inferences is tool calling. So, one of inferences is tool calling. So, one way to get rid of that is by having way to get rid of that is by having way to get rid of that is by having background agents do the tool calling background agents do the tool calling background agents do the tool calling for you and kind of for you and kind of for you and kind of um push the tools back into the context um push the tools back into the context um push the tools back into the context of the main agent so that it thinks it of the main agent so that it thinks it of the main agent so that it thinks it made the tool call, but made the tool call, but made the tool call, but um um um but it it it really didn't. but it it it really didn't. but it it it really didn't. So, So, So, uh so, we remember from like detections uh so, we remember from like detections uh so, we remember from like detections from before. from before. from before. So, well, what happened is each one of So, well, what happened is each one of So, well, what happened is each one of these detections is going to trigger a these detections is going to trigger a these detections is going to trigger a um an early um an early um an early kind of generation of the agent and we kind of generation of the agent and we kind of generation of the agent and we but we won't actually but we won't actually but we won't actually emit this out until we're confirming emit this out until we're confirming emit this out until we're confirming that the user has finished speaking. So, that the user has finished speaking. So, that the user has finished speaking. So, in this case, the user says, "Sure." The in this case, the user says, "Sure." The in this case, the user says, "Sure." The agent kind of knows that the user is agent kind of knows that the user is agent kind of knows that the user is about to say something else. Our about to say something else. Our about to say something else. Our background tool calling here, which is background tool calling here, which is background tool calling here, which is going to be helping us find figure out going to be helping us find figure out going to be helping us find figure out the name and the date of birth from the the name and the date of birth from the the name and the date of birth from the user detection, is not firing. So, user detection, is not firing. So, user detection, is not firing. So, nothing much there.

nothing much there. nothing much there. Um the next instant detection comes in. Um the next instant detection comes in. Um the next instant detection comes in. It says that, It says that, It says that, you know, still not really a name. Um you know, still not really a name. Um you know, still not really a name. Um our agent kind of plays along and our agent kind of plays along and our agent kind of plays along and continues there. continues there. continues there. Now, kind of a more more context come Now, kind of a more more context come Now, kind of a more more context come comes back. The agent kind of feels like comes back. The agent kind of feels like comes back. The agent kind of feels like there should be a name. It's going to there should be a name. It's going to there should be a name. It's going to ask to spell it out because it's ask to spell it out because it's ask to spell it out because it's probably thinking there's some probably thinking there's some probably thinking there's some transcription error here. Still no name transcription error here. Still no name transcription error here. Still no name or date of birth. or date of birth. or date of birth. And then finally, this you remember this And then finally, this you remember this And then finally, this you remember this is kind of our corrected um final is kind of our corrected um final is kind of our corrected um final instant detection from the transcriber instant detection from the transcriber instant detection from the transcriber from the Scribe V2. from the Scribe V2. from the Scribe V2. Um here, our eager kind of agent Um here, our eager kind of agent Um here, our eager kind of agent generation that was made without any generation that was made without any generation that was made without any tool calls is going to get canceled tool calls is going to get canceled tool calls is going to get canceled because the background agent finally is because the background agent finally is because the background agent finally is able to find the name and date of birth able to find the name and date of birth able to find the name and date of birth it's looking for. So, it's going to it's looking for. So, it's going to it's looking for. So, it's going to retrigger and now the the agent actually retrigger and now the the agent actually retrigger and now the the agent actually has the context it needs. has the context it needs. has the context it needs. Um and you see here, it's kind of we're Um and you see here, it's kind of we're Um and you see here, it's kind of we're doing it the tool call here is a little doing it the tool call here is a little doing it the tool call here is a little bit um um bit um um bit um um some intelligence there. We're going to some intelligence there. We're going to some intelligence there. We're going to be like, you know, correcting be like, you know, correcting be like, you know, correcting mis-transcriptions of name, and doing mis-transcriptions of name, and doing mis-transcriptions of name, and doing some like phonetic matching here.

some like phonetic matching here. some like phonetic matching here. Um and yeah, and then we'll kind of Um and yeah, and then we'll kind of Um and yeah, and then we'll kind of once we've understood that this is the once we've understood that this is the once we've understood that this is the end of the user utterance, we'll kind of end of the user utterance, we'll kind of end of the user utterance, we'll kind of emit it out. So, pretty standard. emit it out. So, pretty standard. emit it out. So, pretty standard. Okay, and then the next layer here is Okay, and then the next layer here is Okay, and then the next layer here is going to be text-to-speech. So, with going to be text-to-speech. So, with going to be text-to-speech. So, with text-to-speech text-to-speech text-to-speech the goal is to kind of take what the the goal is to kind of take what the the goal is to kind of take what the agent said, and the agent's going to be agent said, and the agent's going to be agent said, and the agent's going to be emitting this in a streaming fashion. emitting this in a streaming fashion. emitting this in a streaming fashion. So, we're going to need to So, we're going to need to So, we're going to need to um produce audio as quickly as possible. um produce audio as quickly as possible. um produce audio as quickly as possible. And ideally, what you can do is before And ideally, what you can do is before And ideally, what you can do is before the agent has even finished generating the agent has even finished generating the agent has even finished generating the full text you can have the audio the full text you can have the audio the full text you can have the audio play, so it's kind of hiding the latency play, so it's kind of hiding the latency play, so it's kind of hiding the latency of finishing the generation. of finishing the generation. of finishing the generation. So, So, So, um I'm going to kind of play the um I'm going to kind of play the um I'm going to kind of play the streaming um streaming um streaming um the stream the streaming uh agent output the stream the streaming uh agent output the stream the streaming uh agent output now. So, starts with you. now. So, starts with you. now. So, starts with you. And yeah, actually before I uh further, And yeah, actually before I uh further, And yeah, actually before I uh further, there's this new concept that we're there's this new concept that we're there's this new concept that we're introducing here called the prefix introducing here called the prefix introducing here called the prefix cache. So, the prefix cache is going to cache. So, the prefix cache is going to cache. So, the prefix cache is going to be looking at the be looking at the be looking at the um agent stream, and seeing if we um agent stream, and seeing if we um agent stream, and seeing if we already have generated audio for that already have generated audio for that already have generated audio for that sequence of words um from like a prior sequence of words um from like a prior sequence of words um from like a prior generation, or maybe like the same generation, or maybe like the same generation, or maybe like the same generation generation generation um in this um in this um in this uh in this call as as well.

uh in this call as as well. uh in this call as as well. So, um it sees the word you. Uh we for So, um it sees the word you. Uh we for So, um it sees the word you. Uh we for this prefix cache, we're going to be, this prefix cache, we're going to be, this prefix cache, we're going to be, you know, we don't want to like you know, we don't want to like you know, we don't want to like immediately hit on every single word. immediately hit on every single word. immediately hit on every single word. We're going to be waiting for a little We're going to be waiting for a little We're going to be waiting for a little bit more words. bit more words. bit more words. Um so, after three words, the prefix Um so, after three words, the prefix Um so, after three words, the prefix cache gets our first hit. cache gets our first hit. cache gets our first hit. And um over here on the right, this is And um over here on the right, this is And um over here on the right, this is kind of our text-to-speech standard kind of our text-to-speech standard kind of our text-to-speech standard provider, you know, Cartesia is a provider, you know, Cartesia is a provider, you know, Cartesia is a text-to-speech engine with web socket text-to-speech engine with web socket text-to-speech engine with web socket support. So, we're we're piping the support. So, we're we're piping the support. So, we're we're piping the agent through the the cache, and also agent through the the cache, and also agent through the the cache, and also piping it through web socket. piping it through web socket. piping it through web socket. Um more tokens come in, more cache, more Um more tokens come in, more cache, more Um more tokens come in, more cache, more sending through web socket. Not much to sending through web socket. Not much to sending through web socket. Not much to say here. say here. say here. And okay, so now we get our first uh And okay, so now we get our first uh And okay, so now we get our first uh kind of first unique thing, which is we kind of first unique thing, which is we kind of first unique thing, which is we found the token that actually causes a found the token that actually causes a found the token that actually causes a cache miss. And it makes sense. If we're cache miss. And it makes sense. If we're cache miss. And it makes sense. If we're kind of caching previous generations, kind of caching previous generations, kind of caching previous generations, um you said your name is is a pretty um you said your name is is a pretty um you said your name is is a pretty common thing, but once you we add in the common thing, but once you we add in the common thing, but once you we add in the name, suddenly we're that's that's going name, suddenly we're that's that's going name, suddenly we're that's that's going to result in the cache miss.

to result in the cache miss. to result in the cache miss. At this point, we're actually going to At this point, we're actually going to At this point, we're actually going to yield out our cached audio. So, you said yield out our cached audio. So, you said yield out our cached audio. So, you said your name is is going to be your name is is going to be your name is is going to be um emitted as the rest of the streaming um emitted as the rest of the streaming um emitted as the rest of the streaming text is coming back. So, at this point, text is coming back. So, at this point, text is coming back. So, at this point, the user hears the agent and user the user hears the agent and user the user hears the agent and user doesn't really know what's going on. doesn't really know what's going on. doesn't really know what's going on. They just looks like really fast They just looks like really fast They just looks like really fast response times to them. response times to them. response times to them. Um and now the kind of remaining text Um and now the kind of remaining text Um and now the kind of remaining text flows through. flows through. flows through. And at this point, we've already emitted And at this point, we've already emitted And at this point, we've already emitted from the cache. The cache has done its from the cache. The cache has done its from the cache. The cache has done its job. Um the rest we can kind of throw job. Um the rest we can kind of throw job. Um the rest we can kind of throw into Cartesia. into Cartesia. into Cartesia. And here's kind of the trick where And here's kind of the trick where And here's kind of the trick where Cartesia has seen the entire transcript Cartesia has seen the entire transcript Cartesia has seen the entire transcript up to this point. It up to this point. It up to this point. It to to Cartesia, like it doesn't know to to Cartesia, like it doesn't know to to Cartesia, like it doesn't know about the existence of this prefix about the existence of this prefix about the existence of this prefix cache. It's just going to generate this cache. It's just going to generate this cache. It's just going to generate this full sentence with, you know, standard full sentence with, you know, standard full sentence with, you know, standard natural prosody. natural prosody. natural prosody. But, what we do is when the generation But, what we do is when the generation But, what we do is when the generation comes back, since we've already played comes back, since we've already played comes back, since we've already played the audio here, we can actually suppress the audio here, we can actually suppress the audio here, we can actually suppress the audio from Cartesia here and just the audio from Cartesia here and just the audio from Cartesia here and just play out the remaining stuff. So, play out the remaining stuff. So, play out the remaining stuff. So, the user, there's might be a tiny bit of the user, there's might be a tiny bit of the user, there's might be a tiny bit of a hiccup. You know, I'll play some audio a hiccup. You know, I'll play some audio a hiccup. You know, I'll play some audio later and you'll know that you probably later and you'll know that you probably later and you'll know that you probably won't be able to notice.

won't be able to notice. won't be able to notice. And effectively, we're kind of taking And effectively, we're kind of taking And effectively, we're kind of taking this and this and this and or taking this part and just emitting it or taking this part and just emitting it or taking this part and just emitting it directly after this audio has finished directly after this audio has finished directly after this audio has finished playing. So, it looks seamless to the playing. So, it looks seamless to the playing. So, it looks seamless to the user. user. user. And yeah, so I guess at this point kind And yeah, so I guess at this point kind And yeah, so I guess at this point kind of this is a lot of technical details. of this is a lot of technical details. of this is a lot of technical details. I'll I mean the best way is just kind of I'll I mean the best way is just kind of I'll I mean the best way is just kind of play a call and see play a call and see play a call and see see this in action. So here I recorded a see this in action. So here I recorded a see this in action. So here I recorded a call and let's just hear what happens. call and let's just hear what happens. call and let's just hear what happens. >> This is Bo OBGYN Elise speaking. >> This is Bo OBGYN Elise speaking. >> This is Bo OBGYN Elise speaking. >> Hi, I think I might be pregnant and I >> Hi, I think I might be pregnant and I >> Hi, I think I might be pregnant and I want to schedule an ultrasound to want to schedule an ultrasound to want to schedule an ultrasound to confirm. confirm. confirm. >> That's exciting news. Can you provide >> That's exciting news. Can you provide >> That's exciting news. Can you provide your name and date of birth? your name and date of birth? your name and date of birth? >> Sure, it's Elise Trial. >> Sure, it's Elise Trial. >> Sure, it's Elise Trial. Date of birth Date of birth Date of birth 2303. >> Thank you. Are you a new patient with >> Thank you. Are you a new patient with us?

us? us? >> Yes. >> Welcome to Bo OBGYN. Can I text you a >> Welcome to Bo OBGYN. Can I text you a link to upload your insurance link to upload your insurance link to upload your insurance information? >> Sure. >> Sure. >> Okay, sent. Let me know when you've >> Okay, sent. Let me know when you've >> Okay, sent. Let me know when you've received it. I've received your insurance I've received your insurance information. The earliest appointment I information. The earliest appointment I information. The earliest appointment I have is Thursday, July 2nd at 10:00 a.m. have is Thursday, July 2nd at 10:00 a.m. have is Thursday, July 2nd at 10:00 a.m. >> Actually, give me a second. >> Actually, give me a second. >> Actually, give me a second. Let me check my calendar. >> Sure, take your time. >> Sure, take your time. >> Um do you have something for next week? >> For next week, I can offer an ultrasound >> For next week, I can offer an ultrasound with Dr. Avery Stone at North Clinic on with Dr. Avery Stone at North Clinic on with Dr. Avery Stone at North Clinic on Tuesday, July 7th at 2:00 p.m. or 3:00 Tuesday, July 7th at 2:00 p.m. or 3:00 Tuesday, July 7th at 2:00 p.m. or 3:00 p.m. Do either of those work for you? p.m. Do either of those work for you? p.m. Do either of those work for you? >> Yeah, 2:00 p.m. works. >> Yeah, 2:00 p.m. works. >> Yeah, 2:00 p.m. works. >> Great. Your appointment has been booked.

>> Great. Your appointment has been booked. >> Great. Your appointment has been booked. We look forward to seeing you then. We look forward to seeing you then. We look forward to seeing you then. >> Thanks. Bye-bye. >> Thanks. Bye-bye. >> Thanks. Bye-bye. >> All right. Yeah, that's pretty much it. >> All right. Yeah, that's pretty much it. >> All right. Yeah, that's pretty much it. Um Um Um Yeah, you can kind of see our Yeah, you can kind of see our Yeah, you can kind of see our all this like uh all this like uh all this like uh streaming streaming streaming and you know, a lot of things are and you know, a lot of things are and you know, a lot of things are happening in the background and and you happening in the background and and you happening in the background and and you know, this is what really makes like know, this is what really makes like know, this is what really makes like voice agents interesting. And um there's voice agents interesting. And um there's voice agents interesting. And um there's a lot of effort that can be done in the a lot of effort that can be done in the a lot of effort that can be done in the harness to really kind of get a um harness to really kind of get a um harness to really kind of get a um natural conversation, which is what natural conversation, which is what natural conversation, which is what we're after. we're after. we're after. Uh okay. Yeah, so I guess briefly, you Uh okay. Yeah, so I guess briefly, you Uh okay. Yeah, so I guess briefly, you know, in the last part, I want to just know, in the last part, I want to just know, in the last part, I want to just talk a little bit about Elise. So I talk a little bit about Elise. So I talk a little bit about Elise. So I think Elise, you know, our headquarters think Elise, you know, our headquarters think Elise, you know, our headquarters are in New York and kind of we're trying are in New York and kind of we're trying are in New York and kind of we're trying to expand our presence here in the Bay to expand our presence here in the Bay to expand our presence here in the Bay Area. Um we I think it's maybe like a Area. Um we I think it's maybe like a Area. Um we I think it's maybe like a different style of company that I think different style of company that I think different style of company that I think people are people are people are like uh think of when they think about like uh think of when they think about like uh think of when they think about AI startups in San Francisco. Where AI startups in San Francisco. Where AI startups in San Francisco. Where we're actually very focused on um we're actually very focused on um we're actually very focused on um just like helping people and helping just like helping people and helping just like helping people and helping people where they need it, like kind of people where they need it, like kind of people where they need it, like kind of the life's most critical areas. We work the life's most critical areas. We work the life's most critical areas. We work on housing, health care and uh we're on housing, health care and uh we're on housing, health care and uh we're doing really well and you know, here's doing really well and you know, here's doing really well and you know, here's there's a link here to there's a link here to there's a link here to um you kind of join our team and there's um you kind of join our team and there's um you kind of join our team and there's going to we're going to be uh posting a going to we're going to be uh posting a going to we're going to be uh posting a lot on Twitter, so you can follow us at lot on Twitter, so you can follow us at lot on Twitter, so you can follow us at EliseAI as well.

EliseAI as well. EliseAI as well. Um yeah, that's that's it. Um yeah, that's that's it. Um yeah, that's that's it. >> [applause]