"My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow

>> Okay, hello everyone. >> Okay, hello everyone. So, my name is Midam Kim. I am an ML my name is Midam Kim. I am an ML engineer from ServiceNow and I'll be engineer from ServiceNow and I'll be engineer from ServiceNow and I'll be talking about a linguistic framework for talking about a linguistic framework for talking about a linguistic framework for voice AI. So, So, quick background of me so you know where quick background of me so you know where quick background of me so you know where I'm coming from. I'm coming from. I'm coming from. Like I said, I'm an ML engineer at Like I said, I'm an ML engineer at Like I said, I'm an ML engineer at ServiceNow, but I'm also a researcher, ServiceNow, but I'm also a researcher, ServiceNow, but I'm also a researcher, lifelong researcher, of speech lifelong researcher, of speech lifelong researcher, of speech communication in the wild. communication in the wild. communication in the wild. So, my motto is doing linguistics and So, my motto is doing linguistics and So, my motto is doing linguistics and what I'm going to be doing today is to what I'm going to be doing today is to what I'm going to be doing today is to hand you that lens of linguistics. So, have you experienced voice AI So, have you experienced voice AI failures? failures? failures? Yeah, like everyone. Yeah, like everyone. Yeah, like everyone. >> [laughter] >> So, I'm going to introduce an example >> So, I'm going to introduce an example that I experienced myself. that I experienced myself. that I experienced myself. So, the bot asked me, "Could you please So, the bot asked me, "Could you please So, the bot asked me, "Could you please spell your first name?"

spell your first name?" spell your first name?" And then I slowly start to spell my And then I slowly start to spell my And then I slowly start to spell my name. name. name. Yes, it is m i d a m. Yes, it is m i d a m. Yes, it is m i d a m. And the bot says, "Confirming with you, And the bot says, "Confirming with you, And the bot says, "Confirming with you, is it m i d a n?" is it m i d a n?" is it m i d a n?" And then I say, "No, it is m i d a m." And then I say, "No, it is m i d a m." And then I say, "No, it is m i d a m." Um Um Um and the bot says, and the bot says, and the bot says, "Thank you for your correction. Happy to "Thank you for your correction. Happy to "Thank you for your correction. Happy to help you today, Madam." help you today, Madam." help you today, Madam." And I then I And I then I And I then I get slightly annoyed, more annoyed, get slightly annoyed, more annoyed, get slightly annoyed, more annoyed, because my name is Midam, not Madam. because my name is Midam, not Madam. because my name is Midam, not Madam. And then it asked me about, "Now, what And then it asked me about, "Now, what And then it asked me about, "Now, what is your account number? is your account number? is your account number? And then, I start start getting And then, I start start getting And then, I start start getting confused. What is that account number confused. What is that account number confused. What is that account number thing? thing? thing? And then, And then, And then, I try to find uh information about that. I try to find uh information about that. I try to find uh information about that. So, So, So, which one? Um it must be and I start which one? Um it must be and I start which one? Um it must be and I start uh uh uh slowly start spelling the account slowly start spelling the account slowly start spelling the account number. So, it is A X 4 5 1.

number. So, it is A X 4 5 1. number. So, it is A X 4 5 1. And then, I take time because I'm not And then, I take time because I'm not And then, I take time because I'm not used to reading this strange number. used to reading this strange number. used to reading this strange number. And then, the bot cuts me off. And then, the bot cuts me off. And then, the bot cuts me off. And then, it says, I couldn't find your And then, it says, I couldn't find your And then, it says, I couldn't find your record. record. record. And then, without even trying, it asked And then, without even trying, it asked And then, without even trying, it asked me to repeat that again. Can you please me to repeat that again. Can you please me to repeat that again. Can you please repeat that? And then, I get super repeat that? And then, I get super repeat that? And then, I get super annoyed and then, I can say, can I talk annoyed and then, I can say, can I talk annoyed and then, I can say, can I talk to a person? to a person? to a person? I just don't want to deal with you I just don't want to deal with you I just don't want to deal with you anymore. anymore. anymore. So, this is a very typical pattern of So, this is a very typical pattern of So, this is a very typical pattern of voice AI, unfortunately, at this point. voice AI, unfortunately, at this point. voice AI, unfortunately, at this point. So, I just want to navigate how we can So, I just want to navigate how we can So, I just want to navigate how we can solve this problem solve this problem solve this problem with linguistics. So, voice AI is booming. So, voice AI is booming. But users are still often preferring But users are still often preferring But users are still often preferring human agents over voice agents. human agents over voice agents. human agents over voice agents. How can we mitigate this issue? How can we mitigate this issue? How can we mitigate this issue? But in the first place, what are the But in the first place, what are the But in the first place, what are the actual problems? actual problems? actual problems? So, I think we can think about a So, I think we can think about a So, I think we can think about a fundamental frame framework to fundamental frame framework to fundamental frame framework to understand this into an architecture of understand this into an architecture of understand this into an architecture of voice AI, voice AI, voice AI, which is called linguistics.

So, as all of us already know, So, as all of us already know, human communication is a joint activity, human communication is a joint activity, human communication is a joint activity, like the thing that we're doing right like the thing that we're doing right like the thing that we're doing right now. now. now. So, I give you my sounds and words. So, I give you my sounds and words. So, I give you my sounds and words. You hear them. You hear them. You hear them. And then, if it is a conversation, And then, if it is a conversation, And then, if it is a conversation, you're going to give me your sounds and you're going to give me your sounds and you're going to give me your sounds and your words. your words. your words. And then this is going back and forth And then this is going back and forth And then this is going back and forth through interaction. And then in this process, we're And then in this process, we're continuously continuously continuously processing and updating our mental processing and updating our mental processing and updating our mental models. models. models. So that's a joint activity So that's a joint activity So that's a joint activity for human communication. for human communication. for human communication. And I would like to say And I would like to say And I would like to say in the voice AI human communication, in the voice AI human communication, in the voice AI human communication, it also has to be a joint activity like it also has to be a joint activity like it also has to be a joint activity like this. this. this. Because that's the only thing that we Because that's the only thing that we Because that's the only thing that we know about human communication as a know about human communication as a know about human communication as a human being. We have been evolving human being. We have been evolving human being. We have been evolving thousands of years as communicators, and thousands of years as communicators, and thousands of years as communicators, and this is what we know. So we expect the this is what we know. So we expect the this is what we know. So we expect the same thing to bots.

So let me go over the failure scene of So let me go over the failure scene of my call with the voice agent my call with the voice agent my call with the voice agent in this framework. in this framework. in this framework. So you see there's listen So you see there's listen So you see there's listen and speak for each party. So I start spelling my first name. So I start spelling my first name. And then the bot did not hear that the And then the bot did not hear that the And then the bot did not hear that the difference between M and N correctly, so difference between M and N correctly, so difference between M and N correctly, so it's an it's an it's an SCT failure in the listening level. SCT failure in the listening level. SCT failure in the listening level. And then the TTS applies only And then the TTS applies only And then the TTS applies only English-centric reading rules to my English-centric reading rules to my English-centric reading rules to my name, M I D A M, would read it as Midam name, M I D A M, would read it as Midam name, M I D A M, would read it as Midam in the in the in the American English version. So I'm confused, but at this time I'm So I'm confused, but at this time I'm kind of generous because that happens a kind of generous because that happens a kind of generous because that happens a lot even with human beings. So I'm okay. lot even with human beings. So I'm okay. lot even with human beings. So I'm okay. But then when it brought But then when it brought But then when it brought brought up account number thing brought up account number thing brought up account number thing because I don't know what that is, because I don't know what that is, because I don't know what that is, I'm confused again. I'm confused again. I'm confused again. But I'm adaptive, I can find I can look But I'm adaptive, I can find I can look But I'm adaptive, I can find I can look for it.

for it. for it. So I found the number, start reading it, So I found the number, start reading it, So I found the number, start reading it, but but but the STT did not recognize the word unit the STT did not recognize the word unit the STT did not recognize the word unit correctly, so correctly, so correctly, so it cuts me off, and uh it cuts me off, and uh it cuts me off, and uh uh finally, it's uh eventually talked uh finally, it's uh eventually talked uh finally, it's uh eventually talked over me. over me. over me. So, I get So, I get So, I get really irritated. really irritated. really irritated. And then, when it asked me for the And then, when it asked me for the And then, when it asked me for the repetition of the same information, and repetition of the same information, and repetition of the same information, and then, it is clear that the spot is not then, it is clear that the spot is not then, it is clear that the spot is not tracking the mental model with me. tracking the mental model with me. tracking the mental model with me. And then, very rudely, it's uh does not And then, very rudely, it's uh does not And then, very rudely, it's uh does not even try interactive clarification, even try interactive clarification, even try interactive clarification, which is a common strategy by human which is a common strategy by human which is a common strategy by human beings. beings. beings. So, I don't want to deal with this So, I don't want to deal with this So, I don't want to deal with this anymore, so I say, "Can I talk to a anymore, so I say, "Can I talk to a anymore, so I say, "Can I talk to a person?" So, So, let's go over the uh the framework let's go over the uh the framework let's go over the uh the framework again. So, the these are the linguistic again. So, the these are the linguistic again. So, the these are the linguistic components that are expected and well components that are expected and well components that are expected and well maintained in human-to-human voi- uh maintained in human-to-human voi- uh maintained in human-to-human voi- uh uh conversation.

uh conversation. uh conversation. So, there are listening channels, a So, there are listening channels, a So, there are listening channels, a listening channel and speaking channel, listening channel and speaking channel, listening channel and speaking channel, and there are different components like and there are different components like and there are different components like sounds, words, interaction, and mental sounds, words, interaction, and mental sounds, words, interaction, and mental model. model. model. So, the first component is, does the bot So, the first component is, does the bot So, the first component is, does the bot recognize the user's speech well? recognize the user's speech well? recognize the user's speech well? And all of these technical terms And all of these technical terms And all of these technical terms uh will fall under this. uh will fall under this. uh will fall under this. And then, there was there's going to be And then, there was there's going to be And then, there was there's going to be this second component, which is words in this second component, which is words in this second component, which is words in the listening channel. So, does the bot the listening channel. So, does the bot the listening channel. So, does the bot understand the user's words? understand the user's words? understand the user's words? And then, the third one is, does the bot And then, the third one is, does the bot And then, the third one is, does the bot wait until the right timing to for its wait until the right timing to for its wait until the right timing to for its turn? It's about It's going to be about turn? It's about It's going to be about turn? It's about It's going to be about uh listening channel interaction. And then, uh the last part is mental And then, uh the last part is mental model. So, does the bot understand the model. So, does the bot understand the model. So, does the bot understand the user's intention user's intention user's intention in the listening part? in the listening part? in the listening part? And then, we can also go to the speaking And then, we can also go to the speaking And then, we can also go to the speaking channel, so it's going to be about channel, so it's going to be about channel, so it's going to be about pronunciation for the sound. pronunciation for the sound. pronunciation for the sound. And also there's about understand the And also there's about understand the And also there's about understand the the words users are the words users are the words users are uh there's about choose the words the uh there's about choose the words the uh there's about choose the words the user can understand.

user can understand. user can understand. And in the interaction part, there's And in the interaction part, there's And in the interaction part, there's about speak with the right timing. about speak with the right timing. about speak with the right timing. And lastly, there's about speak with the And lastly, there's about speak with the And lastly, there's about speak with the information the user actually need. So, there are a lot of engineering or So, there are a lot of engineering or linguistic or cognitive science terms linguistic or cognitive science terms linguistic or cognitive science terms that are in here that that are here. Um that are in here that that are here. Um that are in here that that are here. Um you can see now see that all of those you can see now see that all of those you can see now see that all of those have their right spots in this have their right spots in this have their right spots in this linguistic framework. And importantly, these components are And importantly, these components are interdependent, interdependent, interdependent, not separate or uh independent from each not separate or uh independent from each not separate or uh independent from each other. They're interdependent and other. They're interdependent and other. They're interdependent and they're aligned. So, when you want to do they're aligned. So, when you want to do they're aligned. So, when you want to do good things about sounds, good things about sounds, good things about sounds, you have to think about words level. you have to think about words level. you have to think about words level. And then when you want to do good things And then when you want to do good things And then when you want to do good things about these sounds and words, about these sounds and words, about these sounds and words, you also have to uh account for you also have to uh account for you also have to uh account for interaction, so turn taking or turn interaction, so turn taking or turn interaction, so turn taking or turn detection. detection. detection. And then finally, you want to uh have And then finally, you want to uh have And then finally, you want to uh have good uh task completion, which is the good uh task completion, which is the good uh task completion, which is the goal of these mental model uh layer.

goal of these mental model uh layer. goal of these mental model uh layer. Then you have to have all of these. Then you have to have all of these. Then you have to have all of these. Without all of those, without any of Without all of those, without any of Without all of those, without any of those, any of those components, your those, any of those components, your those, any of those components, your voice agent will fail. voice agent will fail. voice agent will fail. And then finally, And then finally, And then finally, uh it has to be well aligned. All of uh it has to be well aligned. All of uh it has to be well aligned. All of these have to be well aligned. these have to be well aligned. these have to be well aligned. And additionally, you have to keep your And additionally, you have to keep your And additionally, you have to keep your mind keep in mind that mind keep in mind that mind keep in mind that this is happening on the timeline. this is happening on the timeline. this is happening on the timeline. What I mean by that is it is silently What I mean by that is it is silently What I mean by that is it is silently tracked. Unlike in chat, in chat you see tracked. Unlike in chat, in chat you see tracked. Unlike in chat, in chat you see the history of what was said the history of what was said the history of what was said uh as text. uh as text. uh as text. But in voice agent experience, But in voice agent experience, But in voice agent experience, uh, you say something, and the bot says uh, you say something, and the bot says uh, you say something, and the bot says something, you go back and forth, something, you go back and forth, something, you go back and forth, and then see, all these waveforms, the and then see, all these waveforms, the and then see, all these waveforms, the air via the vibration in the, uh, in the air via the vibration in the, uh, in the air via the vibration in the, uh, in the air, they're all gone. air, they're all gone. air, they're all gone. And only the user's mental model is the And only the user's mental model is the And only the user's mental model is the thing that's left, and that matters.

thing that's left, and that matters. thing that's left, and that matters. So, sounds, words, interactions vanish So, sounds, words, interactions vanish So, sounds, words, interactions vanish the moment they're spoken, the moment they're spoken, the moment they're spoken, but the mental model proceeds and grows but the mental model proceeds and grows but the mental model proceeds and grows over the timeline. over the timeline. over the timeline. So, this is what you have to So, this is what you have to So, this is what you have to target target target for user satisfaction. for user satisfaction. for user satisfaction. And then, what can we do And then, what can we do And then, what can we do for the bot to meet the standard of the for the bot to meet the standard of the for the bot to meet the standard of the user? So, what we can do, uh, would include, So, what we can do, uh, would include, of course, choosing good ASR models or of course, choosing good ASR models or of course, choosing good ASR models or configurations and do some configurations and do some configurations and do some post-processing, post-processing, post-processing, uh, choosing good TTS models, uh, choosing good TTS models, uh, choosing good TTS models, configurations, and pre-processing, configurations, and pre-processing, configurations, and pre-processing, and, uh, carefully curate the vocabulary and, uh, carefully curate the vocabulary and, uh, carefully curate the vocabulary that can be shared between the bot and that can be shared between the bot and that can be shared between the bot and the user, the user, the user, and do good job of a turn-to-turn and do good job of a turn-to-turn and do good job of a turn-to-turn detection, latency, and turn-taking. detection, latency, and turn-taking. detection, latency, and turn-taking. Um, and very importantly, we have to, it Um, and very importantly, we have to, it Um, and very importantly, we have to, it would be great if we can do good emotion would be great if we can do good emotion would be great if we can do good emotion detection and handling, and context detection and handling, and context detection and handling, and context retention, and by context, what I mean retention, and by context, what I mean retention, and by context, what I mean is context about all of these.

And importantly, And importantly, uh, it has to be dynamic because things uh, it has to be dynamic because things uh, it has to be dynamic because things are always changing, uh, throughout over are always changing, uh, throughout over are always changing, uh, throughout over the course of the call. So, we would the course of the call. So, we would the course of the call. So, we would have to do this management dynamically have to do this management dynamically have to do this management dynamically along the timeline along the timeline along the timeline for different kinds of people. for different kinds of people. for different kinds of people. So, kids or different kinds of people So, kids or different kinds of people So, kids or different kinds of people like these will have different like these will have different like these will have different expectations that we have to satisfy. expectations that we have to satisfy. expectations that we have to satisfy. Uh, not just when they're happy, but Uh, not just when they're happy, but Uh, not just when they're happy, but also when they're not happy. also when they're not happy. also when they're not happy. So, only then you can pursue a dynamic So, only then you can pursue a dynamic So, only then you can pursue a dynamic and truly scalable orchestration of and truly scalable orchestration of and truly scalable orchestration of voice AI. voice AI. voice AI. So, it's a very difficult job to do. We always say that voice is the most We always say that voice is the most natural way of communication, but it is natural way of communication, but it is natural way of communication, but it is not actually not easy. Behind the scene, not actually not easy. Behind the scene, not actually not easy. Behind the scene, it is thanks to this linguistic it is thanks to this linguistic it is thanks to this linguistic orchestration. orchestration. orchestration. When your bot is not good at it, When your bot is not good at it, When your bot is not good at it, it's a catastrophic failure. Um, so paying attention to this Um, so paying attention to this linguistic framework would have lots of linguistic framework would have lots of linguistic framework would have lots of business implications because then you business implications because then you business implications because then you can uh can uh can uh decrease all of these user frustration, decrease all of these user frustration, decrease all of these user frustration, task failures, live agent escalation, or task failures, live agent escalation, or task failures, live agent escalation, or abandoned calls, or silent failures.

So, in ServiceNow, we have made a a good So, in ServiceNow, we have made a a good uh benchmark end-to-end benchmark called uh benchmark end-to-end benchmark called uh benchmark end-to-end benchmark called Eva bench. So, you can try that to Eva bench. So, you can try that to Eva bench. So, you can try that to diagnose your voice agent's uh status. Um, key takeaways. Um, key takeaways. So, voice AI is a joint activity between So, voice AI is a joint activity between So, voice AI is a joint activity between the bot and the user, not just a the bot and the user, not just a the bot and the user, not just a pipeline. pipeline. pipeline. And we must serve users' needs in And we must serve users' needs in And we must serve users' needs in multiple layers real time. multiple layers real time. multiple layers real time. It's not that I have given you a fix It's not that I have given you a fix It's not that I have given you a fix today because there's nothing like that. today because there's nothing like that. today because there's nothing like that. It just uh the fix is in you and your It just uh the fix is in you and your It just uh the fix is in you and your system. system. system. But, what I have given you is today is But, what I have given you is today is But, what I have given you is today is the linguistic framework you can try to the linguistic framework you can try to the linguistic framework you can try to diagnose your system diagnose your system diagnose your system and to build your system upon. and to build your system upon. and to build your system upon. You can try Eva, but also you can learn You can try Eva, but also you can learn You can try Eva, but also you can learn linguistics and hire linguists. linguistics and hire linguists. linguistics and hire linguists. Um, another thing I want to remind you Um, another thing I want to remind you Um, another thing I want to remind you of is that business implications are of is that business implications are of is that business implications are linguistic implications and vice versa linguistic implications and vice versa linguistic implications and vice versa in this voice AI scene. Because voice is in this voice AI scene. Because voice is in this voice AI scene. Because voice is fundamentally a linguistic and very fundamentally a linguistic and very fundamentally a linguistic and very human and cognitive experience.

I would like to ask you a longer term I would like to ask you a longer term question. question. question. Speakers adapt. So, I Speakers adapt. So, I Speakers adapt. So, I I'm pretty sure that in this talk in my I'm pretty sure that in this talk in my I'm pretty sure that in this talk in my talk with you guys today, you have talk with you guys today, you have talk with you guys today, you have learned something about me, about my learned something about me, about my learned something about me, about my speaking style, what kind of accents I speaking style, what kind of accents I speaking style, what kind of accents I speak, what kind of words I'm using. So, speak, what kind of words I'm using. So, speak, what kind of words I'm using. So, next time I see you guys in person, you next time I see you guys in person, you next time I see you guys in person, you would find it more comfortable to talk would find it more comfortable to talk would find it more comfortable to talk to me because you have paid attention to to me because you have paid attention to to me because you have paid attention to me. me. me. Right? So, speakers are always adapting. Right? So, speakers are always adapting. Right? So, speakers are always adapting. So, the user will be adapting to your So, the user will be adapting to your So, the user will be adapting to your voice agent throughout the call. So, is voice agent throughout the call. So, is voice agent throughout the call. So, is your system ready for them to your system ready for them to your system ready for them to use you better, use it your voice agent use you better, use it your voice agent use you better, use it your voice agent better the next time? better the next time? better the next time? And And And language is always change. So, is your language is always change. So, is your language is always change. So, is your voice agent ready for language change in voice agent ready for language change in voice agent ready for language change in 1 year or 6 months even? So, thank you. So, thank you. >> [applause]