From Ambient Documentation to Clinical Intelligence — Chaitanya Asawa, Abridge

>> Thank you so much for everyone being >> Thank you so much for everyone being here. We're going to get started in a here. We're going to get started in a here. We're going to get started in a second. Um but before we get started, second. Um but before we get started, second. Um but before we get started, I am curious, how many of you currently I am curious, how many of you currently I am curious, how many of you currently work in the healthcare industry in some work in the healthcare industry in some work in the healthcare industry in some shape or form? shape or form? shape or form? Oh, that's amazing to hear. Uh how many Oh, that's amazing to hear. Uh how many Oh, that's amazing to hear. Uh how many of you are clinicians by training? of you are clinicians by training? of you are clinicians by training? Okay, a couple. How many people in the Okay, a couple. How many people in the Okay, a couple. How many people in the room are engineers? room are engineers? room are engineers? Okay, awesome. Okay, awesome. Okay, awesome. Um and then how many people have heard Um and then how many people have heard Um and then how many people have heard of Abridge before? of Abridge before? of Abridge before? Okay. Okay. Okay. Awesome. Awesome. Awesome. Uh well, Uh well, Uh well, I'm going to let you hear actually from I'm going to let you hear actually from I'm going to let you hear actually from our users to start off on a little about our users to start off on a little about our users to start off on a little about Abridge. >> Full day of 22 patients, out by 4:30 >> Full day of 22 patients, out by 4:30 p.m., p.m., p.m., >> [music] >> [music] >> [music] >> notes done. That's nice. >> When I think about Abridge, I think the >> When I think about Abridge, I think the thing that comes to mind is it's really thing that comes to mind is it's really thing that comes to mind is it's really a cornerstone of how I practice medicine a cornerstone of how I practice medicine a cornerstone of how I practice medicine today. Um there's just no way um today. Um there's just no way um today. Um there's just no way um I would do a clinic or see a patient I would do a clinic or see a patient I would do a clinic or see a patient without using without using without using >> I can be present throughout my clinical >> I can be present throughout my clinical >> I can be present throughout my clinical encounters. I don't have to think about encounters. I don't have to think about encounters. I don't have to think about um oh wait, did I get that? Do I need to um oh wait, did I get that? Do I need to um oh wait, did I get that? Do I need to write that down? Because I know Abridge write that down? Because I know Abridge write that down? Because I know Abridge has my back and has everything ready for has my back and has everything ready for has my back and has everything ready for me.

>> Abridge makes me feel free cuz I can >> Abridge makes me feel free cuz I can really look at a patient, really listen, really look at a patient, really listen, really look at a patient, really listen, and not have to be thinking about what and not have to be thinking about what and not have to be thinking about what do I need to put in the computer. do I need to put in the computer. do I need to put in the computer. >> Full day of 22 >> Our marketing team produces really good >> Our marketing team produces really good videos and so they always light me up. videos and so they always light me up. videos and so they always light me up. Um Um Um but the goal of this talk for me, and I but the goal of this talk for me, and I but the goal of this talk for me, and I know that we have a lot of engineers in know that we have a lot of engineers in know that we have a lot of engineers in the room, my my goal is to talk about the room, my my goal is to talk about the room, my my goal is to talk about health care as a domain, at least I felt health care as a domain, at least I felt health care as a domain, at least I felt in the past that there was a lot of in the past that there was a lot of in the past that there was a lot of stigma around maybe the technical stigma around maybe the technical stigma around maybe the technical problems weren't as interesting in problems weren't as interesting in problems weren't as interesting in health care. And it is true in some health care. And it is true in some health care. And it is true in some ways, there's some parts of health care ways, there's some parts of health care ways, there's some parts of health care that might not be as tech forward. A lot that might not be as tech forward. A lot that might not be as tech forward. A lot of things run on fax machines, for of things run on fax machines, for of things run on fax machines, for example. Um but I want to give exposure example. Um but I want to give exposure example. Um but I want to give exposure throughout this talk of two things. One, throughout this talk of two things. One, throughout this talk of two things. One, a bridge's journey from clinical a bridge's journey from clinical a bridge's journey from clinical documentation to clinical intelligence documentation to clinical intelligence documentation to clinical intelligence and what that looks like. Um and then and what that looks like. Um and then and what that looks like. Um and then two, I want to expose you to some of the two, I want to expose you to some of the two, I want to expose you to some of the technical problems we work on um and technical problems we work on um and technical problems we work on um and that have to be the that are truly that have to be the that are truly that have to be the that are truly frontier AI product problems that have frontier AI product problems that have frontier AI product problems that have the highest stakes.

the highest stakes. the highest stakes. A little about me, my name is Chaitanya, A little about me, my name is Chaitanya, A little about me, my name is Chaitanya, you can call me Chai. you can call me Chai. you can call me Chai. Um my career has always been in AI Um my career has always been in AI Um my career has always been in AI companies and startups. I first started companies and startups. I first started companies and startups. I first started as an research engineer at a company as an research engineer at a company as an research engineer at a company called Vicarious, uh which its goal was called Vicarious, uh which its goal was called Vicarious, uh which its goal was actually to develop AGI, but they took actually to develop AGI, but they took actually to develop AGI, but they took very different methods. They wanted very different methods. They wanted very different methods. They wanted methods inspired by neuroscience and methods inspired by neuroscience and methods inspired by neuroscience and probabilistic graphical models. probabilistic graphical models. probabilistic graphical models. Um and they concretely worked on Um and they concretely worked on Um and they concretely worked on robotics. robotics. robotics. Uh then I started working at this Uh then I started working at this Uh then I started working at this company called Glean cuz I faced this company called Glean cuz I faced this company called Glean cuz I faced this problem in my workplace itself. How like problem in my workplace itself. How like problem in my workplace itself. How like information scattered all over the information scattered all over the information scattered all over the place, context is everywhere and it's so place, context is everywhere and it's so place, context is everywhere and it's so key to decision-making. And Glean was key to decision-making. And Glean was key to decision-making. And Glean was building basically the ChatGPT for your building basically the ChatGPT for your building basically the ChatGPT for your workplace. I was there about 6 and 1/2 workplace. I was there about 6 and 1/2 workplace. I was there about 6 and 1/2 years, as one of the earliest engineers years, as one of the earliest engineers years, as one of the earliest engineers as we went from 10 people to over 1,100 as we went from 10 people to over 1,100 as we went from 10 people to over 1,100 people and work with some of the largest people and work with some of the largest people and work with some of the largest companies all over the world. companies all over the world. companies all over the world. Um but that journey was amazing. I love Um but that journey was amazing. I love Um but that journey was amazing. I love that product. I love that company. I that product. I love that company. I that product. I love that company. I love the people there. Um love the people there. Um love the people there. Um but I've actually always really been but I've actually always really been but I've actually always really been interested in health care. I remember a interested in health care. I remember a interested in health care. I remember a decade ago, cover of Nature magazine was decade ago, cover of Nature magazine was decade ago, cover of Nature magazine was AI to detect skin cancer. I was like, AI to detect skin cancer. I was like, AI to detect skin cancer. I was like, "Wow." As someone interested in AI, that "Wow." As someone interested in AI, that "Wow." As someone interested in AI, that was amazing. At the same time, I went to was amazing. At the same time, I went to was amazing. At the same time, I went to the hospital for something, and I the hospital for something, and I the hospital for something, and I remember seeing Oh, coming back It was a remember seeing Oh, coming back It was a remember seeing Oh, coming back It was a really minor thing, but I remember really minor thing, but I remember really minor thing, but I remember coming back and looking at the bill, and coming back and looking at the bill, and coming back and looking at the bill, and I was like, I was like, I was like, "I'm not really sure what exactly I paid "I'm not really sure what exactly I paid "I'm not really sure what exactly I paid for." Um and so, there's these known for." Um and so, there's these known for." Um and so, there's these known problems in health care, and I'll talk a problems in health care, and I'll talk a problems in health care, and I'll talk a little about some of them, access to little about some of them, access to little about some of them, access to care and cost. And then, we had these care and cost. And then, we had these care and cost. And then, we had these amazing solutions. I was like, "Why amazing solutions. I was like, "Why amazing solutions. I was like, "Why don't we bridge these things together?"

don't we bridge these things together?" don't we bridge these things together?" Um and remember, this is about a decade Um and remember, this is about a decade Um and remember, this is about a decade ago. Um and so, I actually started this ago. Um and so, I actually started this ago. Um and so, I actually started this seminar where I invited speakers who seminar where I invited speakers who seminar where I invited speakers who were physicians, researchers, were physicians, researchers, were physicians, researchers, entrepreneurs to to talk about the entrepreneurs to to talk about the entrepreneurs to to talk about the space. And what I learned was space. And what I learned was space. And what I learned was while there was really, really cool while there was really, really cool while there was really, really cool technology, very little of it made its technology, very little of it made its technology, very little of it made its way into the clinic at that time. way into the clinic at that time. way into the clinic at that time. And so, again, my my journey went a And so, again, my my journey went a And so, again, my my journey went a different way. But as I as I peeked my different way. But as I as I peeked my different way. But as I as I peeked my head out 10 years 10 years later then, head out 10 years 10 years later then, head out 10 years 10 years later then, actually, our technology has gotten actually, our technology has gotten actually, our technology has gotten better than ever, as everyone knows, and better than ever, as everyone knows, and better than ever, as everyone knows, and the AI wave has just taken over the the AI wave has just taken over the the AI wave has just taken over the whole world, has also influenced health whole world, has also influenced health whole world, has also influenced health care. And as you as you saw towards the care. And as you as you saw towards the care. And as you as you saw towards the end of that video, a bridge in the end of that video, a bridge in the end of that video, a bridge in the matter of 2 to 3 years got its way into matter of 2 to 3 years got its way into matter of 2 to 3 years got its way into 300 of the largest health systems 300 of the largest health systems 300 of the largest health systems in the United States, Kaiser, Mayo, in the United States, Kaiser, Mayo, in the United States, Kaiser, Mayo, Johns Hopkins, Sutter, and so forth. And Johns Hopkins, Sutter, and so forth. And Johns Hopkins, Sutter, and so forth. And maybe maybe you've visited some of these maybe maybe you've visited some of these maybe maybe you've visited some of these hospital systems. And once you're inside hospital systems. And once you're inside hospital systems. And once you're inside the hospital systems, you realize the hospital systems, you realize the hospital systems, you realize there's so, so much more you can do. And there's so, so much more you can do. And there's so, so much more you can do. And I'll talk about that journey that we've I'll talk about that journey that we've I'll talk about that journey that we've had. had. had. I specifically work on I specifically work on I specifically work on lead our engineering teams for clinical lead our engineering teams for clinical lead our engineering teams for clinical decision support and our agentic decision support and our agentic decision support and our agentic experiences that the technology is now experiences that the technology is now experiences that the technology is now enabled and how we can bring that to enabled and how we can bring that to enabled and how we can bring that to health care.

But first, maybe maybe some of the But first, maybe maybe some of the problems that inspire us as a company at problems that inspire us as a company at problems that inspire us as a company at a bridge. One, one of the things that a bridge. One, one of the things that a bridge. One, one of the things that we've noticed or many people economists we've noticed or many people economists we've noticed or many people economists have noticed over the past few decades have noticed over the past few decades have noticed over the past few decades is actually in many other industries, is actually in many other industries, is actually in many other industries, you actually see the cost of a good go you actually see the cost of a good go you actually see the cost of a good go down. And that's because the down. And that's because the down. And that's because the productivity has increased. But in productivity has increased. But in productivity has increased. But in health care, we actually see health care, we actually see health care, we actually see administrative costs have only gone up administrative costs have only gone up administrative costs have only gone up over the past over the past over the past a few decades a few decades a few decades and productivity hasn't necessarily and productivity hasn't necessarily and productivity hasn't necessarily increased. increased. increased. And a lot of our problems in health care And a lot of our problems in health care And a lot of our problems in health care we solve with labor, but even that we we solve with labor, but even that we we solve with labor, but even that we cannot actually keep up. So there's this cannot actually keep up. So there's this cannot actually keep up. So there's this bit of this like productivity bit of this like productivity bit of this like productivity paradox you might have heard of like paradox you might have heard of like paradox you might have heard of like Baumol's cost disease. And it's part of Baumol's cost disease. And it's part of Baumol's cost disease. And it's part of partially because technology I think partially because technology I think partially because technology I think hasn't fully touched health care as much hasn't fully touched health care as much hasn't fully touched health care as much as it's touched other industries to as it's touched other industries to as it's touched other industries to increase that productivity. increase that productivity. increase that productivity. >> [clears throat] >> [clears throat] >> [clears throat] >> A few other problems, you know, >> A few other problems, you know, >> A few other problems, you know, hospitals are shutting down and margins hospitals are shutting down and margins hospitals are shutting down and margins are actually razor thin for many health are actually razor thin for many health are actually razor thin for many health systems. Of course, some patients have systems. Of course, some patients have systems. Of course, some patients have massive medical massive medical massive medical debt and then we actually and the debt and then we actually and the debt and then we actually and the problem that's another problem that's problem that's another problem that's problem that's another problem that's very near and dear to our heart is that very near and dear to our heart is that very near and dear to our heart is that we hear all the time that doctors are we hear all the time that doctors are we hear all the time that doctors are burnt out and they actually often don't burnt out and they actually often don't burnt out and they actually often don't recommend it as a profession to recommend it as a profession to recommend it as a profession to to their children.

to their children. to their children. So what we started as as a company was So what we started as as a company was So what we started as as a company was working on clinical documentation. So working on clinical documentation. So working on clinical documentation. So the idea here, if you're not familiar the idea here, if you're not familiar the idea here, if you're not familiar with it, is at the end of every patient with it, is at the end of every patient with it, is at the end of every patient visit, the the clinician must create a visit, the the clinician must create a visit, the the clinician must create a note. Our typical format is the SOAP note. Our typical format is the SOAP note. Our typical format is the SOAP note that has a couple different formats note that has a couple different formats note that has a couple different formats like what's the chief complaint of the like what's the chief complaint of the like what's the chief complaint of the patient and a few other sections and patient and a few other sections and patient and a few other sections and then what's the assessment plan? What do then what's the assessment plan? What do then what's the assessment plan? What do we do with this patient? You have to do we do with this patient? You have to do we do with this patient? You have to do it right this after every single visit it right this after every single visit it right this after every single visit and there's some different variations on and there's some different variations on and there's some different variations on this depending on specialty. this depending on specialty. this depending on specialty. Typically clinicians end up often doing Typically clinicians end up often doing Typically clinicians end up often doing it takes like 2 hours a day to write it takes like 2 hours a day to write it takes like 2 hours a day to write just write these notes and you often do just write these notes and you often do just write these notes and you often do it with what's known as pajama time it with what's known as pajama time it with what's known as pajama time after work itself. after work itself. after work itself. And that's a common source of clinician And that's a common source of clinician And that's a common source of clinician burnout spending all this time outside burnout spending all this time outside burnout spending all this time outside of work and it's not the most fun part of work and it's not the most fun part of work and it's not the most fun part of the job. However, these documents are of the job. However, these documents are of the job. However, these documents are actually extremely high stakes because actually extremely high stakes because actually extremely high stakes because these clinical notes are often used as a these clinical notes are often used as a these clinical notes are often used as a basis of billing, but also which is of basis of billing, but also which is of basis of billing, but also which is of course financial things are high stakes, course financial things are high stakes, course financial things are high stakes, but also have clinical impact and the but also have clinical impact and the but also have clinical impact and the reason for this is because reason for this is because reason for this is because these prior these notes are used for for these prior these notes are used for for these prior these notes are used for for next clinician or as you switch health next clinician or as you switch health next clinician or as you switch health systems. They use it they provide systems. They use it they provide systems. They use it they provide context to the clinician of the context to the clinician of the context to the clinician of the patient's longitudinal medical record.

patient's longitudinal medical record. patient's longitudinal medical record. So, it's actually really high stakes to So, it's actually really high stakes to So, it's actually really high stakes to get this right. get this right. get this right. Um Um Um we started here because we it's a known we started here because we it's a known we started here because we it's a known a pain point that's existed for many, a pain point that's existed for many, a pain point that's existed for many, many years. But finally, the many years. But finally, the many years. But finally, the technology's caught up to do really, technology's caught up to do really, technology's caught up to do really, really high-quality medical notes that's really high-quality medical notes that's really high-quality medical notes that's actually personalized to the clinician. actually personalized to the clinician. actually personalized to the clinician. This was an amazing wedge into health This was an amazing wedge into health This was an amazing wedge into health care industry, which has typically been care industry, which has typically been care industry, which has typically been technology reticent because it really it technology reticent because it really it technology reticent because it really it led to led to led to they actually care a lot about getting they actually care a lot about getting they actually care a lot about getting these notes high quality and right. It these notes high quality and right. It these notes high quality and right. It led to higher doctor satis- satis- led to higher doctor satis- satis- led to higher doctor satis- satis- provider satisfaction, and such they provider satisfaction, and such they provider satisfaction, and such they could actually see more patients. could actually see more patients. could actually see more patients. Um and it can actually help create a Um and it can actually help create a Um and it can actually help create a higher record um that helps prevent uh higher record um that helps prevent uh higher record um that helps prevent uh as it relates to billing, auditing, and as it relates to billing, auditing, and as it relates to billing, auditing, and other reasons. other reasons. other reasons. So, we started there. Just this product So, we started there. Just this product So, we started there. Just this product alone scaled to 300 hospital systems. alone scaled to 300 hospital systems. alone scaled to 300 hospital systems. But I want to show you a little about But I want to show you a little about But I want to show you a little about where we're going next. And um and the where we're going next. And um and the where we're going next. And um and the core thesis of the company is that every core thesis of the company is that every core thesis of the company is that every year in health care is everything is year in health care is everything is year in health care is everything is around the conversation. So, we started around the conversation. So, we started around the conversation. So, we started over here with the conversation to over here with the conversation to over here with the conversation to clinical note. clinical note. clinical note. Everything else is downstream of that.

Everything else is downstream of that. Everything else is downstream of that. Whether you it relates to billing, Whether you it relates to billing, Whether you it relates to billing, whether it relates to things like whether it relates to things like whether it relates to things like clinical trial matching, or whether it clinical trial matching, or whether it clinical trial matching, or whether it relates to clinical decision support. relates to clinical decision support. relates to clinical decision support. It's all about the conversation, that It's all about the conversation, that It's all about the conversation, that sacred sacred sacred doctor and patient conversation, and doctor and patient conversation, and doctor and patient conversation, and we've just built all this administrative we've just built all this administrative we've just built all this administrative machinery around that. But how can we machinery around that. But how can we machinery around that. But how can we bring it back to that conversation and bring it back to that conversation and bring it back to that conversation and actually automate some of that uh actually automate some of that uh actually automate some of that uh administrative machinery? administrative machinery? administrative machinery? So, to give you a tactical example of So, to give you a tactical example of So, to give you a tactical example of what this looks like, um and I'll play what this looks like, um and I'll play what this looks like, um and I'll play this video. Of where where we're going from here. Of where where we're going from here. >> We've been building a solution that >> We've been building a solution that >> We've been building a solution that allows the physician to interact with allows the physician to interact with allows the physician to interact with Abridge directly by using their voice. Abridge directly by using their voice. Abridge directly by using their voice. Hey Abridge, Hey Abridge, Hey Abridge, is Neaten eligible for any clinical is Neaten eligible for any clinical is Neaten eligible for any clinical trials? >> He may be eligible for the Abridge HF >> He may be eligible for the Abridge HF study. He remains symptomatic despite study. He remains symptomatic despite study. He remains symptomatic despite maximal therapy, and most screening maximal therapy, and most screening maximal therapy, and most screening criteria are already met. But, an criteria are already met. But, an criteria are already met. But, an updated echocardiogram is needed to updated echocardiogram is needed to updated echocardiogram is needed to confirm his ejection fraction and confirm his ejection fraction and confirm his ejection fraction and complete eligibility assessment.

complete eligibility assessment. complete eligibility assessment. >> All right. Please order that echo for >> All right. Please order that echo for >> All right. Please order that echo for him. him. him. >> Done. Confirmatory echo ordered. >> Done. Confirmatory echo ordered. >> Done. Confirmatory echo ordered. >> And then, when I'm done for the day, I >> And then, when I'm done for the day, I >> And then, when I'm done for the day, I can just ask Abridge, "Hey, Abridge, can can just ask Abridge, "Hey, Abridge, can can just ask Abridge, "Hey, Abridge, can you prepare my charts for tomorrow?" you prepare my charts for tomorrow?" you prepare my charts for tomorrow?" And Abridge is working for me. >> Pause right there. One of the things >> Pause right there. One of the things that you'll you'll notice is that we are that you'll you'll notice is that we are that you'll you'll notice is that we are thinking about how to revolutionize the thinking about how to revolutionize the thinking about how to revolutionize the entire visit for a clinician. From entire visit for a clinician. From entire visit for a clinician. From pre-visit, how pre-visit, how pre-visit, how earlier in the we have suggested earlier in the we have suggested earlier in the we have suggested discussion topics. Here's things that discussion topics. Here's things that discussion topics. Here's things that you can talk about with your patient, you can talk about with your patient, you can talk about with your patient, whether they're clinical or more billing whether they're clinical or more billing whether they're clinical or more billing related. related. related. We have after the visit, we actually We have after the visit, we actually We have after the visit, we actually create everything for you. The patient create everything for you. The patient create everything for you. The patient visit summary, the actual clinical note, visit summary, the actual clinical note, visit summary, the actual clinical note, and we actually penned orders as you and we actually penned orders as you and we actually penned orders as you might have seen. We're able to use might have seen. We're able to use might have seen. We're able to use We are able to do this all by reading We are able to do this all by reading We are able to do this all by reading all this context. We have access to all all this context. We have access to all all this context. We have access to all of the EHR context, so we know of the EHR context, so we know of the EHR context, so we know everything about the patient, the prior everything about the patient, the prior everything about the patient, the prior labs, the prior notes. We have access to labs, the prior notes. We have access to labs, the prior notes. We have access to the live conversation between the doctor the live conversation between the doctor the live conversation between the doctor and the patient. That's where the and the patient. That's where the and the patient. That's where the debugging happens in health care, where debugging happens in health care, where debugging happens in health care, where you learn about what the patient is you learn about what the patient is you learn about what the patient is facing now.

facing now. facing now. And then we have access to world's And then we have access to world's And then we have access to world's medical literature that we can ground medical literature that we can ground medical literature that we can ground and clinical guidelines that we can and clinical guidelines that we can and clinical guidelines that we can ground all of our work in. ground all of our work in. ground all of our work in. So, I want to I want to switch now, given I want to I want to switch now, given the context of where we're going as a as the context of where we're going as a as the context of where we're going as a as a product, I want to switch into some of a product, I want to switch into some of a product, I want to switch into some of the key technical and engineering the key technical and engineering the key technical and engineering problems we face and inspire you on some problems we face and inspire you on some problems we face and inspire you on some of the what I think are very much of the what I think are very much of the what I think are very much frontier AI challenges. frontier AI challenges. frontier AI challenges. So, if you've ever worked on a gentic So, if you've ever worked on a gentic So, if you've ever worked on a gentic product before, the regardless of product before, the regardless of product before, the regardless of vertical, vertical, vertical, um, three KPIs that tend to matter are um, three KPIs that tend to matter are um, three KPIs that tend to matter are quality and then latency and cost. quality and then latency and cost. quality and then latency and cost. In healthcare, I feel that we're In healthcare, I feel that we're In healthcare, I feel that we're actually playing on hard mode for all of actually playing on hard mode for all of actually playing on hard mode for all of these three KPIs. these three KPIs. these three KPIs. This is a high-stakes scenario, This is a high-stakes scenario, This is a high-stakes scenario, especially when you're doing something especially when you're doing something especially when you're doing something like clinical decision support. You have like clinical decision support. You have like clinical decision support. You have to be right because the downside is to be right because the downside is to be right because the downside is extremely high when you're wrong. When I extremely high when you're wrong. When I extremely high when you're wrong. When I used to work at Glynn, you know, while I used to work at Glynn, you know, while I used to work at Glynn, you know, while I loved that product, I could be wrong and loved that product, I could be wrong and loved that product, I could be wrong and it would have been fine. Maybe we it would have been fine. Maybe we it would have been fine. Maybe we answered a question incorrectly. But in answered a question incorrectly. But in answered a question incorrectly. But in healthcare, if we answer something healthcare, if we answer something healthcare, if we answer something incorrectly, there's actually incorrectly, there's actually incorrectly, there's actually consequences and we entirely lose our consequences and we entirely lose our consequences and we entirely lose our trust. So, quality needs to be trust. So, quality needs to be trust. So, quality needs to be absolutely high and I'll talk a little absolutely high and I'll talk a little absolutely high and I'll talk a little about how we keep that bar high.

about how we keep that bar high. about how we keep that bar high. And then latency and cost also really And then latency and cost also really And then latency and cost also really matter for us when you're live in the matter for us when you're live in the matter for us when you're live in the conversation. You can't, with latency, conversation. You can't, with latency, conversation. You can't, with latency, you can't act on information too late you can't act on information too late you can't act on information too late and you have to act on the also at the and you have to act on the also at the and you have to act on the also at the right time for it to be useful. And then right time for it to be useful. And then right time for it to be useful. And then finally, cost at the scale where we're finally, cost at the scale where we're finally, cost at the scale where we're doing this at. doing this at. doing this at. Um, and and as an interesting aside, it Um, and and as an interesting aside, it Um, and and as an interesting aside, it actually relates to our we have a motto actually relates to our we have a motto actually relates to our we have a motto inside the company that our goal is to inside the company that our goal is to inside the company that our goal is to save lives, save time, save money for save lives, save time, save money for save lives, save time, save money for uh, for the hospital system and for the uh, for the hospital system and for the uh, for the hospital system and for the healthcare industry as a whole. And it healthcare industry as a whole. And it healthcare industry as a whole. And it actually, I think it's funny that it actually, I think it's funny that it actually, I think it's funny that it really maps to the three KPIs that you really maps to the three KPIs that you really maps to the three KPIs that you care about in an A Gentic product. care about in an A Gentic product. care about in an A Gentic product. So, talking a little about quality, how So, talking a little about quality, how So, talking a little about quality, how do we keep that bar high? do we keep that bar high? do we keep that bar high? For us, we really treat evals as the For us, we really treat evals as the For us, we really treat evals as the operating system, the life's blood of operating system, the life's blood of operating system, the life's blood of the of the company. the of the company. the of the company. This starts from internal benchmarks and This starts from internal benchmarks and This starts from internal benchmarks and offline evaluation. Before we develop offline evaluation. Before we develop offline evaluation. Before we develop any product, you know, whether we're any product, you know, whether we're any product, you know, whether we're talking about clinical trial matching, talking about clinical trial matching, talking about clinical trial matching, clinical note, clinical decision clinical note, clinical decision clinical note, clinical decision support, coding, we start with a robust support, coding, we start with a robust support, coding, we start with a robust set of internal benchmarks.

set of internal benchmarks. set of internal benchmarks. This is pre-deployment and then we test This is pre-deployment and then we test This is pre-deployment and then we test that against, you know, things that that against, you know, things that that against, you know, things that we've actually seen in the wild. we've actually seen in the wild. we've actually seen in the wild. Then we all have a staged rollout. We Then we all have a staged rollout. We Then we all have a staged rollout. We know that we need to make contact with know that we need to make contact with know that we need to make contact with the reality reality. Not everything the reality reality. Not everything the reality reality. Not everything offline will perfectly represent what offline will perfectly represent what offline will perfectly represent what happens in practice. And so, we slowly happens in practice. And so, we slowly happens in practice. And so, we slowly roll it out. Maybe it starts with the roll it out. Maybe it starts with the roll it out. Maybe it starts with the alpha group of clinicians that we trust alpha group of clinicians that we trust alpha group of clinicians that we trust and under they understand the stakes. We and under they understand the stakes. We and under they understand the stakes. We rely to beta, maybe there's AB testing rely to beta, maybe there's AB testing rely to beta, maybe there's AB testing at scale. And then even after it's fully at scale. And then even after it's fully at scale. And then even after it's fully rolled out, you always need continual a rolled out, you always need continual a rolled out, you always need continual a monitor. Again, the stakes are really monitor. Again, the stakes are really monitor. Again, the stakes are really high and you cannot get away with just high and you cannot get away with just high and you cannot get away with just being like a prototype that you ship out being like a prototype that you ship out being like a prototype that you ship out there and be like, "Yeah, I mean I there and be like, "Yeah, I mean I there and be like, "Yeah, I mean I tested on a few cases and it works." tested on a few cases and it works." tested on a few cases and it works." How we do this is we always have expert How we do this is we always have expert How we do this is we always have expert calibrated LM judges. So, we have calibrated LM judges. So, we have calibrated LM judges. So, we have clinicians embedded throughout the clinicians embedded throughout the clinicians embedded throughout the entire company. The clinicians are are entire company. The clinicians are are entire company. The clinicians are are domain experts, but not all of us are domain experts, but not all of us are domain experts, but not all of us are clinicians. I'm not a clinician. So, how clinicians. I'm not a clinician. So, how clinicians. I'm not a clinician. So, how can we, the rest of the company, still can we, the rest of the company, still can we, the rest of the company, still move fast is by encoding that clinician move fast is by encoding that clinician move fast is by encoding that clinician judgement into LM judges. You know, I judgement into LM judges. You know, I judgement into LM judges. You know, I think a really great evaluation system think a really great evaluation system think a really great evaluation system has a property that it reflects the has a property that it reflects the has a property that it reflects the behaviors that you want in your product.

behaviors that you want in your product. behaviors that you want in your product. At the end of the day, we are making a At the end of the day, we are making a At the end of the day, we are making a product for clinicians and so who best product for clinicians and so who best product for clinicians and so who best other than our clinicians to actually other than our clinicians to actually other than our clinicians to actually create our judges that represent what create our judges that represent what create our judges that represent what they want. And those judges, once you they want. And those judges, once you they want. And those judges, once you have that, create a feedback loop so have that, create a feedback loop so have that, create a feedback loop so that anyone, whether you're a clinician that anyone, whether you're a clinician that anyone, whether you're a clinician or not, can actually or not, can actually or not, can actually uh hill climb and learn from that. uh hill climb and learn from that. uh hill climb and learn from that. We also have a lot of online signals, We also have a lot of online signals, We also have a lot of online signals, whether how you're editing the clinical whether how you're editing the clinical whether how you're editing the clinical note and your typical thumbs up, thumbs note and your typical thumbs up, thumbs note and your typical thumbs up, thumbs down, uh and star ratings and other down, uh and star ratings and other down, uh and star ratings and other free-form text. free-form text. free-form text. So, this is a general framework we use So, this is a general framework we use So, this is a general framework we use for our for all our products. I want to for our for all our products. I want to for our for all our products. I want to deep dive into the product that I work deep dive into the product that I work deep dive into the product that I work on, which is clinical decision support. on, which is clinical decision support. on, which is clinical decision support. So, to give you an example of a clinical So, to give you an example of a clinical So, to give you an example of a clinical decision support looks like, and decision support looks like, and decision support looks like, and specifically, we're building something specifically, we're building something specifically, we're building something novel, which is contextual clinical novel, which is contextual clinical novel, which is contextual clinical decision support. Maybe a provider asks decision support. Maybe a provider asks decision support. Maybe a provider asks a question like, "Hey, does this patient a question like, "Hey, does this patient a question like, "Hey, does this patient meet the criteria for febrile meet the criteria for febrile meet the criteria for febrile neutropenia?" Um and so, what we have to neutropenia?" Um and so, what we have to neutropenia?" Um and so, what we have to do here is actually a lot of context is do here is actually a lot of context is do here is actually a lot of context is under-specified in this question. Where under-specified in this question. Where under-specified in this question. Where So, the first thing we're going to do is So, the first thing we're going to do is So, the first thing we're going to do is we're actually going to pull from the we're actually going to pull from the we're actually going to pull from the EHR data, previous previous lab values.

EHR data, previous previous lab values. EHR data, previous previous lab values. Then, using that context, we're going to Then, using that context, we're going to Then, using that context, we're going to uh use the uh use the uh use the we're also going to use the live we're also going to use the live we're also going to use the live conversation and we're going to use conversation and we're going to use conversation and we're going to use clinical guidelines and medical journals clinical guidelines and medical journals clinical guidelines and medical journals to come use as the reasoning sources for to come use as the reasoning sources for to come use as the reasoning sources for combining all this context to actually combining all this context to actually combining all this context to actually answer the provider's question. answer the provider's question. answer the provider's question. Now, but I want to focus on evaluation. Now, but I want to focus on evaluation. Now, but I want to focus on evaluation. Again, the stakes are really high here. Again, the stakes are really high here. Again, the stakes are really high here. We we really can't get this wrong. So, We we really can't get this wrong. So, We we really can't get this wrong. So, how do you tell whether or not an answer how do you tell whether or not an answer how do you tell whether or not an answer is correct? And sometimes I I is correct? And sometimes I I is correct? And sometimes I I And this is a case where the generator And this is a case where the generator And this is a case where the generator and the verifier gap is really small. and the verifier gap is really small. and the verifier gap is really small. What I mean by this is in some problems What I mean by this is in some problems What I mean by this is in some problems in AI, such as like Sudoku, it's really in AI, such as like Sudoku, it's really in AI, such as like Sudoku, it's really really hard to generate a solution to really hard to generate a solution to really hard to generate a solution to Sudoku, but it's extremely easy to Sudoku, but it's extremely easy to Sudoku, but it's extremely easy to verify verify verify once you do have the solution. And that once you do have the solution. And that once you do have the solution. And that makes that makes it much easier to hill makes that makes it much easier to hill makes that makes it much easier to hill climb against and build evaluation for. climb against and build evaluation for. climb against and build evaluation for. But in a case like this, the generator But in a case like this, the generator But in a case like this, the generator and verifier gap is really small. If I and verifier gap is really small. If I and verifier gap is really small. If I had a really really good generator had a really really good generator had a really really good generator verifier, then that would just be my verifier, then that would just be my verifier, then that would just be my generator itself. So, how do I create a generator itself. So, how do I create a generator itself. So, how do I create a reference that isn't just a language reference that isn't just a language reference that isn't just a language model itself and ground itself to I have model itself and ground itself to I have model itself and ground itself to I have trust?

trust? trust? So, what we do is we tackle this by So, what we do is we tackle this by So, what we do is we tackle this by having many many different signals. having many many different signals. having many many different signals. We have a clinical quality judge, which We have a clinical quality judge, which We have a clinical quality judge, which I'm going to dive deep into, and then we I'm going to dive deep into, and then we I'm going to dive deep into, and then we tackle from we have many signals from a tackle from we have many signals from a tackle from we have many signals from a boundary and adversarial judge. We have boundary and adversarial judge. We have boundary and adversarial judge. We have a clinical safety judge. And we also a clinical safety judge. And we also a clinical safety judge. And we also have judges that represent product have judges that represent product have judges that represent product aspects like tone and style, and that aspects like tone and style, and that aspects like tone and style, and that matters a lot as well for AI products. matters a lot as well for AI products. matters a lot as well for AI products. So, all of these are different signals So, all of these are different signals So, all of these are different signals that try to get a piece of this like that try to get a piece of this like that try to get a piece of this like really really hard to measure problem really really hard to measure problem really really hard to measure problem and guarantee it in the way that we want and guarantee it in the way that we want and guarantee it in the way that we want the product to be. the product to be. the product to be. So, diving into the clinical quality So, diving into the clinical quality So, diving into the clinical quality judge. So, again, I said the generator judge. So, again, I said the generator judge. So, again, I said the generator verifier gap is really small here. So, verifier gap is really small here. So, verifier gap is really small here. So, what we need is we actually need human what we need is we actually need human what we need is we actually need human references to tell are we generating the references to tell are we generating the references to tell are we generating the right thing. But you can't just create a right thing. But you can't just create a right thing. But you can't just create a human golden response because there is a human golden response because there is a human golden response because there is a lot of variability in the potential lot of variability in the potential lot of variability in the potential responses. So, what we did is we took a responses. So, what we did is we took a responses. So, what we did is we took a lot of real clinical cases. We had lot of real clinical cases. We had lot of real clinical cases. We had independent physicians create a rubric. independent physicians create a rubric. independent physicians create a rubric. So, this rubric said elements of what we So, this rubric said elements of what we So, this rubric said elements of what we wanted in the response. So, it's not wanted in the response. So, it's not wanted in the response. So, it's not here's the exact response because again, here's the exact response because again, here's the exact response because again, there's many infinite possible there's many infinite possible there's many infinite possible responses, but a good rubric elements responses, but a good rubric elements responses, but a good rubric elements that what a good response would look that what a good response would look that what a good response would look like. And then we had a separate like. And then we had a separate like. And then we had a separate physician that actually adjudicated it, physician that actually adjudicated it, physician that actually adjudicated it, brought these two independent rubrics brought these two independent rubrics brought these two independent rubrics together, created a final rubric, and we together, created a final rubric, and we together, created a final rubric, and we actually had a fourth clinician do QA on actually had a fourth clinician do QA on actually had a fourth clinician do QA on these rubrics.

these rubrics. these rubrics. Once you have these rubrics, and here Once you have these rubrics, and here Once you have these rubrics, and here here's a sample rubric what it looks here's a sample rubric what it looks here's a sample rubric what it looks like. Here's actually a question and like. Here's actually a question and like. Here's actually a question and then there's more context in the case then there's more context in the case then there's more context in the case itself. You have a rubric of what are itself. You have a rubric of what are itself. You have a rubric of what are the elements that a response should look the elements that a response should look the elements that a response should look like. Now, we can actually have an LM like. Now, we can actually have an LM like. Now, we can actually have an LM judge that compares our agent's judge that compares our agent's judge that compares our agent's responses to these rubric elements and responses to these rubric elements and responses to these rubric elements and does some basic semantic match to tell, does some basic semantic match to tell, does some basic semantic match to tell, "Hey, is our model performing well?" As "Hey, is our model performing well?" As "Hey, is our model performing well?" As we continue to hill climb, whether it's we continue to hill climb, whether it's we continue to hill climb, whether it's our agent architecture, our models, or our agent architecture, our models, or our agent architecture, our models, or search ranking algorithms. I want to talk a little now about cost I want to talk a little now about cost and latency, two other really really and latency, two other really really and latency, two other really really hard problems for us. So, we do this as hard problems for us. So, we do this as hard problems for us. So, we do this as we said in that intro video, we do this we said in that intro video, we do this we said in that intro video, we do this on the live in the conversation, and we on the live in the conversation, and we on the live in the conversation, and we do on the run rate of 100 million do on the run rate of 100 million do on the run rate of 100 million medical conversations a year. How do we medical conversations a year. How do we medical conversations a year. How do we do this in a way that doesn't really do this in a way that doesn't really do this in a way that doesn't really break the bank for us? break the bank for us? break the bank for us? So, one of one place that this problem So, one of one place that this problem So, one of one place that this problem comes up comes up comes up is is actually in generating the is is actually in generating the is is actually in generating the clinical note. So, when you're clinical note. So, when you're clinical note. So, when you're generating the clinical note, there's generating the clinical note, there's generating the clinical note, there's many different sections to it. There's a many different sections to it. There's a many different sections to it. There's a history of present illness, past medical history of present illness, past medical history of present illness, past medical history, and there's the assessment and history, and there's the assessment and history, and there's the assessment and plan. So, one of the core insights for plan. So, one of the core insights for plan. So, one of the core insights for us is rather than say using a foundation us is rather than say using a foundation us is rather than say using a foundation model to generate all this, is we can model to generate all this, is we can model to generate all this, is we can actually break decompose this problem actually break decompose this problem actually break decompose this problem into simpler, smaller workflows.

into simpler, smaller workflows. into simpler, smaller workflows. Health care is actually many specific Health care is actually many specific Health care is actually many specific workflows. You don't need, you know, workflows. You don't need, you know, workflows. You don't need, you know, Fable 5 to actually solve all of your Fable 5 to actually solve all of your Fable 5 to actually solve all of your clinical notes. We we don't need clinical notes. We we don't need clinical notes. We we don't need frontier level intelligence for every frontier level intelligence for every frontier level intelligence for every problem. So, we actually post train a problem. So, we actually post train a problem. So, we actually post train a lot of smaller models for different lot of smaller models for different lot of smaller models for different problems, such as different actually problems, such as different actually problems, such as different actually even to the granularity of different even to the granularity of different even to the granularity of different sections in the clinical note. And that sections in the clinical note. And that sections in the clinical note. And that lets us use much smaller models because lets us use much smaller models because lets us use much smaller models because it's a more specific problem and at at it's a more specific problem and at at it's a more specific problem and at at much cheaper cost and latency. much cheaper cost and latency. much cheaper cost and latency. And we have this data flywheel that we And we have this data flywheel that we And we have this data flywheel that we have this unique data set of a hundred have this unique data set of a hundred have this unique data set of a hundred million medical conversations a year. million medical conversations a year. million medical conversations a year. And as far as we know, no one else has And as far as we know, no one else has And as far as we know, no one else has such a such a such a large data set. So, our key insight is large data set. So, our key insight is large data set. So, our key insight is having a right to win in training having a right to win in training having a right to win in training models. There problems where the quality models. There problems where the quality models. There problems where the quality is already maxed out. And so, you should is already maxed out. And so, you should is already maxed out. And so, you should train models then to reduce quality and train models then to reduce quality and train models then to reduce quality and latency. But there are other problems latency. But there are other problems latency. But there are other problems where the quality isn't maxed out and where the quality isn't maxed out and where the quality isn't maxed out and people say, "Oh, the frontier model people say, "Oh, the frontier model people say, "Oh, the frontier model would just steamroll you." Our key would just steamroll you." Our key would just steamroll you." Our key insight is we can actually potentially insight is we can actually potentially insight is we can actually potentially beat the rate of change on the frontier beat the rate of change on the frontier beat the rate of change on the frontier model if we have the right to win by model if we have the right to win by model if we have the right to win by having the right data that they may not having the right data that they may not having the right data that they may not have and the focus on a problem that have and the focus on a problem that have and the focus on a problem that they may not be focusing on. And that they may not be focusing on. And that they may not be focusing on. And that lets us still maximize quality.

lets us still maximize quality. lets us still maximize quality. Another problem that I'll quickly touch Another problem that I'll quickly touch Another problem that I'll quickly touch on is in-visit orders. So, on is in-visit orders. So, on is in-visit orders. So, doctors really doctors really doctors really aren't a big fans of pending orders, but aren't a big fans of pending orders, but aren't a big fans of pending orders, but often they'll mention orders during the often they'll mention orders during the often they'll mention orders during the visit itself, medication or visit itself, medication or visit itself, medication or non-medication orders. So, what we have non-medication orders. So, what we have non-medication orders. So, what we have is while we're listening in the visit, is while we're listening in the visit, is while we're listening in the visit, as the clinician says order, we actually as the clinician says order, we actually as the clinician says order, we actually queue it up in the background queue it up in the background queue it up in the background and let them actually sign it off in the and let them actually sign it off in the and let them actually sign it off in the EHR. But you can imagine if we did this EHR. But you can imagine if we did this EHR. But you can imagine if we did this in a very naive way, like every few in a very naive way, like every few in a very naive way, like every few seconds are just listening for orders, seconds are just listening for orders, seconds are just listening for orders, that would really break the bank. that would really break the bank. that would really break the bank. And so, And so, And so, a lot of our tricks are like, how do we a lot of our tricks are like, how do we a lot of our tricks are like, how do we find the right events in the find the right events in the find the right events in the conversation to actually trigger heavier conversation to actually trigger heavier conversation to actually trigger heavier models that will actually do the order models that will actually do the order models that will actually do the order matching because you need to match the matching because you need to match the matching because you need to match the order, not just is the order said, but order, not just is the order said, but order, not just is the order said, but does it match and references orders that does it match and references orders that does it match and references orders that are approved by the system and are are approved by the system and are are approved by the system and are relevant to the conversation. So, we relevant to the conversation. So, we relevant to the conversation. So, we have a number of different gates that have a number of different gates that have a number of different gates that are cheaper and faster that let us are cheaper and faster that let us are cheaper and faster that let us trigger actually larger models and hand trigger actually larger models and hand trigger actually larger models and hand off to them for actually doing the off to them for actually doing the off to them for actually doing the end-to-end work. But the last message, this was of course But the last message, this was of course a very quick talk, but the message I a very quick talk, but the message I a very quick talk, but the message I want to leave you with is healthcare is want to leave you with is healthcare is want to leave you with is healthcare is a domain that needs frontier AI and a domain that needs frontier AI and a domain that needs frontier AI and actually puts it to the test at high actually puts it to the test at high actually puts it to the test at high stakes. In the past, I as an engineer stakes. In the past, I as an engineer stakes. In the past, I as an engineer myself was worried about working in myself was worried about working in myself was worried about working in healthcare. Does healthcare. Does healthcare. Does like does healthcare technology actually like does healthcare technology actually like does healthcare technology actually work? Well, Elbridge has proven this at work? Well, Elbridge has proven this at work? Well, Elbridge has proven this at scale for for scale for for scale for for And I as hopefully I gave you a taste of And I as hopefully I gave you a taste of And I as hopefully I gave you a taste of some of the frontier problems that we some of the frontier problems that we some of the frontier problems that we work on.

work on. work on. So, thank you so much. My name is So, thank you so much. My name is So, thank you so much. My name is Chaitanya again. You can and feel free Chaitanya again. You can and feel free Chaitanya again. You can and feel free to connect with me at Twitter or to connect with me at Twitter or to connect with me at Twitter or LinkedIn. Thank you. LinkedIn. Thank you. LinkedIn. Thank you. >> [applause]