Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo

>> This is a clinical note an AI wrote from >> This is a clinical note an AI wrote from a real consultation. a real consultation. a real consultation. Take a few seconds and read it. Take a few seconds and read it. Take a few seconds and read it. It reads like a routine headache. It reads like a routine headache. It reads like a routine headache. A new headache, likely tension type, A new headache, likely tension type, A new headache, likely tension type, take some paracetamol, come back if it take some paracetamol, come back if it take some paracetamol, come back if it doesn't settle. doesn't settle. doesn't settle. Looks completely fine, doesn't it? Looks completely fine, doesn't it? Looks completely fine, doesn't it? Here's what's missing. Here's what's missing. Here's what's missing. In the room, she also mentioned her jaw In the room, she also mentioned her jaw In the room, she also mentioned her jaw aches when she chews. A new headache, aches when she chews. A new headache, aches when she chews. A new headache, over 50 with jaw pain on chewing, over 50 with jaw pain on chewing, over 50 with jaw pain on chewing, that's giant cell arthritis. that's giant cell arthritis. that's giant cell arthritis. And untreated, it can take her sight And untreated, it can take her sight And untreated, it can take her sight within days. It's a same day start within days. It's a same day start within days. It's a same day start steroids now emergency. steroids now emergency. steroids now emergency. And that one line, it never made it into And that one line, it never made it into And that one line, it never made it into the note. the note. the note. On the page, it's a paracetamol On the page, it's a paracetamol On the page, it's a paracetamol headache. headache. headache. And nothing in the note is technically And nothing in the note is technically And nothing in the note is technically wrong. wrong. wrong. It's the dangerous part is what isn't It's the dangerous part is what isn't It's the dangerous part is what isn't there. there. there. And so that's what I'm going to talk And so that's what I'm going to talk And so that's what I'm going to talk about today. about today. about today. The dangerous failures are often the The dangerous failures are often the The dangerous failures are often the ones that actually look completely fine. ones that actually look completely fine. ones that actually look completely fine. Firstly, Firstly, Firstly, who am I? I'm Seb, medical doctor by who am I? I'm Seb, medical doctor by who am I? I'm Seb, medical doctor by background, and now I'm Composure, where background, and now I'm Composure, where background, and now I'm Composure, where we build AI evaluation systems for we build AI evaluation systems for we build AI evaluation systems for high-stakes domains.

So, that one was a subtle kind of error, So, that one was a subtle kind of error, but sometimes it's not subtle at all. but sometimes it's not subtle at all. but sometimes it's not subtle at all. A man in his 20s sees his GP for a sore A man in his 20s sees his GP for a sore A man in his 20s sees his GP for a sore throat, tonsillitis. throat, tonsillitis. throat, tonsillitis. The AI writes that up. It gets him chest The AI writes that up. It gets him chest The AI writes that up. It gets him chest pain, suspected angina, diabetes pain, suspected angina, diabetes pain, suspected angina, diabetes medications he's never taken, and an medications he's never taken, and an medications he's never taken, and an address for a hospital that doesn't address for a hospital that doesn't address for a hospital that doesn't exist. exist. exist. And I I really like the LLM for this And I I really like the LLM for this And I I really like the LLM for this one. I think it's it's a good attempt at one. I think it's it's a good attempt at one. I think it's it's a good attempt at hospital name. Um hospital name. Um hospital name. Um and weeks later, he's invited to and weeks later, he's invited to and weeks later, he's invited to diabetic eye screening for diabetes he diabetic eye screening for diabetes he diabetic eye screening for diabetes he doesn't have. doesn't have. doesn't have. That's genuinely a real case that That's genuinely a real case that That's genuinely a real case that happened recently. happened recently. happened recently. Obviously, these kind of crazy ones Obviously, these kind of crazy ones Obviously, these kind of crazy ones someone notices, but it's those quiet someone notices, but it's those quiet someone notices, but it's those quiet ones that sit in the record uncalled ones that sit in the record uncalled ones that sit in the record uncalled that are the most challenging that are the most challenging that are the most challenging and can actually do a lot more damage. and can actually do a lot more damage. and can actually do a lot more damage. And they're not rare at all. In the And they're not rare at all. In the And they're not rare at all. In the largest real-world study of these notes, largest real-world study of these notes, largest real-world study of these notes, about 1 in 20 carried an error that was about 1 in 20 carried an error that was about 1 in 20 carried an error that was serious enough that it could cause serious enough that it could cause serious enough that it could cause significant harm to the patient. significant harm to the patient. significant harm to the patient. 1 in 20. That's not theoretical in 1 in 20. That's not theoretical in 1 in 20. That's not theoretical in testing, that's in production on real testing, that's in production on real testing, that's in production on real patients.

patients. patients. And that's only the serious the ones. If And that's only the serious the ones. If And that's only the serious the ones. If you widen that lens to all errors, you widen that lens to all errors, you widen that lens to all errors, nearly 1 in 5 had an important omission nearly 1 in 5 had an important omission nearly 1 in 5 had an important omission and more than 1 in 10 had a and more than 1 in 10 had a and more than 1 in 10 had a hallucination. hallucination. hallucination. And AI is being deployed at scale across And AI is being deployed at scale across And AI is being deployed at scale across healthcare fast. Ambient Scribes are one healthcare fast. Ambient Scribes are one healthcare fast. Ambient Scribes are one of the leading cases, already in about a of the leading cases, already in about a of the leading cases, already in about a third of US practices and climbing. third of US practices and climbing. third of US practices and climbing. Physician AI use doubled last year and Physician AI use doubled last year and Physician AI use doubled last year and none of this is tracked. none of this is tracked. none of this is tracked. So, for most of these systems, there's So, for most of these systems, there's So, for most of these systems, there's no adverse event reporting at all. The no adverse event reporting at all. The no adverse event reporting at all. The errors never show up as incidents, they errors never show up as incidents, they errors never show up as incidents, they just sit in the record. just sit in the record. just sit in the record. So, So, So, errors this common that are going unseen errors this common that are going unseen errors this common that are going unseen is is quite hard for me to believe that is is quite hard for me to believe that is is quite hard for me to believe that it's not already affecting patients. it's not already affecting patients. it's not already affecting patients. It's not that we checked and it's fine, It's not that we checked and it's fine, It's not that we checked and it's fine, it's that we're flying blind. it's that we're flying blind. it's that we're flying blind. And And And this isn't just a healthcare problem, this isn't just a healthcare problem, this isn't just a healthcare problem, it's every high-stakes use of AI. it's every high-stakes use of AI. it's every high-stakes use of AI. Healthcare shows it more viscerally Healthcare shows it more viscerally Healthcare shows it more viscerally because here being confidently wrong because here being confidently wrong because here being confidently wrong can be life and death. can be life and death. can be life and death. But, everything I show you can map But, everything I show you can map But, everything I show you can map straight back onto other domains as straight back onto other domains as straight back onto other domains as well.

well. well. So, here's what I want to do. I'm going So, here's what I want to do. I'm going So, here's what I want to do. I'm going to show you what exactly is going wrong, to show you what exactly is going wrong, to show you what exactly is going wrong, why it's going wrong, why it's going wrong, why it's going wrong, why the systems we built to catch it why the systems we built to catch it why the systems we built to catch it don't work, and don't work, and don't work, and a suggestion at how maybe we can start a suggestion at how maybe we can start a suggestion at how maybe we can start to fix that. to fix that. to fix that. So, first, what's going wrong and why? So, first, what's going wrong and why? So, first, what's going wrong and why? So, So, So, LLMs are getting good, obviously. They LLMs are getting good, obviously. They LLMs are getting good, obviously. They don't make stupid mistakes anymore most don't make stupid mistakes anymore most don't make stupid mistakes anymore most of the time. So, it's not about dumb of the time. So, it's not about dumb of the time. So, it's not about dumb errors. errors. errors. Everything here came out of three of the Everything here came out of three of the Everything here came out of three of the best production Ambient scribes on the best production Ambient scribes on the best production Ambient scribes on the market. market. market. Ones that we all know. Ones that we all know. Ones that we all know. We generated a load of notes across them We generated a load of notes across them We generated a load of notes across them last week. And this is exactly what's last week. And this is exactly what's last week. And this is exactly what's going on right now. This is every going on right now. This is every going on right now. This is every failure we found. Each dot is an error failure we found. Each dot is an error failure we found. Each dot is an error colored by type. colored by type. colored by type. Left to right, how much it matters. Left to right, how much it matters. Left to right, how much it matters. Bottom to top, whether a strong Bottom to top, whether a strong Bottom to top, whether a strong automated check catches it. automated check catches it. automated check catches it. And that split is the point. And that split is the point. And that split is the point. A handful up top get caught. But almost A handful up top get caught. But almost A handful up top get caught. But almost everything sits below the line. The ones everything sits below the line. The ones everything sits below the line. The ones I care about most are these on the I care about most are these on the I care about most are these on the bottom right. The high stakes and missed bottom right. The high stakes and missed bottom right. The high stakes and missed ones.

ones. ones. Let me show you what a couple of those Let me show you what a couple of those Let me show you what a couple of those looks like. looks like. looks like. So, a woman comes in with a headache. So, a woman comes in with a headache. So, a woman comes in with a headache. Doctor asks, Doctor asks, Doctor asks, "Did it come on suddenly or build up "Did it come on suddenly or build up "Did it come on suddenly or build up gradually?" gradually?" gradually?" She says she doesn't know. It just She says she doesn't know. It just She says she doesn't know. It just happened. The note records that as happened. The note records that as happened. The note records that as abrupt sudden onset. abrupt sudden onset. abrupt sudden onset. And sudden onset is a red flag. You can And sudden onset is a red flag. You can And sudden onset is a red flag. You can see why it just happened could maybe be see why it just happened could maybe be see why it just happened could maybe be interpreted and inferred as abrupt interpreted and inferred as abrupt interpreted and inferred as abrupt onset. onset. onset. But, that's a feature that points to a But, that's a feature that points to a But, that's a feature that points to a bleed on the brain. She never said it. bleed on the brain. She never said it. bleed on the brain. She never said it. The model decided it. And now that one The model decided it. And now that one The model decided it. And now that one word drives the whole workup. word drives the whole workup. word drives the whole workup. Here's another. Here's another. Here's another. Doctor suggests running some tests. Doctor suggests running some tests. Doctor suggests running some tests. Patient says, "Can we just try try Patient says, "Can we just try try Patient says, "Can we just try try antibiotics instead?" antibiotics instead?" antibiotics instead?" They agree, hold off on the tests, They agree, hold off on the tests, They agree, hold off on the tests, treat, and see how it goes. treat, and see how it goes. treat, and see how it goes. Note records the opposite. Note records the opposite. Note records the opposite. Arrange tests today. Arrange tests today. Arrange tests today. It kept the plan that they talked out It kept the plan that they talked out It kept the plan that they talked out of, not the one they chose. Every line of, not the one they chose. Every line of, not the one they chose. Every line in the note reads fine because it's not in the note reads fine because it's not in the note reads fine because it's not really a hallucination at all. It's not really a hallucination at all. It's not really a hallucination at all. It's not wrong. It was there in the original.

wrong. It was there in the original. wrong. It was there in the original. But, it's just not what they ended up But, it's just not what they ended up But, it's just not what they ended up deciding. deciding. deciding. So, why are these happening? So, why are these happening? So, why are these happening? There's, you know, in ambient scribes, There's, you know, in ambient scribes, There's, you know, in ambient scribes, there's first transcription and then there's first transcription and then there's first transcription and then generation. generation. generation. A lot of it A lot of it A lot of it does happen on the transcription layer. does happen on the transcription layer. does happen on the transcription layer. It can be words misheard for their It can be words misheard for their It can be words misheard for their sound-alikes. So, Humalog heard Humulin. sound-alikes. So, Humalog heard Humulin. sound-alikes. So, Humalog heard Humulin. Two insulins on completely different Two insulins on completely different Two insulins on completely different timelines, so swapping them could crash timelines, so swapping them could crash timelines, so swapping them could crash a blood sugar. a blood sugar. a blood sugar. Hyperthyroidism becomes hypothyroidism, Hyperthyroidism becomes hypothyroidism, Hyperthyroidism becomes hypothyroidism, the opposite condition. the opposite condition. the opposite condition. Or a drop to no Or a drop to no Or a drop to no on uh no evidence of cancer that becomes on uh no evidence of cancer that becomes on uh no evidence of cancer that becomes evidence of cancer. evidence of cancer. evidence of cancer. So, these these are really hard So, these these are really hard So, these these are really hard problems, and they are common. problems, and they are common. problems, and they are common. Not the ones I'm going to focus on, Not the ones I'm going to focus on, Not the ones I'm going to focus on, because because because most of what goes wrong is actually even most of what goes wrong is actually even most of what goes wrong is actually even with a perfect transcript. with a perfect transcript. with a perfect transcript. It's the model reading the words It's the model reading the words It's the model reading the words correctly and still doing one of three correctly and still doing one of three correctly and still doing one of three things. things. things. Either it adds something that was never Either it adds something that was never Either it adds something that was never said, it changes something that was, or said, it changes something that was, or said, it changes something that was, or it omits something that should be there.

it omits something that should be there. it omits something that should be there. Now, the blatant version of each of Now, the blatant version of each of Now, the blatant version of each of these is is really easy to catch. The these is is really easy to catch. The these is is really easy to catch. The hard part in all three hard part in all three hard part in all three is is is the same. It's telling whether that the same. It's telling whether that the same. It's telling whether that thing that was added or changed or thing that was added or changed or thing that was added or changed or dropped actually matters. dropped actually matters. dropped actually matters. It's detecting that slight over It's detecting that slight over It's detecting that slight over inference versus the dangerous inference versus the dangerous inference versus the dangerous fabrication. The harmless rephrase fabrication. The harmless rephrase fabrication. The harmless rephrase versus the meaningful edit. versus the meaningful edit. versus the meaningful edit. A dropped line of small talk versus a A dropped line of small talk versus a A dropped line of small talk versus a dropped allergy. dropped allergy. dropped allergy. So, the ones that matter slip through So, the ones that matter slip through So, the ones that matter slip through along with all of the ones that don't. That call which different matters That call which different matters is taste, effectively. is taste, effectively. is taste, effectively. Not aesthetic taste, but essentially Not aesthetic taste, but essentially Not aesthetic taste, but essentially judgment. It's It's whether in this judgment. It's It's whether in this judgment. It's It's whether in this context a missed allergy might kill context a missed allergy might kill context a missed allergy might kill someone or is not important. someone or is not important. someone or is not important. And I think there's there's three And I think there's there's three And I think there's there's three properties that really matter about properties that really matter about properties that really matter about this. this. this. It's tacit, so your domain experts have It's tacit, so your domain experts have It's tacit, so your domain experts have it, but they can't fully write it down. it, but they can't fully write it down. it, but they can't fully write it down. It's contextual, so the same detail is It's contextual, so the same detail is It's contextual, so the same detail is critical in one note, noise in the next. critical in one note, noise in the next. critical in one note, noise in the next. And it's moving. The model changes, And it's moving. The model changes, And it's moving. The model changes, guidelines change, two good doctors guidelines change, two good doctors guidelines change, two good doctors disagree, different hospitals have disagree, different hospitals have disagree, different hospitals have different definitions. So, there's no different definitions. So, there's no different definitions. So, there's no fixed target to write down.

fixed target to write down. fixed target to write down. And so, the model knows the facts, And so, the model knows the facts, And so, the model knows the facts, ultimately. They're extraordinarily ultimately. They're extraordinarily ultimately. They're extraordinarily capable, but what they lack is a sense capable, but what they lack is a sense capable, but what they lack is a sense of what matters of what matters of what matters here, for this specific example. And here, for this specific example. And here, for this specific example. And that's why even brilliant models make that's why even brilliant models make that's why even brilliant models make these mistakes. these mistakes. these mistakes. So, So, So, one natural move, you're never going to one natural move, you're never going to one natural move, you're never going to make that generator perfect. Generator make that generator perfect. Generator make that generator perfect. Generator is cheap. Generation is cheap, so stop is cheap. Generation is cheap, so stop is cheap. Generation is cheap, so stop fixing it at the source. fixing it at the source. fixing it at the source. Let it write, put a checker after it, Let it write, put a checker after it, Let it write, put a checker after it, pass only what clears the bar. pass only what clears the bar. pass only what clears the bar. And that checker should be the easier And that checker should be the easier And that checker should be the easier job. The generator has to get everything job. The generator has to get everything job. The generator has to get everything right and it pay attention to lots of right and it pay attention to lots of right and it pay attention to lots of varying instructions. varying instructions. varying instructions. Whereas the checker only has to find the Whereas the checker only has to find the Whereas the checker only has to find the one thing that's wrong and just focus on one thing that's wrong and just focus on one thing that's wrong and just focus on that task. You can also give it more that task. You can also give it more that task. You can also give it more time, more tokens, the exact failure time, more tokens, the exact failure time, more tokens, the exact failure modes to hunt for. modes to hunt for. modes to hunt for. Evaluation should be easier than Evaluation should be easier than Evaluation should be easier than generation. generation. generation. It's the asymmetry of verification, It's the asymmetry of verification, It's the asymmetry of verification, verifies law. That's why AI is raced verifies law. That's why AI is raced verifies law. That's why AI is raced ahead anyway, you can cheaply check the ahead anyway, you can cheaply check the ahead anyway, you can cheaply check the answer, maths and code. answer, maths and code. answer, maths and code. And And And doing this is exactly what the best doing this is exactly what the best doing this is exactly what the best teams do. They put a lot of energy into teams do. They put a lot of energy into teams do. They put a lot of energy into evaluation. It starts with the gold evaluation. It starts with the gold evaluation. It starts with the gold standard, which is expert humans standard, which is expert humans standard, which is expert humans reviewing notes, which obviously works reviewing notes, which obviously works reviewing notes, which obviously works offline, but you can't put a human on offline, but you can't put a human on offline, but you can't put a human on every note in production.

every note in production. every note in production. So, they automate it. So, they automate it. So, they automate it. They build a They build a They build a serious system and serious system and serious system and some of the best versions of this that some of the best versions of this that some of the best versions of this that I've seen are I've seen are I've seen are you take the transcript and the note you take the transcript and the note you take the transcript and the note and context, put in front of the judge, and context, put in front of the judge, and context, put in front of the judge, a detailed rubric for faithfulness with a detailed rubric for faithfulness with a detailed rubric for faithfulness with worked, pass and fail examples. worked, pass and fail examples. worked, pass and fail examples. The rubric maybe auto-optimized with GPA The rubric maybe auto-optimized with GPA The rubric maybe auto-optimized with GPA or something like that. Maybe you have or something like that. Maybe you have or something like that. Maybe you have some deterministic NLP to sort of count some deterministic NLP to sort of count some deterministic NLP to sort of count up medical concepts that are differing up medical concepts that are differing up medical concepts that are differing between the two. between the two. between the two. That's a powerful system. That's a powerful system. That's a powerful system. And yet, I pulled all of those errors And yet, I pulled all of those errors And yet, I pulled all of those errors earlier earlier earlier out of Ambient Scribes in an afternoon. out of Ambient Scribes in an afternoon. out of Ambient Scribes in an afternoon. So, if the evaluation is this good, how So, if the evaluation is this good, how So, if the evaluation is this good, how are these errors still getting through? are these errors still getting through? are these errors still getting through? So, So, So, I built this system and ran those same I built this system and ran those same I built this system and ran those same notes through it. notes through it. notes through it. And it scored most of them fine. And it scored most of them fine. And it scored most of them fine. It flagged a handful of them It flagged a handful of them It flagged a handful of them and signed off the rest. and signed off the rest. and signed off the rest. >> [clears throat] >> [clears throat] >> [clears throat] >> But one in five of those clean passes >> But one in five of those clean passes >> But one in five of those clean passes still had some sort of serious error still had some sort of serious error still had some sort of serious error buried in it.

buried in it. buried in it. And often that was an omission. And often that was an omission. And often that was an omission. The things that should have been there The things that should have been there The things that should have been there and actually quietly weren't. and actually quietly weren't. and actually quietly weren't. And that's the best version of a judge And that's the best version of a judge And that's the best version of a judge I've seen in a lot of teams and it waved I've seen in a lot of teams and it waved I've seen in a lot of teams and it waved them through. them through. them through. Why did it do that? It's not stupid. Why did it do that? It's not stupid. Why did it do that? It's not stupid. It's a frontier model, serious It's a frontier model, serious It's a frontier model, serious engineering behind it, more than clever engineering behind it, more than clever engineering behind it, more than clever enough to read the whole encounter and enough to read the whole encounter and enough to read the whole encounter and catch every obvious error. And it's not catch every obvious error. And it's not catch every obvious error. And it's not blind, either. blind, either. blind, either. And and that's part of the trap. If you And and that's part of the trap. If you And and that's part of the trap. If you take a note that says start amoxicillin, take a note that says start amoxicillin, take a note that says start amoxicillin, when the real decision was actually to when the real decision was actually to when the real decision was actually to wait and see, wait and see, wait and see, it's faithful to the words, amoxicillin it's faithful to the words, amoxicillin it's faithful to the words, amoxicillin did come up, but it's a lie about the did come up, but it's a lie about the did come up, but it's a lie about the intent. intent. intent. A good judge might catch that, might. A good judge might catch that, might. A good judge might catch that, might. But whether it flags that versus the But whether it flags that versus the But whether it flags that versus the other other other doesn't have other things that it could doesn't have other things that it could doesn't have other things that it could comment on comment on comment on depends on it knowing what decision depends on it knowing what decision depends on it knowing what decision matters most. matters most. matters most. And so And so And so it's not blind, it just can't tell what it's not blind, it just can't tell what it's not blind, it just can't tell what counts, essentially. counts, essentially. counts, essentially. So the note passes confidently and you So the note passes confidently and you So the note passes confidently and you put a judge like that in front of your put a judge like that in front of your put a judge like that in front of your system, you've not added a safety net, system, you've not added a safety net, system, you've not added a safety net, you've added a second silent failure you've added a second silent failure you've added a second silent failure that just nods along with the first.

that just nods along with the first. that just nods along with the first. And here's the root of it. So in math or And here's the root of it. So in math or And here's the root of it. So in math or code, the verifier comes with free, a code, the verifier comes with free, a code, the verifier comes with free, a unit test, a compiler. unit test, a compiler. unit test, a compiler. But for is this note safe and complete, But for is this note safe and complete, But for is this note safe and complete, there's no unit test. there's no unit test. there's no unit test. You have to build the verifier yourself You have to build the verifier yourself You have to build the verifier yourself and verification is only easier than and verification is only easier than and verification is only easier than generation for the easy bit, i.e. spot generation for the easy bit, i.e. spot generation for the easy bit, i.e. spot the difference between transcription the difference between transcription the difference between transcription note. note. note. But that's not the hard bit. The hard But that's not the hard bit. The hard But that's not the hard bit. The hard bit is knowing of all those differences bit is knowing of all those differences bit is knowing of all those differences you've seen, which matter. And that's you've seen, which matter. And that's you've seen, which matter. And that's harder than writing that plausibly good harder than writing that plausibly good harder than writing that plausibly good general note in the first place. general note in the first place. general note in the first place. Because that standard of good was never Because that standard of good was never Because that standard of good was never written down anywhere that the judge can written down anywhere that the judge can written down anywhere that the judge can read it. read it. read it. A rubric that you pre-specify is only A rubric that you pre-specify is only A rubric that you pre-specify is only the taste you could write down. the taste you could write down. the taste you could write down. The taste that matters is the part that The taste that matters is the part that The taste that matters is the part that you couldn't. you couldn't. you couldn't. And so here's here's a bit more detail And so here's here's a bit more detail And so here's here's a bit more detail on what what matters looks like. on what what matters looks like. on what what matters looks like. Two patients, both with blood in their Two patients, both with blood in their Two patients, both with blood in their urine, both notes dropped the same kind urine, both notes dropped the same kind urine, both notes dropped the same kind of line where they'd been on holiday. of line where they'd been on holiday. of line where they'd been on holiday. One had been to France, the other to One had been to France, the other to One had been to France, the other to Lake Malawi. Same English emission, same Lake Malawi. Same English emission, same Lake Malawi. Same English emission, same shape, same mistake.

shape, same mistake. shape, same mistake. Well, not really, because blood in the Well, not really, because blood in the Well, not really, because blood in the urine obviously warranted away urine obviously warranted away urine obviously warranted away and you're going to have to investigate and you're going to have to investigate and you're going to have to investigate it, but the France trip is irrelevant. it, but the France trip is irrelevant. it, but the France trip is irrelevant. The Lake Malawi trip is the diagnosis. The Lake Malawi trip is the diagnosis. The Lake Malawi trip is the diagnosis. Fresh water in sub-Saharan Africa means Fresh water in sub-Saharan Africa means Fresh water in sub-Saharan Africa means schistosomiasis until proven otherwise schistosomiasis until proven otherwise schistosomiasis until proven otherwise and it completely changes what the and it completely changes what the and it completely changes what the management plan is. So that same dropped management plan is. So that same dropped management plan is. So that same dropped line in one note is pure noise, in the line in one note is pure noise, in the line in one note is pure noise, in the other it's the answer. other it's the answer. other it's the answer. And which one it is, you simply just And which one it is, you simply just And which one it is, you simply just can't write all of that down in advance. can't write all of that down in advance. can't write all of that down in advance. So So So if you can't write it down, you can't if you can't write it down, you can't if you can't write it down, you can't write taste down, how do you get that write taste down, how do you get that write taste down, how do you get that into your evaluator and your whole into your evaluator and your whole into your evaluator and your whole application system? application system? application system? Well, Well, Well, we've answered a version of this before. we've answered a version of this before. we've answered a version of this before. RLHF exists because you can't write the RLHF exists because you can't write the RLHF exists because you can't write the reward function for good. You learn it reward function for good. You learn it reward function for good. You learn it from examples by showing it. from examples by showing it. from examples by showing it. The only question is where you keep what The only question is where you keep what The only question is where you keep what you've learned. you've learned. you've learned. And there's three places. And there's three places. And there's three places. You can either specify it up front, you You can either specify it up front, you You can either specify it up front, you can can can stuff the prompt, write the perfect stuff the prompt, write the perfect stuff the prompt, write the perfect rubric. We just watched that fail rubric. We just watched that fail rubric. We just watched that fail essentially. essentially. essentially. You can bake into the weights, You can bake into the weights, You can bake into the weights, fine-tuning or continual learning, but fine-tuning or continual learning, but fine-tuning or continual learning, but for a standard that's still moving and a for a standard that's still moving and a for a standard that's still moving and a score that has to be explainable, score that has to be explainable, score that has to be explainable, the weights, I think, are the wrong the weights, I think, are the wrong the weights, I think, are the wrong place to keep that.

place to keep that. place to keep that. They go stale, they can't tell you why They go stale, they can't tell you why They go stale, they can't tell you why and and and you can't change them without a retrain. you can't change them without a retrain. you can't change them without a retrain. So there's the third option, which I'll So there's the third option, which I'll So there's the third option, which I'll show you, which is you essentially just show you, which is you essentially just show you, which is you essentially just keep the taste as the examples keep the taste as the examples keep the taste as the examples themselves. Past judgments, expert themselves. Past judgments, expert themselves. Past judgments, expert corrections, references, and for each corrections, references, and for each corrections, references, and for each output, you retrieve the ones that bear output, you retrieve the ones that bear output, you retrieve the ones that bear on it into the judges context, on it into the judges context, on it into the judges context, add one and it's live on the next call. add one and it's live on the next call. add one and it's live on the next call. You can point at exactly what moved the You can point at exactly what moved the You can point at exactly what moved the score. score. score. For this problem, it's both better and For this problem, it's both better and For this problem, it's both better and also cheaper to do. So, that's the way to do that is one So, that's the way to do that is one repeating loop, three steps. repeating loop, three steps. repeating loop, three steps. Discover the failure modes from real Discover the failure modes from real Discover the failure modes from real outputs, capture how your experts judge outputs, capture how your experts judge outputs, capture how your experts judge them, calibrate every output against them, calibrate every output against them, calibrate every output against that, and when the standard moves, the that, and when the standard moves, the that, and when the standard moves, the loop moves with it. loop moves with it. loop moves with it. So, in more detail, discover. You don't So, in more detail, discover. You don't So, in more detail, discover. You don't write that rubric in a vacuum. write that rubric in a vacuum. write that rubric in a vacuum. You have to put the system in production You have to put the system in production You have to put the system in production and look at the real outputs. and look at the real outputs. and look at the real outputs. Cluster what goes wrong and the failure Cluster what goes wrong and the failure Cluster what goes wrong and the failure modes surface on their own. You name modes surface on their own. You name modes surface on their own. You name them. them. them. This is your failure mode ontology. This is your failure mode ontology. This is your failure mode ontology. Discover from your data, not guess on a Discover from your data, not guess on a Discover from your data, not guess on a whiteboard.

whiteboard. whiteboard. And you can't shortcut it. The ways that And you can't shortcut it. The ways that And you can't shortcut it. The ways that a real system goes wrong are effectively a real system goes wrong are effectively a real system goes wrong are effectively unbounded and synthetic test cases only unbounded and synthetic test cases only unbounded and synthetic test cases only cover the failures you already imagined. cover the failures you already imagined. cover the failures you already imagined. The ones that hurt you are often the The ones that hurt you are often the The ones that hurt you are often the ones that you didn't. And you'll only ones that you didn't. And you'll only ones that you didn't. And you'll only find those in real outputs. find those in real outputs. find those in real outputs. So, this ontology is your map, what to So, this ontology is your map, what to So, this ontology is your map, what to capture judgment on, what to retrieve capture judgment on, what to retrieve capture judgment on, what to retrieve against, including the failures that you against, including the failures that you against, including the failures that you never thought to check for. never thought to check for. never thought to check for. After that, it's capture and then After that, it's capture and then After that, it's capture and then calibrate. So, those discovered modes, calibrate. So, those discovered modes, calibrate. So, those discovered modes, they're not a checklist that the judge they're not a checklist that the judge they're not a checklist that the judge runs, but they organize everything. What runs, but they organize everything. What runs, but they organize everything. What you What you ask your experts about, how you What you ask your experts about, how you What you ask your experts about, how you index the cases that you'll you index the cases that you'll you index the cases that you'll retrieve, retrieve, retrieve, and and and capturing is a simple part. You put real capturing is a simple part. You put real capturing is a simple part. You put real outputs in front of your experts. outputs in front of your experts. outputs in front of your experts. Clinicians spend a focused few hours Clinicians spend a focused few hours Clinicians spend a focused few hours leaving comments. A session doesn't have leaving comments. A session doesn't have leaving comments. A session doesn't have to be a month-long labeling project to to be a month-long labeling project to to be a month-long labeling project to start with. And you collect their start with. And you collect their start with. And you collect their judgment. judgment. judgment. Not just a score, but the reasoning and Not just a score, but the reasoning and Not just a score, but the reasoning and corrections. And over time, you build up corrections. And over time, you build up corrections. And over time, you build up that record of how your experts actually that record of how your experts actually that record of how your experts actually judge. judge. judge. You then calibrate. You then calibrate. You then calibrate. That's That's That's the the the generic part of this you can the the the generic part of this you can the the the generic part of this you can write down once easily. For example, be write down once easily. For example, be write down once easily. For example, be faithful or don't drop anything faithful or don't drop anything faithful or don't drop anything important.

important. important. But what you can't write down is what But what you can't write down is what But what you can't write down is what counts as a serious miss for this counts as a serious miss for this counts as a serious miss for this specific note. That's contextual. And it specific note. That's contextual. And it specific note. That's contextual. And it shifts from note to note. shifts from note to note. shifts from note to note. So, So, So, what we recommend is you assemble that what we recommend is you assemble that what we recommend is you assemble that on the fly. For each output, your on the fly. For each output, your on the fly. For each output, your judging agent pulls in everything that judging agent pulls in everything that judging agent pulls in everything that bears on this one case. It's memory of bears on this one case. It's memory of bears on this one case. It's memory of the most similar outputs the most similar outputs the most similar outputs that it's judged before and how they that it's judged before and how they that it's judged before and how they scored, the expert corrections that scored, the expert corrections that scored, the expert corrections that apply, the reference documents and apply, the reference documents and apply, the reference documents and guidelines. guidelines. guidelines. Just context engineering per output. Just context engineering per output. Just context engineering per output. And And And crucially not just one pre-specified crucially not just one pre-specified crucially not just one pre-specified rubric in a vacuum, and not a model that rubric in a vacuum, and not a model that rubric in a vacuum, and not a model that you have to retrain every week, you have to retrain every week, you have to retrain every week, but a full sort of case-specific but a full sort of case-specific but a full sort of case-specific standard assembled for this output. standard assembled for this output. standard assembled for this output. And it's a loop as well. And it's a loop as well. And it's a loop as well. Every output you judge, every Every output you judge, every Every output you judge, every correction, sharpens the next. correction, sharpens the next. correction, sharpens the next. And when a brand new failure mode And when a brand new failure mode And when a brand new failure mode appears, Discovery surface it, and it appears, Discovery surface it, and it appears, Discovery surface it, and it flows straight back in. flows straight back in. flows straight back in. And so, to make that a little bit more And so, to make that a little bit more And so, to make that a little bit more concrete, concrete, concrete, that headache that I opened with, the that headache that I opened with, the that headache that I opened with, the one that was really a possible blindness one that was really a possible blindness one that was really a possible blindness emergency, emergency, emergency, here's the kinds of things that you here's the kinds of things that you here's the kinds of things that you would want to pull in for that note.

would want to pull in for that note. would want to pull in for that note. The nearest cases that your experts have The nearest cases that your experts have The nearest cases that your experts have judged, judged, judged, not this exact patient, but the same not this exact patient, but the same not this exact patient, but the same shape, maybe a red flag filed as shape, maybe a red flag filed as shape, maybe a red flag filed as routine. routine. routine. Uh the corrections that apply, like a Uh the corrections that apply, like a Uh the corrections that apply, like a new headache over 50, um new headache over 50, um new headache over 50, um suggests something that you need to suggests something that you need to suggests something that you need to check red flags on, and some criteria check red flags on, and some criteria check red flags on, and some criteria and guidelines, and you pull all of that and guidelines, and you pull all of that and guidelines, and you pull all of that in. in. in. It hasn't memorized this case. It's a It hasn't memorized this case. It's a It hasn't memorized this case. It's a capable model. capable model. capable model. And handed the right context to reason And handed the right context to reason And handed the right context to reason from, held against that, the dropped red from, held against that, the dropped red from, held against that, the dropped red flag stands out. It was never actually flag stands out. It was never actually flag stands out. It was never actually hard to catch. It just didn't know what hard to catch. It just didn't know what hard to catch. It just didn't know what mattered. mattered. mattered. And so, if you take that same data set And so, if you take that same data set And so, if you take that same data set generated notes from the start and pass generated notes from the start and pass generated notes from the start and pass it through these three it through these three it through these three judging systems, judging systems, judging systems, the first, a strong off-the-shelf the first, a strong off-the-shelf the first, a strong off-the-shelf judge with a rubric frontier model, judge with a rubric frontier model, judge with a rubric frontier model, um um um it's better than a coin flip, but it it's better than a coin flip, but it it's better than a coin flip, but it misses most of what matters. misses most of what matters. misses most of what matters. The second, that sort of serious system The second, that sort of serious system The second, that sort of serious system that we talked about before, that we talked about before, that we talked about before, rubric, deeper, rubric, deeper, rubric, deeper, maybe some detona stick checks, better maybe some detona stick checks, better maybe some detona stick checks, better again, again, again, but still missing quite a lot of what but still missing quite a lot of what but still missing quite a lot of what counts.

counts. counts. The third, the judge running this loop, The third, the judge running this loop, The third, the judge running this loop, discovered failure modes, calibrated for discovered failure modes, calibrated for discovered failure modes, calibrated for output against what experts judged, output against what experts judged, output against what experts judged, is is is performing a lot better on this specific performing a lot better on this specific performing a lot better on this specific data set. data set. data set. Same notes. Same notes. Same notes. The only thing that changes is what the The only thing that changes is what the The only thing that changes is what the judge was shown. judge was shown. judge was shown. And the difference here, it's not more And the difference here, it's not more And the difference here, it's not more compute or a better prompt, it's that compute or a better prompt, it's that compute or a better prompt, it's that the first two fight taste and lose. They the first two fight taste and lose. They the first two fight taste and lose. They guess the criteria, they freeze one guess the criteria, they freeze one guess the criteria, they freeze one standard, and they go stale. standard, and they go stale. standard, and they go stale. This repeating evolving loop does the This repeating evolving loop does the This repeating evolving loop does the opposite. It discovers the modes, fits opposite. It discovers the modes, fits opposite. It discovers the modes, fits the standard to each mode, and keeps the standard to each mode, and keeps the standard to each mode, and keeps learning. learning. learning. So, you might not write chemical notes, So, you might not write chemical notes, So, you might not write chemical notes, but if you ship anything where being but if you ship anything where being but if you ship anything where being confidently wrong has a cost, the confidently wrong has a cost, the confidently wrong has a cost, the contract review that misses the clauses contract review that misses the clauses contract review that misses the clauses that change the deal, the support agent that change the deal, the support agent that change the deal, the support agent that promises a refund you don't offer, that promises a refund you don't offer, that promises a refund you don't offer, the same thing is true for all of those. the same thing is true for all of those. the same thing is true for all of those. It's watched, if at all, by a judge with It's watched, if at all, by a judge with It's watched, if at all, by a judge with no taste for what matters in your no taste for what matters in your no taste for what matters in your domain. So, domain. So, domain. So, three things. Discover your failure three things. Discover your failure three things. Discover your failure modes from real outputs, don't guess modes from real outputs, don't guess modes from real outputs, don't guess them. Capture your experts' judgment on them. Capture your experts' judgment on them. Capture your experts' judgment on them, the standard that they can't write them, the standard that they can't write them, the standard that they can't write down.

down. down. Calibrate every output against the cases Calibrate every output against the cases Calibrate every output against the cases that they've already judged, not a that they've already judged, not a that they've already judged, not a static rubric, not a retrained model. static rubric, not a retrained model. static rubric, not a retrained model. Then keep that loop running. Then keep that loop running. Then keep that loop running. And if you take one thing away, And if you take one thing away, And if you take one thing away, easiest place to start is your experts easiest place to start is your experts easiest place to start is your experts leaving free-form comments on real leaving free-form comments on real leaving free-form comments on real outputs. outputs. outputs. That's the real That's the raw material That's the real That's the raw material That's the real That's the raw material for everything else. for everything else. for everything else. Your judge can verify anything that you Your judge can verify anything that you Your judge can verify anything that you write down in advance, but the standard write down in advance, but the standard write down in advance, but the standard of good never could be. And so, stop of good never could be. And so, stop of good never could be. And so, stop trying to write it all down in advance, trying to write it all down in advance, trying to write it all down in advance, and just start capturing it case by and just start capturing it case by and just start capturing it case by case, and evolving it. case, and evolving it. case, and evolving it. That's why evaluation can't be a thing That's why evaluation can't be a thing That's why evaluation can't be a thing you build once and freeze. you build once and freeze. you build once and freeze. The standard it checks against doesn't The standard it checks against doesn't The standard it checks against doesn't exist on paper. exist on paper. exist on paper. It has to be discovered from real It has to be discovered from real It has to be discovered from real outputs captured from the people who outputs captured from the people who outputs captured from the people who hold it and kept alive as it moves. hold it and kept alive as it moves. hold it and kept alive as it moves. Evaluation isn't something you have, Evaluation isn't something you have, Evaluation isn't something you have, it's something that you do continuously it's something that you do continuously it's something that you do continuously over time. over time. over time. Thank you. Thank you. Thank you. >> [applause]