Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health

>> Hello and good morning. Chetana gave us >> Hello and good morning. Chetana gave us a great overview of you know what a a great overview of you know what a a great overview of you know what a bridge does. bridge does. bridge does. Today I'm here to talk more from a Today I'm here to talk more from a Today I'm here to talk more from a practitioner's view of you know how we practitioner's view of you know how we practitioner's view of you know how we are building health care AI within Hinge are building health care AI within Hinge are building health care AI within Hinge Health. So hi, I'm Rashi Agarwal. I lead Health. So hi, I'm Rashi Agarwal. I lead Health. So hi, I'm Rashi Agarwal. I lead AI and ML at Hinge Health and today I AI and ML at Hinge Health and today I AI and ML at Hinge Health and today I will be talking about guardrails that will be talking about guardrails that will be talking about guardrails that are needed to build member-facing health are needed to build member-facing health are needed to build member-facing health care AI. I want to talk a little bit about the I want to talk a little bit about the state of health care AI right now. We do state of health care AI right now. We do state of health care AI right now. We do have a lot of frontier models which are have a lot of frontier models which are have a lot of frontier models which are running and believe it or not, 40 running and believe it or not, 40 running and believe it or not, 40 million people actually use these models million people actually use these models million people actually use these models for triaging their health care issues. for triaging their health care issues. for triaging their health care issues. But there is a caveat and these are some But there is a caveat and these are some But there is a caveat and these are some of the headlines that have been of the headlines that have been of the headlines that have been happening in the past few in the past happening in the past few in the past happening in the past few in the past one year or so. one year or so. one year or so. Poisoned by a chatbot. Poisoned by a chatbot. Poisoned by a chatbot. Let's start with this one person. A Let's start with this one person. A Let's start with this one person. A 60-year-old healthy man asked a popular 60-year-old healthy man asked a popular 60-year-old healthy man asked a popular AI assistant how to cut salt from his AI assistant how to cut salt from his AI assistant how to cut salt from his diet. diet. diet. The LLM told him to swap it with bromine The LLM told him to swap it with bromine The LLM told him to swap it with bromine sodium bromide.

sodium bromide. sodium bromide. He did it for 3 months. He did it for 3 months. He did it for 3 months. He landed in the ER with paranoia and He landed in the ER with paranoia and He landed in the ER with paranoia and hallucinations. hallucinations. hallucinations. Bromide levels 200 times the safe limit. Bromide levels 200 times the safe limit. Bromide levels 200 times the safe limit. 3 weeks in the hospital. 3 weeks in the hospital. 3 weeks in the hospital. For what? For following diet advice? For what? For following diet advice? For what? For following diet advice? Let's look at another pattern. Let's look at another pattern. Let's look at another pattern. The first independent safety test of a The first independent safety test of a The first independent safety test of a consumer health AI out of Mount Sinai consumer health AI out of Mount Sinai consumer health AI out of Mount Sinai found that this health AI is under found that this health AI is under found that this health AI is under triaging life-threatening emergency 50% triaging life-threatening emergency 50% triaging life-threatening emergency 50% of the times. of the times. of the times. Diabetic ketoacidosis, respiratory Diabetic ketoacidosis, respiratory Diabetic ketoacidosis, respiratory failure. failure. failure. And it told the people to go see a And it told the people to go see a And it told the people to go see a doctor in a day or two. doctor in a day or two. doctor in a day or two. The right answer was ER right now. The right answer was ER right now. The right answer was ER right now. And this is in French. And this is in French. And this is in French. In February, ECRI In February, ECRI In February, ECRI the patient safety group that hospitals the patient safety group that hospitals the patient safety group that hospitals trust to rank their top risks trust to rank their top risks trust to rank their top risks named AI chatbot misuse as the number named AI chatbot misuse as the number named AI chatbot misuse as the number one health technology hazard of 2026.

one health technology hazard of 2026. one health technology hazard of 2026. Number one on the list that they publish Number one on the list that they publish Number one on the list that they publish every year. every year. every year. So, this is not This is not really a So, this is not This is not really a So, this is not This is not really a frontier problem. frontier problem. frontier problem. This is the production baseline that This is the production baseline that This is the production baseline that we're working with right now. we're working with right now. we're working with right now. So, the question comes, how do you ship So, the question comes, how do you ship So, the question comes, how do you ship AI to somebody who's already trusted you AI to somebody who's already trusted you AI to somebody who's already trusted you with their health? with their health? with their health? The next 20 minutes are all about that. It starts with three non-negotiable It starts with three non-negotiable foundations. foundations. foundations. One, One, One, the constraint is the architecture. the constraint is the architecture. the constraint is the architecture. Most AI safety failures in health care Most AI safety failures in health care Most AI safety failures in health care are not model failures. are not model failures. are not model failures. They are architectural decisions that They are architectural decisions that They are architectural decisions that were made before even a single token was were made before even a single token was were made before even a single token was generated. generated. generated. Two, Two, Two, deterministic rules belong above the deterministic rules belong above the deterministic rules belong above the model, not inside it. model, not inside it. model, not inside it. What can never be wrong What can never be wrong What can never be wrong cannot be left to probability. cannot be left to probability. cannot be left to probability. And three, And three, And three, safety is a continuous evaluation layer, safety is a continuous evaluation layer, safety is a continuous evaluation layer, not a one-time gate. not a one-time gate. not a one-time gate. Launch of your product is where the real Launch of your product is where the real Launch of your product is where the real risk starts, not where it ends.

risk starts, not where it ends. risk starts, not where it ends. That's the first part of what I want to That's the first part of what I want to That's the first part of what I want to talk about today. talk about today. talk about today. The second half is what happens when the The second half is what happens when the The second half is what happens when the architecture is not enough architecture is not enough architecture is not enough and a human has to make a decision of and a human has to make a decision of and a human has to make a decision of what ships versus what holds. what ships versus what holds. what ships versus what holds. >> [snorts] >> [snorts] >> [snorts] >> Let's start with layer one. >> Let's start with layer one. >> Let's start with layer one. Protecting PHI takes both Protecting PHI takes both Protecting PHI takes both policy and architecture. policy and architecture. policy and architecture. Policy tells you what to protect and Policy tells you what to protect and Policy tells you what to protect and architecture makes sure that it actually architecture makes sure that it actually architecture makes sure that it actually happens. happens. happens. The first thing that changes when you The first thing that changes when you The first thing that changes when you start shipping member facing health AI start shipping member facing health AI start shipping member facing health AI is where PHI lives. is where PHI lives. is where PHI lives. Most teams treat PHI as a runtime Most teams treat PHI as a runtime Most teams treat PHI as a runtime problem. problem. problem. Something to redact when a log gets Something to redact when a log gets Something to redact when a log gets written to a dashboard. written to a dashboard. written to a dashboard. That's the reactive version. That's the reactive version. That's the reactive version. The architecture version strips PHI at The architecture version strips PHI at The architecture version strips PHI at the pipeline boundary. At ingestion the pipeline boundary. At ingestion the pipeline boundary. At ingestion before it ever reaches the data lake. before it ever reaches the data lake. before it ever reaches the data lake. By the time the data is stored, the PHI By the time the data is stored, the PHI By the time the data is stored, the PHI is gone.

is gone. is gone. So, a developer opens a dashboard So, a developer opens a dashboard So, a developer opens a dashboard there's nothing to redact. there's nothing to redact. there's nothing to redact. The PHI was never there. The PHI was never there. The PHI was never there. The rest of the architecture works in a The rest of the architecture works in a The rest of the architecture works in a similar way. similar way. similar way. Production and non-production stay Production and non-production stay Production and non-production stay completely separate. No pipes in between completely separate. No pipes in between completely separate. No pipes in between because even a single pipe is all that because even a single pipe is all that because even a single pipe is all that it takes for member data to leak into a it takes for member data to leak into a it takes for member data to leak into a dev environment. And HIPAA laws are very dev environment. And HIPAA laws are very dev environment. And HIPAA laws are very stringent, especially in health care. stringent, especially in health care. stringent, especially in health care. You know, the regulatory bar is much, You know, the regulatory bar is much, You know, the regulatory bar is much, much higher. So, you have to be very much higher. So, you have to be very much higher. So, you have to be very careful about the architecture that careful about the architecture that careful about the architecture that you're designing. you're designing. you're designing. Another big thing access depends on two Another big thing access depends on two Another big thing access depends on two things. Your role and your geographic things. Your role and your geographic things. Your role and your geographic region. We all work with the teams which region. We all work with the teams which region. We all work with the teams which are geographically distributed, but not are geographically distributed, but not are geographically distributed, but not everybody has access to PHI. That is a everybody has access to PHI. That is a everybody has access to PHI. That is a certification, a policy that is applied certification, a policy that is applied certification, a policy that is applied to specific regions only. to specific regions only. to specific regions only. An engineer outside the regulated region An engineer outside the regulated region An engineer outside the regulated region cannot reach raw PHI at all. cannot reach raw PHI at all. cannot reach raw PHI at all. And the compliance rules, HIPAA, FDA's And the compliance rules, HIPAA, FDA's And the compliance rules, HIPAA, FDA's good machine learning practice, state good machine learning practice, state good machine learning practice, state laws like Texas, Triaga, they are not laws like Texas, Triaga, they are not laws like Texas, Triaga, they are not afterthoughts.

afterthoughts. afterthoughts. They are the grounding input in how you They are the grounding input in how you They are the grounding input in how you actually design your systems. actually design your systems. actually design your systems. You cannot slap on HIPAA on top of, you You cannot slap on HIPAA on top of, you You cannot slap on HIPAA on top of, you know, an underlying system or an know, an underlying system or an know, an underlying system or an architecture. architecture. architecture. You start with it and let the You start with it and let the You start with it and let the architecture grow around it. architecture grow around it. architecture grow around it. When PHI is protected at the When PHI is protected at the When PHI is protected at the architecture level, you're not just architecture level, you're not just architecture level, you're not just trusting that the policies will get trusting that the policies will get trusting that the policies will get followed. followed. followed. You're actually relying on a system You're actually relying on a system You're actually relying on a system that's incapable of certain failures. Let's move to the layer two. Let's move to the layer two. Probabilistic systems are great at Probabilistic systems are great at Probabilistic systems are great at generation. We all know that. generation. We all know that. generation. We all know that. However, they are unreliable for things However, they are unreliable for things However, they are unreliable for things that cannot can never be wrong. So, the that cannot can never be wrong. So, the that cannot can never be wrong. So, the rule is very simple. rule is very simple. rule is very simple. Must not fail behavior belongs above Must not fail behavior belongs above Must not fail behavior belongs above your prompt, above the model. your prompt, above the model. your prompt, above the model. And what does above the prompt actually And what does above the prompt actually And what does above the prompt actually mean? mean? mean? It means that there is a code layer that It means that there is a code layer that It means that there is a code layer that runs first runs first runs first on every turn before the model even on every turn before the model even on every turn before the model even runs.

runs. runs. The code layer is what makes your The code layer is what makes your The code layer is what makes your irreversible decisions. The decision of, irreversible decisions. The decision of, irreversible decisions. The decision of, you know, whether this is an emergency you know, whether this is an emergency you know, whether this is an emergency escalation, should they be routed to escalation, should they be routed to escalation, should they be routed to 911, should a clinician step into the 911, should a clinician step into the 911, should a clinician step into the loop. All of those are irreversible loop. All of those are irreversible loop. All of those are irreversible decisions which need to lie at a decisions which need to lie at a decisions which need to lie at a deterministic code level layer. deterministic code level layer. deterministic code level layer. The model handles the long tail of your The model handles the long tail of your The model handles the long tail of your conversations and interactions with your conversations and interactions with your conversations and interactions with your members. The picture to hold in your head is a The picture to hold in your head is a stack. stack. stack. Code on top, Code on top, Code on top, model below. model below. model below. Every turn goes through the code layer Every turn goes through the code layer Every turn goes through the code layer first. Most turns do reach the model, but the Most turns do reach the model, but the model never gets a vote on high stake model never gets a vote on high stake model never gets a vote on high stake calls. Here's how you can think about it in a Here's how you can think about it in a different way. different way. different way. A model is not a guardrail. A model is not a guardrail. A model is not a guardrail. A model with a system prompt is also not A model with a system prompt is also not A model with a system prompt is also not a guardrail. a guardrail. a guardrail. Code that runs above the model is Code that runs above the model is Code that runs above the model is closer. closer. closer. Even the labs that be build these Even the labs that be build these Even the labs that be build these frontier models publish the authority frontier models publish the authority frontier models publish the authority hierarchy.

hierarchy. hierarchy. Root, system, developer, user, Root, system, developer, user, Root, system, developer, user, guideline. guideline. guideline. Every layer above user is one prompt Every layer above user is one prompt Every layer above user is one prompt injection away from being overridden. injection away from being overridden. injection away from being overridden. If the labs themselves don't trust the If the labs themselves don't trust the If the labs themselves don't trust the prompt as a security boundary, neither prompt as a security boundary, neither prompt as a security boundary, neither should you. >> [snorts] >> [snorts] >> So, what does live in this code layer? >> So, what does live in this code layer? >> So, what does live in this code layer? Let's examine it a little bit. Let's Let's examine it a little bit. Let's Let's examine it a little bit. Let's take three examples. take three examples. take three examples. First, very very relevant to health First, very very relevant to health First, very very relevant to health care, which is emergency escalation. care, which is emergency escalation. care, which is emergency escalation. If a member mentions self-harm, suicidal If a member mentions self-harm, suicidal If a member mentions self-harm, suicidal ideation, or an acute medical emergency, ideation, or an acute medical emergency, ideation, or an acute medical emergency, the system must route to 911 or 988. The the system must route to 911 or 988. The the system must route to 911 or 988. The model should not even see this turn. model should not even see this turn. model should not even see this turn. Code runs first, decides, and routes, Code runs first, decides, and routes, Code runs first, decides, and routes, and makes a decision right away. and makes a decision right away. and makes a decision right away. Another example, intent routing. Another example, intent routing. Another example, intent routing. Which capability in your underlying Which capability in your underlying Which capability in your underlying multi-agentic system, multi-agentic multi-agentic system, multi-agentic multi-agentic system, multi-agentic architecture handles a conversation architecture handles a conversation architecture handles a conversation turn?

turn? turn? Is it clinical? Is it tech support? Is Is it clinical? Is it tech support? Is Is it clinical? Is it tech support? Is it education from the millions of, you it education from the millions of, you it education from the millions of, you know, accredited articles? Is it know, accredited articles? Is it know, accredited articles? Is it exercise recommendation? exercise recommendation? exercise recommendation? The model can help to classify, but The model can help to classify, but The model can help to classify, but high-stakes path must must again take a high-stakes path must must again take a high-stakes path must must again take a deterministic route deterministic route deterministic route at the top itself. at the top itself. at the top itself. You You don't want like a clinical You You don't want like a clinical You You don't want like a clinical question quietly being routed to your question quietly being routed to your question quietly being routed to your generic tech support agent. That's generic tech support agent. That's generic tech support agent. That's unrecoverable. Third, identity verification. Third, identity verification. Anything that touches member data has to Anything that touches member data has to Anything that touches member data has to check that the right member is at the check that the right member is at the check that the right member is at the other end. other end. other end. That's an authentication check. And That's an authentication check. And That's an authentication check. And authentication is a security bound authentication is a security bound authentication is a security bound boundary. Prompts are not. boundary. Prompts are not. boundary. Prompts are not. The underlying pattern across all three, The underlying pattern across all three, The underlying pattern across all three, code runs first. Code makes the code runs first. Code makes the code runs first. Code makes the irreversible decisions. The model irreversible decisions. The model irreversible decisions. The model handles what's left.

Last but not least, layer three. Last but not least, layer three. As we all know, safety is not a gate you As we all know, safety is not a gate you As we all know, safety is not a gate you pass once. It is a continuous layer that pass once. It is a continuous layer that pass once. It is a continuous layer that runs the whole time. runs the whole time. runs the whole time. Most teams treat evals as a pre-launch Most teams treat evals as a pre-launch Most teams treat evals as a pre-launch checklist. checklist. checklist. You run your tests, you ship, you move You run your tests, you ship, you move You run your tests, you ship, you move on. on. on. That's necessary, of course, but that's That's necessary, of course, but that's That's necessary, of course, but that's hardly enough. hardly enough. hardly enough. What actually holds up in production is What actually holds up in production is What actually holds up in production is judges that continuously keep scoring judges that continuously keep scoring judges that continuously keep scoring real conversations as they happen. real conversations as they happen. real conversations as they happen. Not a saved golden data set. Not a saved golden data set. Not a saved golden data set. Live traffic. Live traffic. Live traffic. Scored on a lot of dimensions all the Scored on a lot of dimensions all the Scored on a lot of dimensions all the time. time. time. These signals come from three sources, These signals come from three sources, These signals come from three sources, and each one catches something and each one catches something and each one catches something different. different. different. First, automated judges. 30, 40, name First, automated judges. 30, 40, name First, automated judges. 30, 40, name it, you know, as as much as you can it, you know, as as much as you can it, you know, as as much as you can scale. Automated judges with multiple scale. Automated judges with multiple scale. Automated judges with multiple dimensions, always refreshing. dimensions, always refreshing. dimensions, always refreshing. Clinical accuracy, safety, escalation, Clinical accuracy, safety, escalation, Clinical accuracy, safety, escalation, relevance, drift, refusal, etc., etc. I relevance, drift, refusal, etc., etc. I relevance, drift, refusal, etc., etc. I can keep going on. But you you get the can keep going on. But you you get the can keep going on. But you you get the point.

point. point. These are the automated judges that are These are the automated judges that are These are the automated judges that are always going to catch regressions and always going to catch regressions and always going to catch regressions and any even sensitive drops in quality. any even sensitive drops in quality. any even sensitive drops in quality. Second, Second, Second, your gold mine of information. That's your gold mine of information. That's your gold mine of information. That's going to be member feedback. going to be member feedback. going to be member feedback. Thumbs up, thumbs down on each and every Thumbs up, thumbs down on each and every Thumbs up, thumbs down on each and every single message. That's the truth signal. single message. That's the truth signal. single message. That's the truth signal. That's your member communicating with That's your member communicating with That's your member communicating with you. you. you. And it's the only one that comes And it's the only one that comes And it's the only one that comes straight from the person that you're straight from the person that you're straight from the person that you're serving it to. serving it to. serving it to. It catches tone problems and things that It catches tone problems and things that It catches tone problems and things that judges miss. judges miss. judges miss. >> [snorts] >> [snorts] >> [snorts] >> Third, sample traces. Random samples >> Third, sample traces. Random samples >> Third, sample traces. Random samples spread across capabilities with spread across capabilities with spread across capabilities with high-stake cases checked every single high-stake cases checked every single high-stake cases checked every single time. 100% sampling on those. time. 100% sampling on those. time. 100% sampling on those. Ultimately, people need to read these Ultimately, people need to read these Ultimately, people need to read these signals. People are going to catch what no single People are going to catch what no single metric is going to catch. metric is going to catch. metric is going to catch. And here's the part that nobody really And here's the part that nobody really And here's the part that nobody really warns you about. The bottleneck is not warns you about. The bottleneck is not warns you about. The bottleneck is not the compute, the models, the capability.

the compute, the models, the capability. the compute, the models, the capability. It's actually having enough people to It's actually having enough people to It's actually having enough people to read the signal and act on it. One more thing about layer three. One more thing about layer three. Some failures you can't just prompt Some failures you can't just prompt Some failures you can't just prompt away. away. away. You ship the fix, it comes back under You ship the fix, it comes back under You ship the fix, it comes back under new conditions. new conditions. new conditions. New prompts, new tools, the model New prompts, new tools, the model New prompts, new tools, the model shifts. You ship the fix again. shifts. You ship the fix again. shifts. You ship the fix again. Each round buys you less and less. Each round buys you less and less. Each round buys you less and less. The rate never hits zero. The rate never hits zero. The rate never hits zero. At this point, monitoring is not a last At this point, monitoring is not a last At this point, monitoring is not a last resort. It is the first resort which is resort. It is the first resort which is resort. It is the first resort which is always on. always on. always on. A new failure that you see in production A new failure that you see in production A new failure that you see in production simply means you now have a new judge. simply means you now have a new judge. simply means you now have a new judge. Your underlying architecture and your Your underlying architecture and your Your underlying architecture and your system needs to be able to keep scaling system needs to be able to keep scaling system needs to be able to keep scaling with new judges, new monitoring as you with new judges, new monitoring as you with new judges, new monitoring as you keep scaling your, you know, consumers. keep scaling your, you know, consumers. keep scaling your, you know, consumers. And [snorts] that's the point. And [snorts] that's the point. And [snorts] that's the point. Monitoring is how you know that the Monitoring is how you know that the Monitoring is how you know that the architecture is still holding. architecture is still holding. architecture is still holding. >> [snorts] >> But monitoring also tells you when the >> But monitoring also tells you when the architecture is not enough.

architecture is not enough. architecture is not enough. And when the architecture is not enough, And when the architecture is not enough, And when the architecture is not enough, a human has to decide. a human has to decide. a human has to decide. And this is the second part of my talk And this is the second part of my talk And this is the second part of my talk where I want to focus on the decisioning where I want to focus on the decisioning where I want to focus on the decisioning frameworks. frameworks. frameworks. >> [snorts] >> [snorts] >> [snorts] >> Let's take an example. You're about to >> Let's take an example. You're about to >> Let's take an example. You're about to ship, you know, uh consumer AI again in ship, you know, uh consumer AI again in ship, you know, uh consumer AI again in the healthcare space and you have a the healthcare space and you have a the healthcare space and you have a feature, a specific capability that feature, a specific capability that feature, a specific capability that you're about to launch. And there is one you're about to launch. And there is one you're about to launch. And there is one issue left on the board 5 days before issue left on the board 5 days before issue left on the board 5 days before your launch. your launch. your launch. And you have multiple different And you have multiple different And you have multiple different stakeholders. stakeholders. stakeholders. Five stakeholders look at the same Five stakeholders look at the same Five stakeholders look at the same issue. issue. issue. Each one sees a different risk. Each one sees a different risk. Each one sees a different risk. And they don't agree what to do about And they don't agree what to do about And they don't agree what to do about it. it. it. Clinical sees member safety risk. They Clinical sees member safety risk. They Clinical sees member safety risk. They want to hold the launch. want to hold the launch. want to hold the launch. Legal sees regulatory exposure. Legal sees regulatory exposure. Legal sees regulatory exposure. Compliance sees audit risk. Compliance sees audit risk. Compliance sees audit risk. Product sees adoption risk. Product sees adoption risk. Product sees adoption risk. The If the If it ships broken, the The If the If it ships broken, the The If the If it ships broken, the feature won't land. feature won't land. feature won't land. And engineering sees velocity risk. And engineering sees velocity risk. And engineering sees velocity risk. They can't fix it without slipping the They can't fix it without slipping the They can't fix it without slipping the date they want to ship.

date they want to ship. date they want to ship. Five rational people, five different Five rational people, five different Five rational people, five different risks, and five very different fixes. risks, and five very different fixes. risks, and five very different fixes. So, what do you do? Do you hold the So, what do you do? Do you hold the So, what do you do? Do you hold the launch and fix, or do you actually ship? launch and fix, or do you actually ship? launch and fix, or do you actually ship? The next slide is the framework I The next slide is the framework I The next slide is the framework I actually use for making these decisions. actually use for making these decisions. actually use for making these decisions. Five rules. Five rules. Five rules. This is how I think about decisions when This is how I think about decisions when This is how I think about decisions when stakeholders disagree. stakeholders disagree. stakeholders disagree. Rule one. Rule one. Rule one. Worst case always wins. Worst case always wins. Worst case always wins. Severity is set by the worst pos- Severity is set by the worst pos- Severity is set by the worst pos- plausible outcome, not the average. And plausible outcome, not the average. And plausible outcome, not the average. And this is extremely relevant in health this is extremely relevant in health this is extremely relevant in health care. care. care. A bug that lightly annoys 100% of users A bug that lightly annoys 100% of users A bug that lightly annoys 100% of users is way less severe than one that could is way less severe than one that could is way less severe than one that could cause serious harm in 0.1% cause serious harm in 0.1% cause serious harm in 0.1% of cases. This is non-negotiable. of cases. This is non-negotiable. of cases. This is non-negotiable. The worst case matters more than the The worst case matters more than the The worst case matters more than the average case, always. average case, always. average case, always. So, when you're triaging, don't ask, So, when you're triaging, don't ask, So, when you're triaging, don't ask, "How often does this happen?"

"How often does this happen?" "How often does this happen?" Ask, Ask, Ask, "What's the worst version of this?" That "What's the worst version of this?" That "What's the worst version of this?" That sets the severity. sets the severity. sets the severity. Rule two. Rule two. Rule two. Severity is not capacity. This one keeps Severity is not capacity. This one keeps Severity is not capacity. This one keeps politics out of it. politics out of it. politics out of it. As we all know, as we ship features, As we all know, as we ship features, As we all know, as we ship features, there's always a little bit of there's always a little bit of there's always a little bit of contention between timelines, features, contention between timelines, features, contention between timelines, features, deliverables. deliverables. deliverables. But, a bug's severity comes from the But, a bug's severity comes from the But, a bug's severity comes from the harm that it causes. harm that it causes. harm that it causes. Not who owns it, not whether your team Not who owns it, not whether your team Not who owns it, not whether your team has the capacity to fix it, not how hard has the capacity to fix it, not how hard has the capacity to fix it, not how hard the fix is. the fix is. the fix is. You have three options in front of you You have three options in front of you You have three options in front of you at this point. Fix, delay the launch, at this point. Fix, delay the launch, at this point. Fix, delay the launch, or accept the risk with explicit or accept the risk with explicit or accept the risk with explicit sign-off. Those are the three. sign-off. Those are the three. sign-off. Those are the three. You never quietly downgrade a bug just You never quietly downgrade a bug just You never quietly downgrade a bug just because you can't get to it. Rule three, Rule three, asymmetric default. asymmetric default. asymmetric default. When you don't know what to do, always When you don't know what to do, always When you don't know what to do, always pick the safer mistake. pick the safer mistake. pick the safer mistake. And there are two spectrums to it. One And there are two spectrums to it. One And there are two spectrums to it. One is safety bugs and polish. The other is safety bugs and polish. The other is safety bugs and polish. The other side is polish bugs.

side is polish bugs. side is polish bugs. For safety bugs, the math is one-sided. For safety bugs, the math is one-sided. For safety bugs, the math is one-sided. Shipping a real safety bug is much worse Shipping a real safety bug is much worse Shipping a real safety bug is much worse than delaying for a false alarm. than delaying for a false alarm. than delaying for a false alarm. So, for safety bugs, when you're not So, for safety bugs, when you're not So, for safety bugs, when you're not sure, always hold and fix. sure, always hold and fix. sure, always hold and fix. On the other side, for polish bugs, the On the other side, for polish bugs, the On the other side, for polish bugs, the math runs the other way. math runs the other way. math runs the other way. Delaying a launch Delaying a launch Delaying a launch costs more than shipping a small flaw. costs more than shipping a small flaw. costs more than shipping a small flaw. So, when you're not sure, ship in case So, when you're not sure, ship in case So, when you're not sure, ship in case of polish bugs. of polish bugs. of polish bugs. Ultimately, the framework doesn't decide Ultimately, the framework doesn't decide Ultimately, the framework doesn't decide for you. It just tells you which way to for you. It just tells you which way to for you. It just tells you which way to lean. lean. lean. Rule four, Rule four, Rule four, revealed risk tolerance, revealed risk tolerance, revealed risk tolerance, not stated risk tolerance. not stated risk tolerance. not stated risk tolerance. Your launch bar is what your org already Your launch bar is what your org already Your launch bar is what your org already accepts in production, accepts in production, accepts in production, not what it says it will accept. not what it says it will accept. not what it says it will accept. If a behavior has been live in your If a behavior has been live in your If a behavior has been live in your existing product for weeks, months existing product for weeks, months existing product for weeks, months without escalation, without member without escalation, without member without escalation, without member complaints, without leadership concern, complaints, without leadership concern, complaints, without leadership concern, you cannot you cannot call it a launch you cannot you cannot call it a launch you cannot you cannot call it a launch blocker just for a new thing.

blocker just for a new thing. blocker just for a new thing. Your stated risk tolerance might be no Your stated risk tolerance might be no Your stated risk tolerance might be no bugs in production, bugs in production, bugs in production, but your revealed risk tolerance is but your revealed risk tolerance is but your revealed risk tolerance is what's actually shipping today. what's actually shipping today. what's actually shipping today. Calibrate to the revealed one. That's Calibrate to the revealed one. That's Calibrate to the revealed one. That's the floor. Rule five, Rule five, humans are the constraint. humans are the constraint. humans are the constraint. Judges scale Judges scale Judges scale pattern interpretation doesn't. pattern interpretation doesn't. pattern interpretation doesn't. Always always design for human in the Always always design for human in the Always always design for human in the loop. loop. loop. Judges code traces automatically. Judges code traces automatically. Judges code traces automatically. Dashboards refresh every few hours. Dashboards refresh every few hours. Dashboards refresh every few hours. None of that is hard anymore. None of that is hard anymore. None of that is hard anymore. But what's hard is having enough people But what's hard is having enough people But what's hard is having enough people to read the signal and act on it. to read the signal and act on it. to read the signal and act on it. One more piece around this. One more piece around this. One more piece around this. Fast follows are committed debt, not an Fast follows are committed debt, not an Fast follows are committed debt, not an optional backlog. optional backlog. optional backlog. If you didn't ship it at launch, it's If you didn't ship it at launch, it's If you didn't ship it at launch, it's not a wish list item. It's already not a wish list item. It's already not a wish list item. It's already committed. The five rules tell you how to decide, The five rules tell you how to decide, but they all assume one thing, but they all assume one thing, but they all assume one thing, that your underlying signal is true.

that your underlying signal is true. that your underlying signal is true. So here's the discipline that needs to So here's the discipline that needs to So here's the discipline that needs to come first. come first. come first. In a non-deterministic system, the judge In a non-deterministic system, the judge In a non-deterministic system, the judge is also non-deterministic. is also non-deterministic. is also non-deterministic. Before you trust the score, verify the Before you trust the score, verify the Before you trust the score, verify the scorer. scorer. scorer. And here's what it looks like in And here's what it looks like in And here's what it looks like in practice. practice. practice. Say you're watching a clinical accuracy Say you're watching a clinical accuracy Say you're watching a clinical accuracy judge in production. judge in production. judge in production. The score has been steady for at 4.9 for The score has been steady for at 4.9 for The score has been steady for at 4.9 for weeks. weeks. weeks. Today, it drops to 4.5. And tomorrow, it Today, it drops to 4.5. And tomorrow, it Today, it drops to 4.5. And tomorrow, it stays at 4.5. stays at 4.5. stays at 4.5. The immediate instinct is, let's start The immediate instinct is, let's start The immediate instinct is, let's start changing the prompts. The agent is changing the prompts. The agent is changing the prompts. The agent is broken. Let's fix the agent. That's broken. Let's fix the agent. That's broken. Let's fix the agent. That's reactive, reactive, reactive, and it's risky. and it's risky. and it's risky. You fix one thing and you break another. You fix one thing and you break another. You fix one thing and you break another. Worse, you're changing the agent based Worse, you're changing the agent based Worse, you're changing the agent based on a signal that might not be true. on a signal that might not be true. on a signal that might not be true. And the discipline needs to be And the discipline needs to be And the discipline needs to be different. different. different. First, ask whether the judge is right. First, ask whether the judge is right. First, ask whether the judge is right. We can solidify that with a with a We can solidify that with a with a We can solidify that with a with a concrete example.

concrete example. concrete example. Uh let's take it side by side. In Uh let's take it side by side. In Uh let's take it side by side. In scenario A, scenario A, scenario A, same question, member asks about same question, member asks about same question, member asks about caffeine. The agent gives FDA standard caffeine. The agent gives FDA standard caffeine. The agent gives FDA standard guidance. 400 mg for most adults, less guidance. 400 mg for most adults, less guidance. 400 mg for most adults, less if pregnant or on certain medications. if pregnant or on certain medications. if pregnant or on certain medications. The judge flags it as a hallucination. The judge flags it as a hallucination. The judge flags it as a hallucination. Because the agent mentioned pregnancy Because the agent mentioned pregnancy Because the agent mentioned pregnancy and medications without checking. and medications without checking. and medications without checking. But that's just clinical context. But that's just clinical context. But that's just clinical context. The judge is over calling in this case. The judge is over calling in this case. The judge is over calling in this case. Fix the judge in this scenario. Fix the judge in this scenario. Fix the judge in this scenario. For the same question, scenario B, For the same question, scenario B, For the same question, scenario B, the agent says 1,000 mg a day is fine. the agent says 1,000 mg a day is fine. the agent says 1,000 mg a day is fine. That's well above the safety limits. That's well above the safety limits. That's well above the safety limits. The judge correctly flags it. The judge correctly flags it. The judge correctly flags it. And the agent is wrong. In this case, And the agent is wrong. In this case, And the agent is wrong. In this case, fix the agent. fix the agent. fix the agent. The rule is always ask, is the judge The rule is always ask, is the judge The rule is always ask, is the judge right before changing the agent's right before changing the agent's right before changing the agent's response? response? response? Fixing a judge prompt is not cheating. Fixing a judge prompt is not cheating. Fixing a judge prompt is not cheating. Judges are software, too. And they need Judges are software, too. And they need Judges are software, too. And they need to continuously evolve.

to continuously evolve. to continuously evolve. This is what production discipline looks This is what production discipline looks This is what production discipline looks like when the system is not like when the system is not like when the system is not deterministic. Here's the whole talk in one slide. If Here's the whole talk in one slide. If you screenshot one thing, this would be you screenshot one thing, this would be you screenshot one thing, this would be it. Six takeaways, three from it. Six takeaways, three from it. Six takeaways, three from architecture, three from decisioning. architecture, three from decisioning. architecture, three from decisioning. On the architecture side, the pattern is On the architecture side, the pattern is On the architecture side, the pattern is very simple. Don't X what you can Y. very simple. Don't X what you can Y. very simple. Don't X what you can Y. Don't policy what you can architect. Don't policy what you can architect. Don't policy what you can architect. Don't prompt what you can code. Don't prompt what you can code. Don't prompt what you can code. Don't gate what you can monitor. Don't gate what you can monitor. Don't gate what you can monitor. On the decisioning side, the pattern is On the decisioning side, the pattern is On the decisioning side, the pattern is how humans decide when the system how humans decide when the system how humans decide when the system cannot. cannot. cannot. Score by the worst case and default to Score by the worst case and default to Score by the worst case and default to the safer mistake. the safer mistake. the safer mistake. Calibrate to your org. Calibrate to your org. Calibrate to your org. And always design for the human in the And always design for the human in the And always design for the human in the loop. loop. loop. Fast follows are debt, not backlog. Yes, building guardrails first is slower Yes, building guardrails first is slower than bolting them on later. than bolting them on later. than bolting them on later. But that's the design, not limitation.

But that's the design, not limitation. But that's the design, not limitation. We are not building a generic low-stakes We are not building a generic low-stakes We are not building a generic low-stakes chatbot. chatbot. chatbot. We are building a system that has to be We are building a system that has to be We are building a system that has to be worthy of someone's health. worthy of someone's health. worthy of someone's health. The architecture is how, The architecture is how, The architecture is how, the decisioning is when, the decisioning is when, the decisioning is when, and member trust is why. and member trust is why. and member trust is why. Thank you. Let's continue the Thank you. Let's continue the Thank you. Let's continue the conversation on LinkedIn. Thank you.