AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

Good afternoon everyone. Thanks for uh Good afternoon everyone. Thanks for uh coming for a post lunch uh talk. Always coming for a post lunch uh talk. Always coming for a post lunch uh talk. Always appreciate that. Um my name is Farup and appreciate that. Um my name is Farup and appreciate that. Um my name is Farup and here's my teammate Nachiket. Uh we are here's my teammate Nachiket. Uh we are here's my teammate Nachiket. Uh we are uh here behalf of the Door Dash Genai uh here behalf of the Door Dash Genai uh here behalf of the Door Dash Genai platform team. Um and we kind of wanted platform team. Um and we kind of wanted platform team. Um and we kind of wanted to share our eval journey. Uh it started to share our eval journey. Uh it started to share our eval journey. Uh it started as uh uh you know eval is another as uh uh you know eval is another as uh uh you know eval is another engineering thing but then it slowly we engineering thing but then it slowly we engineering thing but then it slowly we realized it evolved into a cross realized it evolved into a cross realized it evolved into a cross functional effort and we kind of want to functional effort and we kind of want to functional effort and we kind of want to share our story here. So what is this share our story here. So what is this share our story here. So what is this team? This team is a gen platform team. team? This team is a gen platform team. team? This team is a gen platform team. Uh we are a horizontal team that helps Uh we are a horizontal team that helps Uh we are a horizontal team that helps all other product teams. So product all other product teams. So product all other product teams. So product teams at Door Dash build on top of the teams at Door Dash build on top of the teams at Door Dash build on top of the infrastructure and the primitives that infrastructure and the primitives that infrastructure and the primitives that we provide. Um and we see our uh USP and we provide. Um and we see our uh USP and we provide. Um and we see our uh USP and the value that we provide is that we the value that we provide is that we the value that we provide is that we help product teams balance these three help product teams balance these three help product teams balance these three forces which is accuracy, latency and forces which is accuracy, latency and forces which is accuracy, latency and cost. Um initially we applied this in cost. Um initially we applied this in cost. Um initially we applied this in terms of models but if you think about terms of models but if you think about terms of models but if you think about it it also applies to agents. Um and the it it also applies to agents. Um and the it it also applies to agents. Um and the way we achieve this is we have uh way we achieve this is we have uh way we achieve this is we have uh primitives and building blocks. Um so primitives and building blocks. Um so primitives and building blocks. Um so for example we have an LLM gateway where for example we have an LLM gateway where for example we have an LLM gateway where you can easily switch between different you can easily switch between different you can easily switch between different models uh and try the latest and models uh and try the latest and models uh and try the latest and greatest. Uh we have an agent gateway greatest. Uh we have an agent gateway greatest. Uh we have an agent gateway where you can connect to tools uh and where you can connect to tools uh and where you can connect to tools uh and other agents uh and we help solve other agents uh and we help solve other agents uh and we help solve authentication uh agent identity and authentication uh agent identity and authentication uh agent identity and other things in a central place uh which other things in a central place uh which other things in a central place uh which our security team can bless. Um our security team can bless. Um our security team can bless. Um similarly we pair the LLM gateway with

similarly we pair the LLM gateway with similarly we pair the LLM gateway with open weights models hosting. Uh, of open weights models hosting. Uh, of open weights models hosting. Uh, of course cost is a number one concern course cost is a number one concern course cost is a number one concern these days. Uh, and we uh kind of these days. Uh, and we uh kind of these days. Uh, and we uh kind of invested in open weights models uh and invested in open weights models uh and invested in open weights models uh and have seen significant impact uh already. have seen significant impact uh already. have seen significant impact uh already. Um, and maybe we'll talk about that in a Um, and maybe we'll talk about that in a Um, and maybe we'll talk about that in a future conference. Uh, the fourth pillar future conference. Uh, the fourth pillar future conference. Uh, the fourth pillar is eval and that's the part that we is eval and that's the part that we is eval and that's the part that we would want to share today. Um would want to share today. Um would want to share today. Um when we started talking to product teams when we started talking to product teams when we started talking to product teams internally at Door Dash uh there were internally at Door Dash uh there were internally at Door Dash uh there were varying uh distinct needs across teams. varying uh distinct needs across teams. varying uh distinct needs across teams. We had a consumer discovery and shopping We had a consumer discovery and shopping We had a consumer discovery and shopping assistant team. Uh for those who assistant team. Uh for those who assistant team. Uh for those who attended Ragago talk earlier today uh attended Ragago talk earlier today uh attended Ragago talk earlier today uh you will uh see the need for session you will uh see the need for session you will uh see the need for session level quality judgments. um uh then the level quality judgments. um uh then the level quality judgments. um uh then the personalization ML then you needed uh a personalization ML then you needed uh a personalization ML then you needed uh a way to scale up human judgment and with way to scale up human judgment and with way to scale up human judgment and with multi- aent systems we needed trajectory multi- aent systems we needed trajectory multi- aent systems we needed trajectory based evals now the question is how do based evals now the question is how do based evals now the question is how do you cater to all these different needs you cater to all these different needs you cater to all these different needs under a common platform under a common platform under a common platform and uh as we spoke to these teams we and uh as we spoke to these teams we and uh as we spoke to these teams we realized like um we needed to empower realized like um we needed to empower realized like um we needed to empower the people who are the domain experts the people who are the domain experts the people who are the domain experts and in our case that was strategy and and in our case that was strategy and and in our case that was strategy and operations folks it as product managers operations folks it as product managers operations folks it as product managers uh it was even labeling partners uh and uh it was even labeling partners uh and uh it was even labeling partners uh and not only engineers so we kind of started not only engineers so we kind of started not only engineers so we kind of started with like okay we have to be UI first with like okay we have to be UI first with like okay we have to be UI first and this was the guidance we had from and this was the guidance we had from and this was the guidance we had from Andy Fang our co-founder as well um so Andy Fang our co-founder as well um so Andy Fang our co-founder as well um so we had UIs for non-engineers to we had UIs for non-engineers to we had UIs for non-engineers to contribute uh then we kind of evolved to

contribute uh then we kind of evolved to contribute uh then we kind of evolved to also being API first so that engineers also being API first so that engineers also being API first so that engineers can also build and not be blocked on the can also build and not be blocked on the can also build and not be blocked on the central platform and they can build central platform and they can build central platform and they can build their own uh uh uh systems uh and Then their own uh uh uh systems uh and Then their own uh uh uh systems uh and Then of course with the coding agents now we of course with the coding agents now we of course with the coding agents now we have become workflow first where we kind have become workflow first where we kind have become workflow first where we kind of empower SNO and PMs to also being of empower SNO and PMs to also being of empower SNO and PMs to also being able to uh navigate uh the platform and able to uh navigate uh the platform and able to uh navigate uh the platform and uh run operations as well. Um so with uh run operations as well. Um so with uh run operations as well. Um so with that context I'll hand it off to Nachig that context I'll hand it off to Nachig that context I'll hand it off to Nachig to talk about uh how we went about to talk about uh how we went about to talk about uh how we went about delivering this. Cool. Thanks Harup. Um delivering this. Cool. Thanks Harup. Um delivering this. Cool. Thanks Harup. Um and thanks everyone for joining us. I and thanks everyone for joining us. I and thanks everyone for joining us. I know France is playing right now and I know France is playing right now and I know France is playing right now and I promise you this will be better than promise you this will be better than promise you this will be better than that. I'm kidding. Um so as Faroo was that. I'm kidding. Um so as Faroo was that. I'm kidding. Um so as Faroo was saying uh Evals is not just an saying uh Evals is not just an saying uh Evals is not just an engineering harness it is a cross engineering harness it is a cross engineering harness it is a cross functional effort across different functional effort across different functional effort across different pillars across different uh teams uh pillars across different uh teams uh pillars across different uh teams uh that actually helps us add all the that actually helps us add all the that actually helps us add all the domain specific knowledge into our uh domain specific knowledge into our uh domain specific knowledge into our uh into the quality of the AI itself. So into the quality of the AI itself. So into the quality of the AI itself. So from your traces to your data sets uh from your traces to your data sets uh from your traces to your data sets uh from you know scoring mechanisms uh this from you know scoring mechanisms uh this from you know scoring mechanisms uh this is all basically a team sport. we all is all basically a team sport. we all is all basically a team sport. we all have to play uh and help improve the have to play uh and help improve the have to play uh and help improve the quality of AI.

quality of AI. quality of AI. So going a little bit deeper into the So going a little bit deeper into the So going a little bit deeper into the same aspect uh we have different uh same aspect uh we have different uh same aspect uh we have different uh teams uh at Door Dash who help us teams uh at Door Dash who help us teams uh at Door Dash who help us actually improve the quality of AI. So actually improve the quality of AI. So actually improve the quality of AI. So you're going to have your strategy and you're going to have your strategy and you're going to have your strategy and operations folks who are going to set operations folks who are going to set operations folks who are going to set priorities, set the quality bar that you priorities, set the quality bar that you priorities, set the quality bar that you want to aim for. You're going to have want to aim for. You're going to have want to aim for. You're going to have your product people who are going to your product people who are going to your product people who are going to translate uh these requirements into translate uh these requirements into translate uh these requirements into rubrics workflows. You're going to have rubrics workflows. You're going to have rubrics workflows. You're going to have your operations teams running uh your operations teams running uh your operations teams running uh annotations. You're going to have your annotations. You're going to have your annotations. You're going to have your engineering teams like us uh providing engineering teams like us uh providing engineering teams like us uh providing APIs, telemetry, data sets, judges, all APIs, telemetry, data sets, judges, all APIs, telemetry, data sets, judges, all you know the the cool things. Um and you know the the cool things. Um and you know the the cool things. Um and combining all these together is is what combining all these together is is what combining all these together is is what a recipe is for actually making sure a recipe is for actually making sure a recipe is for actually making sure that you are shipping quality AI that you are shipping quality AI that you are shipping quality AI products through an eval platform. products through an eval platform. products through an eval platform. So we've tried to boil this down uh into So we've tried to boil this down uh into So we've tried to boil this down uh into sort of you know like a a continuous sort of you know like a a continuous sort of you know like a a continuous iteration loop. Uh so right from tracing iteration loop. Uh so right from tracing iteration loop. Uh so right from tracing uh you know having a tracing solution uh you know having a tracing solution uh you know having a tracing solution viewing your sessions your traces to viewing your sessions your traces to viewing your sessions your traces to sampling them down you uh you know to a sampling them down you uh you know to a sampling them down you uh you know to a very small uh set that you actually want very small uh set that you actually want very small uh set that you actually want to look at uh annotating these with the to look at uh annotating these with the to look at uh annotating these with the domain specific expertise that you bring domain specific expertise that you bring domain specific expertise that you bring in with the different teams I mentioned in with the different teams I mentioned in with the different teams I mentioned reviewing those uh then creating those reviewing those uh then creating those reviewing those uh then creating those golden data sets which are going to be golden data sets which are going to be golden data sets which are going to be you you know your uh golden data sets you you know your uh golden data sets you you know your uh golden data sets and that that you want to measure or and that that you want to measure or and that that you want to measure or calibrate against uh and then of course calibrate against uh and then of course calibrate against uh and then of course like you know monitoring this over a like you know monitoring this over a like you know monitoring this over a period of time and then you know rinse period of time and then you know rinse period of time and then you know rinse and repeat uh go through the whole loop and repeat uh go through the whole loop and repeat uh go through the whole loop again. So this is in our experience has

again. So this is in our experience has again. So this is in our experience has been you know like a good sort of been you know like a good sort of been you know like a good sort of continuous loop uh for you know shipping continuous loop uh for you know shipping continuous loop uh for you know shipping quality AI quality AI quality AI at the plat on on the platform level uh at the plat on on the platform level uh at the plat on on the platform level uh we we have two surfaces uh so we have we we have two surfaces uh so we have we we have two surfaces uh so we have the telemetry layer uh where we have all the telemetry layer uh where we have all the telemetry layer uh where we have all our traces our scores uh observations our traces our scores uh observations our traces our scores uh observations that is also sort of the plane where that is also sort of the plane where that is also sort of the plane where users are able to access these traces users are able to access these traces users are able to access these traces using an MCP using an SDK uh using our using an MCP using an SDK uh using our using an MCP using an SDK uh using our APIs and then we have the workflow This APIs and then we have the workflow This APIs and then we have the workflow This is where a lot of our strat ops, our is where a lot of our strat ops, our is where a lot of our strat ops, our product teams operate on the platform. product teams operate on the platform. product teams operate on the platform. So this is where all the annotation So this is where all the annotation So this is where all the annotation tasks are set. Uh you know this is where tasks are set. Uh you know this is where tasks are set. Uh you know this is where they review their golden data sets, uh they review their golden data sets, uh they review their golden data sets, uh create their judges, calibrate their create their judges, calibrate their create their judges, calibrate their judges and so on. judges and so on. judges and so on. So maybe today we'll go through you know So maybe today we'll go through you know So maybe today we'll go through you know these sort of four different uh modules these sort of four different uh modules these sort of four different uh modules or pillars of our platform uh step by or pillars of our platform uh step by or pillars of our platform uh step by step. Uh so again first one uh tracing step. Uh so again first one uh tracing step. Uh so again first one uh tracing and sampling uh which is actually and sampling uh which is actually and sampling uh which is actually capturing what your agents what your capturing what your agents what your capturing what your agents what your LLMs are actually uh you know outputting LLMs are actually uh you know outputting LLMs are actually uh you know outputting for the lack of better words uh and for the lack of better words uh and for the lack of better words uh and actually viewing those.

actually viewing those. actually viewing those. Now in order to also power this uh whole Now in order to also power this uh whole Now in order to also power this uh whole platform we have I think as far platform we have I think as far platform we have I think as far mentioned we have gone in an API first mentioned we have gone in an API first mentioned we have gone in an API first uh approach. Uh what that has allowed us uh approach. Uh what that has allowed us uh approach. Uh what that has allowed us to do is have these table APIs that to do is have these table APIs that to do is have these table APIs that actually uh you know and then you know actually uh you know and then you know actually uh you know and then you know build UIs uh on top of that. Uh so all build UIs uh on top of that. Uh so all build UIs uh on top of that. Uh so all our scores our data sets uh these are our scores our data sets uh these are our scores our data sets uh these are all powered by very stable APIs uh that all powered by very stable APIs uh that all powered by very stable APIs uh that our team owns. Uh so all your API access our team owns. Uh so all your API access our team owns. Uh so all your API access uh including you know like an SDK access uh including you know like an SDK access uh including you know like an SDK access is basically powered by this single uh is basically powered by this single uh is basically powered by this single uh plane. plane. plane. Um again Um again Um again going back uh and you know like just going back uh and you know like just going back uh and you know like just refreshing your memory. Uh step one refreshing your memory. Uh step one refreshing your memory. Uh step one capture your traces uh capture your capture your traces uh capture your capture your traces uh capture your sessions uh measure your scores. Uh then sessions uh measure your scores. Uh then sessions uh measure your scores. Uh then you want to start uh almost you know you want to start uh almost you know you want to start uh almost you know like adding all your judgment your like adding all your judgment your like adding all your judgment your context your domain knowledge uh and context your domain knowledge uh and context your domain knowledge uh and then calibrating your judges is what we then calibrating your judges is what we then calibrating your judges is what we have seen as the whole uh life cycle.

have seen as the whole uh life cycle. have seen as the whole uh life cycle. Step two is on the annotation side. Uh Step two is on the annotation side. Uh Step two is on the annotation side. Uh so you obviously are capturing a lot of so you obviously are capturing a lot of so you obviously are capturing a lot of your uh agentic behavior, your sessions, your uh agentic behavior, your sessions, your uh agentic behavior, your sessions, your traces, but you actually want to your traces, but you actually want to your traces, but you actually want to see what are some places where things see what are some places where things see what are some places where things went well and what are some places where went well and what are some places where went well and what are some places where things did not go well. This is where things did not go well. This is where things did not go well. This is where you can actually titrate your your you you can actually titrate your your you you can actually titrate your your you know and actually look in inside what's know and actually look in inside what's know and actually look in inside what's actually happening uh at the session actually happening uh at the session actually happening uh at the session level and annotate these data sets. Um level and annotate these data sets. Um level and annotate these data sets. Um and as Surup mentioned, we have a lot of and as Surup mentioned, we have a lot of and as Surup mentioned, we have a lot of use cases. we have we we talked to use cases. we have we we talked to use cases. we have we we talked to multiple different teams who have uh multiple different teams who have uh multiple different teams who have uh various uh ways of annotating uh their various uh ways of annotating uh their various uh ways of annotating uh their data sets uh and it's it's almost hard data sets uh and it's it's almost hard data sets uh and it's it's almost hard for a platform team to you know build for a platform team to you know build for a platform team to you know build like a UI specific uh for each use case like a UI specific uh for each use case like a UI specific uh for each use case uh and and you know to give you an uh and and you know to give you an uh and and you know to give you an example uh it's usually going to be an example uh it's usually going to be an example uh it's usually going to be an annotator who's going to annotate these annotator who's going to annotate these annotator who's going to annotate these data sets so the platform team is you data sets so the platform team is you data sets so the platform team is you know in charge of the APIs we have a know in charge of the APIs we have a know in charge of the APIs we have a strategy and of person who's actually strategy and of person who's actually strategy and of person who's actually deciding what to annotate and then you deciding what to annotate and then you deciding what to annotate and then you have an annotator who's actually going have an annotator who's actually going have an annotator who's actually going to annotate uh your data set. So we took to annotate uh your data set. So we took to annotate uh your data set. So we took this approach uh everybody uh has uh you this approach uh everybody uh has uh you this approach uh everybody uh has uh you know access to coding agents uh and we know access to coding agents uh and we know access to coding agents uh and we actually doubled down on that API first actually doubled down on that API first actually doubled down on that API first approach. So because we had these APIs approach. So because we had these APIs approach. So because we had these APIs we were actually uh able to enable our we were actually uh able to enable our we were actually uh able to enable our statops teams to use something like a statops teams to use something like a statops teams to use something like a codeex or a claw code and v code their codeex or a claw code and v code their codeex or a claw code and v code their own annotation UIs. Uh so we had own annotation UIs. Uh so we had own annotation UIs. Uh so we had different use cases. Uh I think we had a different use cases. Uh I think we had a different use cases. Uh I think we had a talk from Ragav before. Uh we had image

talk from Ragav before. Uh we had image talk from Ragav before. Uh we had image annotation use cases. We had some uh you annotation use cases. We had some uh you annotation use cases. We had some uh you know manual testing use cases. What know manual testing use cases. What know manual testing use cases. What stood out to us was the underlying stood out to us was the underlying stood out to us was the underlying patterns were similar. So if we are are patterns were similar. So if we are are patterns were similar. So if we are are API first uh we can actually enable our API first uh we can actually enable our API first uh we can actually enable our our our partners to simply v code these our our partners to simply v code these our our partners to simply v code these UIs for annotation. So it's it's like a UIs for annotation. So it's it's like a UIs for annotation. So it's it's like a very simple example then you know of of very simple example then you know of of very simple example then you know of of a vibe coded UI looks pretty clean does a vibe coded UI looks pretty clean does a vibe coded UI looks pretty clean does the job uh and you get you know the the job uh and you get you know the the job uh and you get you know the annotation that you eat this is annotation that you eat this is annotation that you eat this is basically like a menu from a restaurant basically like a menu from a restaurant basically like a menu from a restaurant uh it's it's you know nothing crazy uh uh it's it's you know nothing crazy uh uh it's it's you know nothing crazy uh but the point I want to make here is but the point I want to make here is but the point I want to make here is that what helped us was to give this that what helped us was to give this that what helped us was to give this workflow in the hands of the operators workflow in the hands of the operators workflow in the hands of the operators so that they can actually build their so that they can actually build their so that they can actually build their own vcoded annotation UIs. Uh so moving own vcoded annotation UIs. Uh so moving own vcoded annotation UIs. Uh so moving on once you have these annotation UIs on once you have these annotation UIs on once you have these annotation UIs you obviously want to you know calibrate you obviously want to you know calibrate you obviously want to you know calibrate your your your judge prompts you your your your judge prompts you your your your judge prompts you obviously have some LM as a judge uh obviously have some LM as a judge uh obviously have some LM as a judge uh metric that you're tracking you want to metric that you're tracking you want to metric that you're tracking you want to now start improving that with these now start improving that with these now start improving that with these golden data sets golden data sets golden data sets u in order to do that uh you know we u in order to do that uh you know we u in order to do that uh you know we have a pretty simple process uh you have a pretty simple process uh you have a pretty simple process uh you you're going to start with you know some you're going to start with you know some you're going to start with you know some judge prompt take a look at you know judge prompt take a look at you know judge prompt take a look at you know what exactly do you want to measure from what exactly do you want to measure from what exactly do you want to measure from the output uh g you know have have us the output uh g you know have have us the output uh g you know have have us have something simple you're going to have something simple you're going to have something simple you're going to have your baseline scores uh where have your baseline scores uh where have your baseline scores uh where you're going to simply run those LLM you're going to simply run those LLM you're going to simply run those LLM judges on your traces and then you're judges on your traces and then you're judges on your traces and then you're going to have that optimization loop. Uh going to have that optimization loop. Uh going to have that optimization loop. Uh so we use uh the JPEA library which is a so we use uh the JPEA library which is a so we use uh the JPEA library which is a pretty commonly used library out there pretty commonly used library out there pretty commonly used library out there for prompt optimization. Uh and once you for prompt optimization. Uh and once you for prompt optimization. Uh and once you know the the iteration loop is complete

know the the iteration loop is complete know the the iteration loop is complete uh our partner teams are happy they're uh our partner teams are happy they're uh our partner teams are happy they're going to then elevate that judge prompt going to then elevate that judge prompt going to then elevate that judge prompt as their LLM as a judge. Now even while as their LLM as a judge. Now even while as their LLM as a judge. Now even while doing that uh LLM as a judge as a doing that uh LLM as a judge as a doing that uh LLM as a judge as a concept the whole prompt calibration concept the whole prompt calibration concept the whole prompt calibration concept might be uh straightforward to a concept might be uh straightforward to a concept might be uh straightforward to a lot of folks but it is still like a lot of folks but it is still like a lot of folks but it is still like a pretty new and evolving field. Uh and pretty new and evolving field. Uh and pretty new and evolving field. Uh and what we wanted to do was really reduce what we wanted to do was really reduce what we wanted to do was really reduce the friction of back and forth with an the friction of back and forth with an the friction of back and forth with an engineering team. So we tried to really engineering team. So we tried to really engineering team. So we tried to really remove all the complicated logic and remove all the complicated logic and remove all the complicated logic and make this into a self-s serve UI. So the make this into a self-s serve UI. So the make this into a self-s serve UI. So the screenshot that you actually see is what screenshot that you actually see is what screenshot that you actually see is what actually exists. uh so uh you know like actually exists. uh so uh you know like actually exists. uh so uh you know like a product manager or an operator is a product manager or an operator is a product manager or an operator is going to come to our UI. They're going going to come to our UI. They're going going to come to our UI. They're going to set some of these configs uh on the to set some of these configs uh on the to set some of these configs uh on the platform and then actually run the platform and then actually run the platform and then actually run the calibration loop themselves. So they calibration loop themselves. So they calibration loop themselves. So they don't have to worry about the different don't have to worry about the different don't have to worry about the different settings that they need to worry about settings that they need to worry about settings that they need to worry about what are the different uh you know what are the different uh you know what are the different uh you know tweaks that they need to do and they can tweaks that they need to do and they can tweaks that they need to do and they can actually like you know run a calibration actually like you know run a calibration actually like you know run a calibration loop using any model of their choice. I loop using any model of their choice. I loop using any model of their choice. I think in this example I have Gemini they think in this example I have Gemini they think in this example I have Gemini they can use run it using uh you know any of can use run it using uh you know any of can use run it using uh you know any of the claude or the openi models too. The the claude or the openi models too. The the claude or the openi models too. The other important piece was actually uh other important piece was actually uh other important piece was actually uh making this reviewable. Uh you know making this reviewable. Uh you know making this reviewable. Uh you know again a lot of this uh is a closed box again a lot of this uh is a closed box again a lot of this uh is a closed box where you can't really it's hard to see where you can't really it's hard to see where you can't really it's hard to see what's actually happening. Uh so the what's actually happening. Uh so the what's actually happening. Uh so the second piece that we built was actually second piece that we built was actually second piece that we built was actually giving them vis visualization and giving them vis visualization and giving them vis visualization and visibility into what's actually visibility into what's actually visibility into what's actually happening. So on the left you can see we happening. So on the left you can see we happening. So on the left you can see we and this is like one of the good and this is like one of the good and this is like one of the good examples where we saw like a significant examples where we saw like a significant examples where we saw like a significant amount of improvement in the judge amount of improvement in the judge amount of improvement in the judge prompt. uh and we actually show the you

prompt. uh and we actually show the you prompt. uh and we actually show the you know the the previous the original know the the previous the original know the the previous the original system prompt and the calibrated prompt system prompt and the calibrated prompt system prompt and the calibrated prompt to our partners so that they are also to our partners so that they are also to our partners so that they are also able to gain that trust uh why as as as able to gain that trust uh why as as as able to gain that trust uh why as as as we build this we build this we build this [clears throat] [clears throat] [clears throat] >> yeah just want to add to that is this >> yeah just want to add to that is this >> yeah just want to add to that is this enables different configurations in enables different configurations in enables different configurations in different teams in some teams you have different teams in some teams you have different teams in some teams you have seen the strategy and operations folks seen the strategy and operations folks seen the strategy and operations folks own the prompt uh you have seen some own the prompt uh you have seen some own the prompt uh you have seen some teams where the product manager owns the teams where the product manager owns the teams where the product manager owns the prompt you have seen some teams where prompt you have seen some teams where prompt you have seen some teams where engineering owns the prompt so this engineering owns the prompt so this engineering owns the prompt so this gives gives the flexibility for teams to gives gives the flexibility for teams to gives gives the flexibility for teams to design and evolve because we are all design and evolve because we are all design and evolve because we are all learning. So the even the org uh design learning. So the even the org uh design learning. So the even the org uh design is improving and we are enabling that. is improving and we are enabling that. is improving and we are enabling that. >> Yeah, that that's a good point. I think >> Yeah, that that's a good point. I think >> Yeah, that that's a good point. I think the overall idea was to you know build the overall idea was to you know build the overall idea was to you know build something which is as self-s served as something which is as self-s served as something which is as self-s served as possible so that uh you know people possible so that uh you know people possible so that uh you know people aren't always necessarily blocked by our aren't always necessarily blocked by our aren't always necessarily blocked by our team helping them out. Um and then team helping them out. Um and then team helping them out. Um and then finally you know uh the quality loop in finally you know uh the quality loop in finally you know uh the quality loop in practice. you know as we've been going practice. you know as we've been going practice. you know as we've been going through this exercise we've seen a lot through this exercise we've seen a lot through this exercise we've seen a lot of improvements happening to our product of improvements happening to our product of improvements happening to our product as well. So you know for example we sort as well. So you know for example we sort as well. So you know for example we sort mentioned we started with the UIs we are mentioned we started with the UIs we are mentioned we started with the UIs we are you know now API and workflow first uh you know now API and workflow first uh you know now API and workflow first uh we're trying to reuse a lot of the we're trying to reuse a lot of the we're trying to reuse a lot of the existing infrastructure that already existing infrastructure that already existing infrastructure that already existed at Door Dash uh and that's existed at Door Dash uh and that's existed at Door Dash uh and that's helped us uh get a long way. Now some of helped us uh get a long way. Now some of helped us uh get a long way. Now some of uh we we we've seen obviously like you uh we we we've seen obviously like you uh we we we've seen obviously like you know really good results. I think a very know really good results. I think a very know really good results. I think a very good result that we we do like to call good result that we we do like to call good result that we we do like to call out is we actually did see a lot of out is we actually did see a lot of out is we actually did see a lot of reduction in the spend uh at per reduction in the spend uh at per reduction in the spend uh at per annotation cost as you as you all can annotation cost as you as you all can annotation cost as you as you all can imagine we do have you know thousands of imagine we do have you know thousands of imagine we do have you know thousands of rows that need to get annotated every

rows that need to get annotated every rows that need to get annotated every week uh and it can get pretty expensive week uh and it can get pretty expensive week uh and it can get pretty expensive at doash scale uh and having this at doash scale uh and having this at doash scale uh and having this selfserve uh annotation platform really selfserve uh annotation platform really selfserve uh annotation platform really helped us reduce increase the velocity helped us reduce increase the velocity helped us reduce increase the velocity and reduce the cost that we were and reduce the cost that we were and reduce the cost that we were actually spending with these annotators. actually spending with these annotators. actually spending with these annotators. to to annotate the data for us. Uh to to annotate the data for us. Uh to to annotate the data for us. Uh obviously uh this resulted in faster obviously uh this resulted in faster obviously uh this resulted in faster loops. Uh teams were able to iterate loops. Uh teams were able to iterate loops. Uh teams were able to iterate faster. They were able to uh you know faster. They were able to uh you know faster. They were able to uh you know calibrate their own judges in a calibrate their own judges in a calibrate their own judges in a completely self-s served way. Uh and completely self-s served way. Uh and completely self-s served way. Uh and thus it has resulted us in in in moving thus it has resulted us in in in moving thus it has resulted us in in in moving with a very very high velocity. with a very very high velocity. with a very very high velocity. So uh finally I just wanted to you know So uh finally I just wanted to you know So uh finally I just wanted to you know quickly touch on this slide again uh the quickly touch on this slide again uh the quickly touch on this slide again uh the eight steps you know continuous loop uh eight steps you know continuous loop uh eight steps you know continuous loop uh which is you know you you have your which is you know you you have your which is you know you you have your traces you want to look at your traces traces you want to look at your traces traces you want to look at your traces your sessions you want to sample it down your sessions you want to sample it down your sessions you want to sample it down to a size which is which you are to a size which is which you are to a size which is which you are comfortable with uh you want to start comfortable with uh you want to start comfortable with uh you want to start annotating your data sets you really annotating your data sets you really annotating your data sets you really want to start uh making the data better want to start uh making the data better want to start uh making the data better with the human knowledge that exists and with the human knowledge that exists and with the human knowledge that exists and the domain knowledge that exists and the domain knowledge that exists and the domain knowledge that exists and then calibrate your workflows was then calibrate your workflows was then calibrate your workflows was calibrate your agents, calibrate your calibrate your agents, calibrate your calibrate your agents, calibrate your LLM judges with this golden data set and LLM judges with this golden data set and LLM judges with this golden data set and then repeat this whole cycle uh over a then repeat this whole cycle uh over a then repeat this whole cycle uh over a period of time to you know to ship period of time to you know to ship period of time to you know to ship reliably and ship with high quality. Um reliably and ship with high quality. Um reliably and ship with high quality. Um yeah, we have 4 minutes left. Thank you yeah, we have 4 minutes left. Thank you yeah, we have 4 minutes left. Thank you once again. I think that was the last once again. I think that was the last once again. I think that was the last slide. Uh thanks for attending and if slide. Uh thanks for attending and if slide. Uh thanks for attending and if there's any questions, we'd be happy to there's any questions, we'd be happy to there's any questions, we'd be happy to hang out after the talk or even happy to hang out after the talk or even happy to hang out after the talk or even happy to answer them now.