SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)

Hi everybody, my name is Shrea. I am the Hi everybody, my name is Shrea. I am the CEO of Snow Globe and we have with us CEO of Snow Globe and we have with us CEO of Snow Globe and we have with us Ammon who is a principal machine Ammon who is a principal machine Ammon who is a principal machine learning engineer at new. And this talk learning engineer at new. And this talk learning engineer at new. And this talk is going to be about simulation maxing is going to be about simulation maxing is going to be about simulation maxing and how you can ship or how new bank and how you can ship or how new bank and how you can ship or how new bank ships uh agents 20x faster using ships uh agents 20x faster using ships uh agents 20x faster using simulations. Hey everyone, I'm Aman. Uh so let me Hey everyone, I'm Aman. Uh so let me talk about New Bank at a glance. We are talk about New Bank at a glance. We are talk about New Bank at a glance. We are the lead leading digital bank in Latin the lead leading digital bank in Latin the lead leading digital bank in Latin America. We have 135 million customers America. We have 135 million customers America. We have 135 million customers uh in Brazil, Mexico, and Colombia and uh in Brazil, Mexico, and Colombia and uh in Brazil, Mexico, and Colombia and we are launching in the US real soon. uh we are launching in the US real soon. uh we are launching in the US real soon. uh our quarterly revenue crossed 5 billion our quarterly revenue crossed 5 billion our quarterly revenue crossed 5 billion in Q1 2026. Our NPS customer love is in Q1 2026. Our NPS customer love is in Q1 2026. Our NPS customer love is very high and we are the perfect company very high and we are the perfect company very high and we are the perfect company for using AI agents for customer support for using AI agents for customer support for using AI agents for customer support where human Xpers are fanatical customer where human Xpers are fanatical customer where human Xpers are fanatical customer care people and AI agents together solve care people and AI agents together solve care people and AI agents together solve customer issues in a fast empathetic and customer issues in a fast empathetic and customer issues in a fast empathetic and correct manner. AI handles a lot of correct manner. AI handles a lot of correct manner. AI handles a lot of routines end to end. uh humans focus on routines end to end. uh humans focus on routines end to end. uh humans focus on the hardest and longtail cases and the hardest and longtail cases and the hardest and longtail cases and together we aim to delight our together we aim to delight our together we aim to delight our customers.

customers. customers. So as Shria said this talk is only about So as Shria said this talk is only about So as Shria said this talk is only about uh one thing really. If you generate uh one thing really. If you generate uh one thing really. If you generate your eval data in sim instead of waiting your eval data in sim instead of waiting your eval data in sim instead of waiting on production data you can ship agents on production data you can ship agents on production data you can ship agents 20x faster and we'll give you evidence 20x faster and we'll give you evidence 20x faster and we'll give you evidence for that. So let's start with the for that. So let's start with the for that. So let's start with the results directly. Uh uh so this is you results directly. Uh uh so this is you results directly. Uh uh so this is you know the average of TNPS which is a know the average of TNPS which is a know the average of TNPS which is a measure of customer satisfaction for measure of customer satisfaction for measure of customer satisfaction for five of our AI agents in production and five of our AI agents in production and five of our AI agents in production and at the beginning uh they were not so at the beginning uh they were not so at the beginning uh they were not so great but now in a with a few months of great but now in a with a few months of great but now in a with a few months of uh of work and a few quarters worth of uh of work and a few quarters worth of uh of work and a few quarters worth of effort we've been able to massively effort we've been able to massively effort we've been able to massively increase the TNPS and customer love for increase the TNPS and customer love for increase the TNPS and customer love for our AI agents and many of them are our AI agents and many of them are our AI agents and many of them are approaching human quality and this data approaching human quality and this data approaching human quality and this data is a bit stale many of them are is a bit stale many of them are is a bit stale many of them are exceeding human quality. So we are at exceeding human quality. So we are at exceeding human quality. So we are at the stage where we are actually able to the stage where we are actually able to the stage where we are actually able to show proof that this actually works in show proof that this actually works in show proof that this actually works in production. Uh here's a QR code for our production. Uh here's a QR code for our production. Uh here's a QR code for our KD paper in case you want to check it KD paper in case you want to check it KD paper in case you want to check it out. It's going to be presented uh in out. It's going to be presented uh in out. It's going to be presented uh in Korea uh in August.

>> Uh awesome. So we open with results and >> Uh awesome. So we open with results and you know it's really about this journey you know it's really about this journey you know it's really about this journey of how do you implement the right of how do you implement the right of how do you implement the right systems uh for evaluation in order to systems uh for evaluation in order to systems uh for evaluation in order to you know be able to achieve those you know be able to achieve those you know be able to achieve those results. So in this talk we basically results. So in this talk we basically results. So in this talk we basically split it up into these three sections. split it up into these three sections. split it up into these three sections. The first is why evals are so important The first is why evals are so important The first is why evals are so important and essential but why they're also the and essential but why they're also the and essential but why they're also the bottleneck from being able to you know bottleneck from being able to you know bottleneck from being able to you know do a lot of high throughput do a lot of high throughput do a lot of high throughput experimentation to get the results that experimentation to get the results that experimentation to get the results that Amen was showing earlier and then why Amen was showing earlier and then why Amen was showing earlier and then why the second part of this talk is about the second part of this talk is about the second part of this talk is about why simulated data uh works as well as why simulated data uh works as well as why simulated data uh works as well as collected data and helps you circumvent collected data and helps you circumvent collected data and helps you circumvent a lot of this bottleneck that we're a lot of this bottleneck that we're a lot of this bottleneck that we're going to talk about and then the third going to talk about and then the third going to talk about and then the third part is really digging deep into the part is really digging deep into the part is really digging deep into the systems and the findings that we had by systems and the findings that we had by systems and the findings that we had by implementing uh you know this framework implementing uh you know this framework implementing uh you know this framework at scale in new at scale in new at scale in new Uh, awesome. So, eval there this there's Uh, awesome. So, eval there this there's Uh, awesome. So, eval there this there's like a whole talk track dedicated at like a whole talk track dedicated at like a whole talk track dedicated at this conference to eval. Evals are this conference to eval. Evals are this conference to eval. Evals are absolutely critical for building good absolutely critical for building good absolutely critical for building good agents and evals are really only about agents and evals are really only about agents and evals are really only about two things, right? There's metrics and two things, right? There's metrics and two things, right? There's metrics and there's data. uh metrics again I hope there's data. uh metrics again I hope there's data. uh metrics again I hope you attended many of the amazing talks you attended many of the amazing talks you attended many of the amazing talks yesterday on the evals track but metrics yesterday on the evals track but metrics yesterday on the evals track but metrics are really while they're challenging we are really while they're challenging we are really while they're challenging we have a playbook for how to build metrics have a playbook for how to build metrics have a playbook for how to build metrics that are really well aligned with you that are really well aligned with you that are really well aligned with you know the rubrics that we care about know the rubrics that we care about know the rubrics that we care about right which is you essentially use um right which is you essentially use um right which is you essentially use um you know LM as a judge style uh you know you know LM as a judge style uh you know you know LM as a judge style uh you know classifiers and you align it with human classifiers and you align it with human classifiers and you align it with human judgment and and getting human data and judgment and and getting human data and judgment and and getting human data and you can iteratively build on it using you can iteratively build on it using you can iteratively build on it using you know auto optimization and you know auto optimization and you know auto optimization and autoprompt tuning uh techniques the autoprompt tuning uh techniques the autoprompt tuning uh techniques the thing that's a bottleneck And that still thing that's a bottleneck And that still thing that's a bottleneck And that still remains very challenging and unsolved is remains very challenging and unsolved is remains very challenging and unsolved is what is the data that you're actually

what is the data that you're actually what is the data that you're actually computing these metrics on and that computing these metrics on and that computing these metrics on and that process is very timeconuming and very process is very timeconuming and very process is very timeconuming and very expensive specifically so for agents. So expensive specifically so for agents. So expensive specifically so for agents. So once again people have been talking once again people have been talking once again people have been talking about you know if if you're around like about you know if if you're around like about you know if if you're around like machine learning era circa you know 2018 machine learning era circa you know 2018 machine learning era circa you know 2018 it would be like ML work is 85% data it would be like ML work is 85% data it would be like ML work is 85% data work right so data has always been work right so data has always been work right so data has always been challenging but with agents the level of challenging but with agents the level of challenging but with agents the level of sophistication that data requires is sophistication that data requires is sophistication that data requires is just so much more expensive. So here's just so much more expensive. So here's just so much more expensive. So here's examples you know of what structured ML examples you know of what structured ML examples you know of what structured ML data looked like uh and what even early data looked like uh and what even early data looked like uh and what even early era of AI uh data for you know chat bots era of AI uh data for you know chat bots era of AI uh data for you know chat bots or single turn QA which was you know or single turn QA which was you know or single turn QA which was you know much more manageable and tractable and much more manageable and tractable and much more manageable and tractable and you could still think of it as these you could still think of it as these you could still think of it as these structured rows. Uh but now for structured rows. Uh but now for structured rows. Uh but now for multi-turn agents each data point is a multi-turn agents each data point is a multi-turn agents each data point is a trajectory with a lot of internal tool trajectory with a lot of internal tool trajectory with a lot of internal tool calls that all need state etc. And that calls that all need state etc. And that calls that all need state etc. And that becomes very even one data point is becomes very even one data point is becomes very even one data point is expensive to generate and annotate right expensive to generate and annotate right expensive to generate and annotate right but scaling it up on a sample set but scaling it up on a sample set but scaling it up on a sample set becomes even harder. becomes even harder. becomes even harder. Um so how teams primarily get eval data Um so how teams primarily get eval data Um so how teams primarily get eval data today is two methods. Uh one is either today is two methods. Uh one is either today is two methods. Uh one is either manual or the second is production manual or the second is production manual or the second is production traces and both of those are painful but traces and both of those are painful but traces and both of those are painful but for different reasons right manual for different reasons right manual for different reasons right manual authoring I mean this one's obvious it authoring I mean this one's obvious it authoring I mean this one's obvious it is so timeconuming especially the kind is so timeconuming especially the kind is so timeconuming especially the kind of data that I was showing you on the of data that I was showing you on the of data that I was showing you on the previous slide. You have to hand plan previous slide. You have to hand plan previous slide. You have to hand plan you know state updates. You have to hand you know state updates. You have to hand you know state updates. You have to hand plan the trajectory that you want to plan the trajectory that you want to plan the trajectory that you want to take this uh you know agent data point take this uh you know agent data point take this uh you know agent data point in and synthetic uh the the state for in and synthetic uh the the state for in and synthetic uh the the state for these data points has to stay consistent these data points has to stay consistent these data points has to stay consistent across all tool calls etc that you make across all tool calls etc that you make across all tool calls etc that you make right production traces in comparison

right production traces in comparison right production traces in comparison are almost free. You don't have to pay are almost free. You don't have to pay are almost free. You don't have to pay for them. You're going to get them for them. You're going to get them for them. You're going to get them anyway. But the cost is that you're anyway. But the cost is that you're anyway. But the cost is that you're testing on real live users every time testing on real live users every time testing on real live users every time you're testing it. So for example, if you're testing it. So for example, if you're testing it. So for example, if you want to run, you know, many parallel you want to run, you know, many parallel you want to run, you know, many parallel experiments at scale, which is often experiments at scale, which is often experiments at scale, which is often something you want to do in a machine something you want to do in a machine something you want to do in a machine learning setting is it becomes very hard learning setting is it becomes very hard learning setting is it becomes very hard to do that in prod. >> So for those of you who build agents, >> So for those of you who build agents, you know how difficult building a new you know how difficult building a new you know how difficult building a new agent and maintaining uh an existing agent and maintaining uh an existing agent and maintaining uh an existing agent is. So if you want to change the agent is. So if you want to change the agent is. So if you want to change the agent harness, you want to change the agent harness, you want to change the agent harness, you want to change the prompt, the tools, anything around the prompt, the tools, anything around the prompt, the tools, anything around the the LLM that powers your agent, that can the LLM that powers your agent, that can the LLM that powers your agent, that can take a few hours and in in the best take a few hours and in in the best take a few hours and in in the best case. But as Fria mentioned, running case. But as Fria mentioned, running case. But as Fria mentioned, running offline evals on hand curated data can offline evals on hand curated data can offline evals on hand curated data can take a few days. And then the the worst take a few days. And then the the worst take a few days. And then the the worst part is if you run an AB test in part is if you run an AB test in part is if you run an AB test in production and monitor regressions, see production and monitor regressions, see production and monitor regressions, see if you get a statistically significant if you get a statistically significant if you get a statistically significant improvement of of a previous version improvement of of a previous version improvement of of a previous version that can take forever because customer that can take forever because customer that can take forever because customer feedback is through some kind of a feedback is through some kind of a feedback is through some kind of a feedback form and it can be sparse, it feedback form and it can be sparse, it feedback form and it can be sparse, it can be noisy and can take forever to can be noisy and can take forever to can be noisy and can take forever to know am I moving in the right direction know am I moving in the right direction know am I moving in the right direction or not. So I'm here to tell you that uh or not. So I'm here to tell you that uh or not. So I'm here to tell you that uh we have verified in production that yes we have verified in production that yes we have verified in production that yes simulations circuit this timeline short simulations circuit this timeline short simulations circuit this timeline short circuit this timeline from a few weeks circuit this timeline from a few weeks circuit this timeline from a few weeks you can go to less than a day sometimes you can go to less than a day sometimes you can go to less than a day sometimes even a few hours a few minutes even a few hours a few minutes even a few hours a few minutes and uh as I mentioned earlier uh you and uh as I mentioned earlier uh you and uh as I mentioned earlier uh you know with simulations it takes a few know with simulations it takes a few know with simulations it takes a few hours uh without simulation it takes a hours uh without simulation it takes a hours uh without simulation it takes a few hours to change the harness and few hours to change the harness and few hours to change the harness and offline evals and all of that's become offline evals and all of that's become offline evals and all of that's become super fast now with simulation the agent

super fast now with simulation the agent super fast now with simulation the agent harness chain still takes a few hours harness chain still takes a few hours harness chain still takes a few hours running offline evals becomes a lot running offline evals becomes a lot running offline evals becomes a lot easier because you have a simulated user easier because you have a simulated user easier because you have a simulated user to fall back on. You're not waiting on to fall back on. You're not waiting on to fall back on. You're not waiting on hand curated data. You're not waiting on hand curated data. You're not waiting on hand curated data. You're not waiting on you know people to create data for you. you know people to create data for you. you know people to create data for you. And the best part if you want to know And the best part if you want to know And the best part if you want to know that am I doing well in production? Is that am I doing well in production? Is that am I doing well in production? Is this version going to do well? That can this version going to do well? That can this version going to do well? That can take just a few hours. So this has been take just a few hours. So this has been take just a few hours. So this has been a real gamecher for us at New Bank. a real gamecher for us at New Bank. a real gamecher for us at New Bank. >> Um all right. So we talked about you >> Um all right. So we talked about you >> Um all right. So we talked about you know like simulations shortening the know like simulations shortening the know like simulations shortening the release cycle but what is mechanically a release cycle but what is mechanically a release cycle but what is mechanically a simulation you know it's existed in simulation you know it's existed in simulation you know it's existed in robotics it's existed in you know a lot robotics it's existed in you know a lot robotics it's existed in you know a lot of different domains but simulations for of different domains but simulations for of different domains but simulations for agent eval you know mean a specific agent eval you know mean a specific agent eval you know mean a specific thing and this is really where snow thing and this is really where snow thing and this is really where snow globe enters and you know sets up a lot globe enters and you know sets up a lot globe enters and you know sets up a lot of that core infrastructure for you. So of that core infrastructure for you. So of that core infrastructure for you. So simulations are essentially you know it simulations are essentially you know it simulations are essentially you know it works by first like wrapping your agent works by first like wrapping your agent works by first like wrapping your agent you know using a um uh like pointing the you know using a um uh like pointing the you know using a um uh like pointing the snow globe SDK at your agent without snow globe SDK at your agent without snow globe SDK at your agent without requiring any code changes and you know requiring any code changes and you know requiring any code changes and you know you can gro what are the tools that you you can gro what are the tools that you you can gro what are the tools that you really need in to mock in order to make really need in to mock in order to make really need in to mock in order to make the simulation really work. The second the simulation really work. The second the simulation really work. The second input is about how you want to drive or input is about how you want to drive or input is about how you want to drive or steer your your simulation. So you know steer your your simulation. So you know steer your your simulation. So you know who are the personas what are the use who are the personas what are the use who are the personas what are the use cases what are the data points that cases what are the data points that cases what are the data points that we're really simulating. Once you have we're really simulating. Once you have we're really simulating. Once you have those inputs ready, snow globe those inputs ready, snow globe those inputs ready, snow globe essentially runs the simulation and on essentially runs the simulation and on essentially runs the simulation and on the output you end up getting you know the output you end up getting you know the output you end up getting you know first just a data set. So you get like first just a data set. So you get like first just a data set. So you get like thousands of multi-turn conversations thousands of multi-turn conversations thousands of multi-turn conversations against your real agents. These against your real agents. These against your real agents. These conversations have you know tools mocked conversations have you know tools mocked conversations have you know tools mocked etc. have all of the bells and whistles etc. have all of the bells and whistles etc. have all of the bells and whistles needed to be actually you know useful

needed to be actually you know useful needed to be actually you know useful for evaluating your agent already ready for evaluating your agent already ready for evaluating your agent already ready and then you run like judges on top of and then you run like judges on top of and then you run like judges on top of it so that on this data set you get you it so that on this data set you get you it so that on this data set you get you know per turn information about how your know per turn information about how your know per turn information about how your agent is really behaving and it then agent is really behaving and it then agent is really behaving and it then like pipes directly into you know your like pipes directly into you know your like pipes directly into you know your evaluation pipeline so that you can evaluation pipeline so that you can evaluation pipeline so that you can really get data on demand. really get data on demand. really get data on demand. Um this is an example of what a sim what Um this is an example of what a sim what Um this is an example of what a sim what the what what a concrete output of the what what a concrete output of the what what a concrete output of simulation is like right so this is as simulation is like right so this is as simulation is like right so this is as an example like one simulated persona an example like one simulated persona an example like one simulated persona and then the corresponding conversation and then the corresponding conversation and then the corresponding conversation or data point for that simulated persona or data point for that simulated persona or data point for that simulated persona so here for example this is totally so here for example this is totally so here for example this is totally synthetic a user named Maria Souza uh synthetic a user named Maria Souza uh synthetic a user named Maria Souza uh you know she wants to order a credit you know she wants to order a credit you know she wants to order a credit card this is who she is she's 34 a card this is who she is she's 34 a card this is who she is she's 34 a designer first-time credit card customer designer first-time credit card customer designer first-time credit card customer in addition to you know this basic in addition to you know this basic in addition to you know this basic information about this synthetic user we information about this synthetic user we information about this synthetic user we also will generate, you know, this also will generate, you know, this also will generate, you know, this essential grounding data as an example, essential grounding data as an example, essential grounding data as an example, this fake address, a fake credit card, this fake address, a fake credit card, this fake address, a fake credit card, etc. that stays consistent across your etc. that stays consistent across your etc. that stays consistent across your agent execution so that you know your agent execution so that you know your agent execution so that you know your agent can actually be evaluated against agent can actually be evaluated against agent can actually be evaluated against that. And then you also have things like that. And then you also have things like that. And then you also have things like tone, voice, how do they really tone, voice, how do they really tone, voice, how do they really interact? And then oops, and then when interact? And then oops, and then when interact? And then oops, and then when you actually run this persona through you actually run this persona through you actually run this persona through the conver through the agent, you end up the conver through the agent, you end up the conver through the agent, you end up getting, you know, this uh conversation getting, you know, this uh conversation getting, you know, this uh conversation that's consistent with the simulated that's consistent with the simulated that's consistent with the simulated intent, right? So you see for example intent, right? So you see for example intent, right? So you see for example that this user has very curt oneline that this user has very curt oneline that this user has very curt oneline messages in terms of how they interact messages in terms of how they interact messages in terms of how they interact because this is how the persona was set because this is how the persona was set because this is how the persona was set up or that you know when the agent wants up or that you know when the agent wants up or that you know when the agent wants to when the agent runs its tool when to when the agent runs its tool when to when the agent runs its tool when Maria enters the chat the agent is Maria enters the chat the agent is Maria enters the chat the agent is actually able to get a valid output

actually able to get a valid output actually able to get a valid output mocked by the simulator that you know is mocked by the simulator that you know is mocked by the simulator that you know is able to identify who the user is where able to identify who the user is where able to identify who the user is where the address is the address can be the address is the address can be the address is the address can be verified etc. Um and then finally all of verified etc. Um and then finally all of verified etc. Um and then finally all of this is also grounded in like account this is also grounded in like account this is also grounded in like account data as well. So for those of us building real agents, So for those of us building real agents, you know how hard things are. Uh you you know how hard things are. Uh you you know how hard things are. Uh you don't know if your agent is good, you don't know if your agent is good, you don't know if your agent is good, you don't know if your evals are good. You don't know if your evals are good. You don't know if your evals are good. You have no idea what's happening. And that have no idea what's happening. And that have no idea what's happening. And that was me a few quarters ago. Uh since the was me a few quarters ago. Uh since the was me a few quarters ago. Uh since the theme of uh this year's conference is theme of uh this year's conference is theme of uh this year's conference is self-improving agents. So we are also self-improving agents. So we are also self-improving agents. So we are also kind of jumping on the bandwagon. So kind of jumping on the bandwagon. So kind of jumping on the bandwagon. So this is our self-improvement loop. You this is our self-improvement loop. You this is our self-improvement loop. You ship your agent, you observe, you create ship your agent, you observe, you create ship your agent, you observe, you create robust evals. You spend a lot of time robust evals. You spend a lot of time robust evals. You spend a lot of time creating robust evals using automated creating robust evals using automated creating robust evals using automated prompt optimization with algorithms like prompt optimization with algorithms like prompt optimization with algorithms like Japa. And then you also do simulation. Japa. And then you also do simulation. Japa. And then you also do simulation. You run simulation and you pipe the You run simulation and you pipe the You run simulation and you pipe the simulation data and real data through simulation data and real data through simulation data and real data through your evals. And that gives you rich your evals. And that gives you rich your evals. And that gives you rich signal to then optimize your agent signal to then optimize your agent signal to then optimize your agent harness. And once you verify the agent harness. And once you verify the agent harness. And once you verify the agent harness optimization is good, you ship.

harness optimization is good, you ship. harness optimization is good, you ship. And this has enabled us to do many many And this has enabled us to do many many And this has enabled us to do many many many AB tests for a given agent. And we many AB tests for a given agent. And we many AB tests for a given agent. And we have many many such agents across the have many many such agents across the have many many such agents across the company. So this has been a real company. So this has been a real company. So this has been a real gamecher for us in terms of improving gamecher for us in terms of improving gamecher for us in terms of improving our agents and the proof as I already our agents and the proof as I already our agents and the proof as I already showed you is that you know the quality showed you is that you know the quality showed you is that you know the quality of our agents is going up uh pretty of our agents is going up uh pretty of our agents is going up uh pretty dramatically. dramatically. dramatically. This is some data on how simulation This is some data on how simulation This is some data on how simulation really tracks real production data. So really tracks real production data. So really tracks real production data. So when you run uh simulations you want to when you run uh simulations you want to when you run uh simulations you want to compare is my simulation comparable to compare is my simulation comparable to compare is my simulation comparable to production data. So we looked at sim production data. So we looked at sim production data. So we looked at sim quality via evals. So far uh you know quality via evals. So far uh you know quality via evals. So far uh you know quality of uh with evals from sim and quality of uh with evals from sim and quality of uh with evals from sim and real data the correlation is pretty high real data the correlation is pretty high real data the correlation is pretty high and we had human review done where 80% and we had human review done where 80% and we had human review done where 80% of our domain expert labels confirmed of our domain expert labels confirmed of our domain expert labels confirmed that sims give us usable data not just that sims give us usable data not just that sims give us usable data not just for uh mature agents but also for green for uh mature agents but also for green for uh mature agents but also for green field agents. So this has been uh pretty field agents. So this has been uh pretty field agents. So this has been uh pretty useful for us in bringing up a new agent useful for us in bringing up a new agent useful for us in bringing up a new agent from scratch and also taking an existing from scratch and also taking an existing from scratch and also taking an existing agent and and massively improving it. agent and and massively improving it. agent and and massively improving it. And as as we as I mentioned, you know, And as as we as I mentioned, you know, And as as we as I mentioned, you know, simulation is not just about improving simulation is not just about improving simulation is not just about improving uh and sh also talked about this. We are uh and sh also talked about this. We are uh and sh also talked about this. We are derisking a lot of things that could derisking a lot of things that could derisking a lot of things that could have made to production. For instance, have made to production. For instance, have made to production. For instance, we caught a regression uh with we caught a regression uh with we caught a regression uh with simulation that could have made it to simulation that could have made it to simulation that could have made it to production, but simulation caught it.

production, but simulation caught it. production, but simulation caught it. And at the same time, we also caught in And at the same time, we also caught in And at the same time, we also caught in another agent an issue uh which could another agent an issue uh which could another agent an issue uh which could have lowered our self-service rate have lowered our self-service rate have lowered our self-service rate making it less efficient. And all of making it less efficient. And all of making it less efficient. And all of this has resulted in pretty large this has resulted in pretty large this has resulted in pretty large improvements. Uh for instance, one of improvements. Uh for instance, one of improvements. Uh for instance, one of our agents, the TNPS, has 2xed uh thanks our agents, the TNPS, has 2xed uh thanks our agents, the TNPS, has 2xed uh thanks to simulation, robust evals, and to simulation, robust evals, and to simulation, robust evals, and investing in something super principled investing in something super principled investing in something super principled rather than, you know, throwing rather than, you know, throwing rather than, you know, throwing something at the wall and see what something at the wall and see what something at the wall and see what sticks. Um it's been pretty awesome to sticks. Um it's been pretty awesome to sticks. Um it's been pretty awesome to see simulation saving us so much time. see simulation saving us so much time. see simulation saving us so much time. So now the team uses snow globe and So now the team uses snow globe and So now the team uses snow globe and tries different ideas and they're able tries different ideas and they're able tries different ideas and they're able to short circuit launching AB test. They to short circuit launching AB test. They to short circuit launching AB test. They don't launch until they're happy with don't launch until they're happy with don't launch until they're happy with the same output. So imagine if you're the same output. So imagine if you're the same output. So imagine if you're running 10 AB tests a quarter for your running 10 AB tests a quarter for your running 10 AB tests a quarter for your agent or probably more if you have more agent or probably more if you have more agent or probably more if you have more data. Now you can just run a bunch of data. Now you can just run a bunch of data. Now you can just run a bunch of stuff through SIM and launch just one AB stuff through SIM and launch just one AB stuff through SIM and launch just one AB test which kind of shortcircuits test which kind of shortcircuits test which kind of shortcircuits launching the first five or six. And at launching the first five or six. And at launching the first five or six. And at the same time uh this this is often the same time uh this this is often the same time uh this this is often discussed in customer service circles discussed in customer service circles discussed in customer service circles that TNPS can come at the cost of that TNPS can come at the cost of that TNPS can come at the cost of self-service rate. Sometimes there's a self-service rate. Sometimes there's a self-service rate. Sometimes there's a trade-off but I'm happy to report that trade-off but I'm happy to report that trade-off but I'm happy to report that we are not compromising all self- we are not compromising all self- we are not compromising all self- service rate with SIM. We are also able service rate with SIM. We are also able service rate with SIM. We are also able to pretty dramatically improve SSR to pretty dramatically improve SSR to pretty dramatically improve SSR across the board. But in one particular across the board. But in one particular across the board. But in one particular case, we were able to improve it by 4%.

case, we were able to improve it by 4%. case, we were able to improve it by 4%. All of this makes us a much more All of this makes us a much more All of this makes us a much more efficient company and makes sure that we efficient company and makes sure that we efficient company and makes sure that we keep our customers really happy and we keep our customers really happy and we keep our customers really happy and we keep our SSR rates and TPS rates very keep our SSR rates and TPS rates very keep our SSR rates and TPS rates very very high. very high. very high. This is one example that I am most This is one example that I am most This is one example that I am most excited about. As you know, all of you excited about. As you know, all of you excited about. As you know, all of you know open source models have been on a know open source models have been on a know open source models have been on a tear recently. So many of them have come tear recently. So many of them have come tear recently. So many of them have come out. So, we've been trying to test out. So, we've been trying to test out. So, we've been trying to test various open source models in our agent various open source models in our agent various open source models in our agent harnesses and see which ones are the harnesses and see which ones are the harnesses and see which ones are the best. And with SIM, we have the perfect best. And with SIM, we have the perfect best. And with SIM, we have the perfect recipe to just throw a bunch of ideas, recipe to just throw a bunch of ideas, recipe to just throw a bunch of ideas, open source models at our agent harness open source models at our agent harness open source models at our agent harness and eval setup and see which model and eval setup and see which model and eval setup and see which model versions uh you know really really work versions uh you know really really work versions uh you know really really work for us. Uh this has saved us multiple for us. Uh this has saved us multiple for us. Uh this has saved us multiple weeks of effort and probably underelling weeks of effort and probably underelling weeks of effort and probably underelling it. It's it's been it's been pretty it. It's it's been it's been pretty it. It's it's been it's been pretty awesome and uh we have AB test running awesome and uh we have AB test running awesome and uh we have AB test running as I speak uh figuring out are these as I speak uh figuring out are these as I speak uh figuring out are these open source models as good as you know open source models as good as you know open source models as good as you know uh the frontier models. So SIM is is is uh the frontier models. So SIM is is is uh the frontier models. So SIM is is is been super super helpful in helping us been super super helpful in helping us been super super helpful in helping us figure out uh kind of you know spreading figure out uh kind of you know spreading figure out uh kind of you know spreading our cards uh a bit.

our cards uh a bit. our cards uh a bit. >> Um awesome. So um the core three >> Um awesome. So um the core three >> Um awesome. So um the core three takeaways from this talk right is about takeaways from this talk right is about takeaways from this talk right is about um where eval is today and how you can um where eval is today and how you can um where eval is today and how you can really remove a lot of bottlenecks to really remove a lot of bottlenecks to really remove a lot of bottlenecks to it. So we lied when we said earlier that it. So we lied when we said earlier that it. So we lied when we said earlier that there's just one thing you should take there's just one thing you should take there's just one thing you should take away from it. That one thing is still away from it. That one thing is still away from it. That one thing is still essential but there's a few key essential but there's a few key essential but there's a few key downstream things that you can unlock if downstream things that you can unlock if downstream things that you can unlock if you know you adopt it which is the first you know you adopt it which is the first you know you adopt it which is the first is if you generate your evaluation data is if you generate your evaluation data is if you generate your evaluation data in simulation rather than solely relying in simulation rather than solely relying in simulation rather than solely relying on production to get signal on how you on production to get signal on how you on production to get signal on how you know different agents are performing uh know different agents are performing uh know different agents are performing uh you'll be able to undercut or you'll be you'll be able to undercut or you'll be you'll be able to undercut or you'll be able to short circuit a lot of the uh able to short circuit a lot of the uh able to short circuit a lot of the uh bottleneck in in releasing you know bottleneck in in releasing you know bottleneck in in releasing you know versions of your agents much faster. The versions of your agents much faster. The versions of your agents much faster. The second is in order for any of these second is in order for any of these second is in order for any of these gains to really be unlocked uh you know gains to really be unlocked uh you know gains to really be unlocked uh you know you really need to close out the sim you really need to close out the sim you really need to close out the sim toreal gap. So you need to you know set toreal gap. So you need to you know set toreal gap. So you need to you know set up like offline online human review kind up like offline online human review kind up like offline online human review kind of metrics to really understand how sim of metrics to really understand how sim of metrics to really understand how sim performs visav real production data that performs visav real production data that performs visav real production data that you've seen uh so that you you you are you've seen uh so that you you you are you've seen uh so that you you you are able to kind of like trust the results able to kind of like trust the results able to kind of like trust the results of these simulations. Um and the third of these simulations. Um and the third of these simulations. Um and the third is again there's so much excitement is again there's so much excitement is again there's so much excitement around you know like auto research around you know like auto research around you know like auto research self-improving agents RSI etc. uh in an self-improving agents RSI etc. uh in an self-improving agents RSI etc. uh in an enterprise setting when you're building enterprise setting when you're building enterprise setting when you're building an agent, it really does come down to an agent, it really does come down to an agent, it really does come down to two things, data and metrics. If you two things, data and metrics. If you two things, data and metrics. If you have align metrics that are able to have align metrics that are able to have align metrics that are able to really catch the signals you care about really catch the signals you care about really catch the signals you care about and you have a reliable way of and you have a reliable way of and you have a reliable way of generating data that those metrics can generating data that those metrics can generating data that those metrics can give you signal on, it's then very easy give you signal on, it's then very easy give you signal on, it's then very easy to put together a loop of an agent that to put together a loop of an agent that to put together a loop of an agent that you know continuously improves itself you know continuously improves itself you know continuously improves itself from like feedback it receives from all from like feedback it receives from all from like feedback it receives from all of these places. Uh which you know again

of these places. Uh which you know again of these places. Uh which you know again is like where the future of this field is like where the future of this field is like where the future of this field is heading. is heading. is heading. Um, awesome. That's that's us. Uh, thank Um, awesome. That's that's us. Uh, thank Um, awesome. That's that's us. Uh, thank you for coming for the talk. Uh, there's you for coming for the talk. Uh, there's you for coming for the talk. Uh, there's few like contact links and details for few like contact links and details for few like contact links and details for us, you know, if you guys want to get in us, you know, if you guys want to get in us, you know, if you guys want to get in touch and otherwise we'll be around here touch and otherwise we'll be around here touch and otherwise we'll be around here if you guys want to ask us any if you guys want to ask us any if you guys want to ask us any questions.