From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

>> Okay. Oh, yeah. Thanks everyone for >> Okay. Oh, yeah. Thanks everyone for coming. And I know this is the last coming. And I know this is the last coming. And I know this is the last session before lunch. So, thanks for session before lunch. So, thanks for session before lunch. So, thanks for staying here. Let's make it smooth and staying here. Let's make it smooth and staying here. Let's make it smooth and with good vibes. Just as that said. And with good vibes. Just as that said. And with good vibes. Just as that said. And thanks that for introduction. thanks that for introduction. thanks that for introduction. And for inviting me. And for inviting me. And for inviting me. So, yeah. My name is Rustam. I'm leading So, yeah. My name is Rustam. I'm leading So, yeah. My name is Rustam. I'm leading AI platform team uh at Snorkel. And uh AI platform team uh at Snorkel. And uh AI platform team uh at Snorkel. And uh today I want to tell you how to turn today I want to tell you how to turn today I want to tell you how to turn agent traces into agent simulations and agent traces into agent simulations and agent traces into agent simulations and why this why this why this uh becomes the next stage for agent uh becomes the next stage for agent uh becomes the next stage for agent evaluations. evaluations. evaluations. So, three main things that I want you to So, three main things that I want you to So, three main things that I want you to take away from my talk is uh every take away from my talk is uh every take away from my talk is uh every company needs a benchmark. It's the only company needs a benchmark. It's the only company needs a benchmark. It's the only way to reliably evaluate, release, and way to reliably evaluate, release, and way to reliably evaluate, release, and improve your agents. improve your agents. improve your agents. It has to be as close to production as It has to be as close to production as It has to be as close to production as possible. Uh it has to mimic your real possible. Uh it has to mimic your real possible. Uh it has to mimic your real tools, real API services, policies, and tools, real API services, policies, and tools, real API services, policies, and workflows. workflows. workflows. And finally, it has to be part of your And finally, it has to be part of your And finally, it has to be part of your agentic life cycle. agentic life cycle. agentic life cycle. It's not a static benchmark. It's not a static benchmark. It's not a static benchmark. It's a constantly populated data set It's a constantly populated data set It's a constantly populated data set from your production traces.

from your production traces. from your production traces. So, why is Snorkel AI giving this talk? So, why is Snorkel AI giving this talk? So, why is Snorkel AI giving this talk? We are uh data as a service company and We are uh data as a service company and We are uh data as a service company and we uh basically selling uh benchmarks. we uh basically selling uh benchmarks. we uh basically selling uh benchmarks. And we're producing benchmarks at scale. And we're producing benchmarks at scale. And we're producing benchmarks at scale. And for us, benchmark construction is an And for us, benchmark construction is an And for us, benchmark construction is an engineering discipline. We run millions engineering discipline. We run millions engineering discipline. We run millions of agent simulations per month. And uh of agent simulations per month. And uh of agent simulations per month. And uh we learned how to do like uh we learned how to do like uh we learned how to do like uh environment built at scale. environment built at scale. environment built at scale. Working using both agents uh and subject Working using both agents uh and subject Working using both agents uh and subject matter experts to build reliable matter experts to build reliable matter experts to build reliable benchmarks that are close to production benchmarks that are close to production benchmarks that are close to production and uh specific domains. and uh specific domains. and uh specific domains. So, a lot of the time when people say So, a lot of the time when people say So, a lot of the time when people say about agent evaluation, they're focused about agent evaluation, they're focused about agent evaluation, they're focused on traces. And traces are very useful. on traces. And traces are very useful. on traces. And traces are very useful. The usual traces like you can see an The usual traces like you can see an The usual traces like you can see an example on the screen. It basically example on the screen. It basically example on the screen. It basically shows, "Okay, here is the input prompt. shows, "Okay, here is the input prompt. shows, "Okay, here is the input prompt. Here are the actions that agent took, Here are the actions that agent took, Here are the actions that agent took, and here is the agent output." And then and here is the agent output." And then and here is the agent output." And then relation can analyze it and and say, relation can analyze it and and say, relation can analyze it and and say, "Okay, was agent successful or not? Was "Okay, was agent successful or not? Was "Okay, was agent successful or not? Was there any edge case?" there any edge case?" there any edge case?" Uh so, it is useful to find failures in Uh so, it is useful to find failures in Uh so, it is useful to find failures in production, production, production, but it's hard to test different but it's hard to test different but it's hard to test different variants. You can run AB testing, and variants. You can run AB testing, and variants. You can run AB testing, and that's one way of checking different that's one way of checking different that's one way of checking different agent configurations, agent configurations, agent configurations, but it's hard to make sure that but it's hard to make sure that but it's hard to make sure that everything is repeatable because you everything is repeatable because you everything is repeatable because you will get different database state, will get different database state, will get different database state, different tool versions, and so on. So, different tool versions, and so on. So, different tool versions, and so on. So, never fully compare apples to apples.

never fully compare apples to apples. never fully compare apples to apples. Offline simulation turns traces into Offline simulation turns traces into Offline simulation turns traces into repeatable experiments. Now you take repeatable experiments. Now you take repeatable experiments. Now you take production traces, you construct tasks, production traces, you construct tasks, production traces, you construct tasks, and then you can run simulation and then you can run simulation and then you can run simulation benchmark um benchmark um benchmark um with different agent configuration with different agent configuration with different agent configuration offline. offline. offline. And you can compare agents using And you can compare agents using And you can compare agents using different metrics, not just success different metrics, not just success different metrics, not just success rate, but cost, latency, and retries. rate, but cost, latency, and retries. rate, but cost, latency, and retries. And you can run those in parallel. But you can ask, "Okay, but why do we But you can ask, "Okay, but why do we need it? Like we already have public need it? Like we already have public need it? Like we already have public benchmarks." The challenge with public benchmarks." The challenge with public benchmarks." The challenge with public benchmarks is that usually they are benchmarks is that usually they are benchmarks is that usually they are focused on a very specific domains. For focused on a very specific domains. For focused on a very specific domains. For example, SweepBench is focused on like example, SweepBench is focused on like example, SweepBench is focused on like uh fixing GitHub issues, TerminalBench uh fixing GitHub issues, TerminalBench uh fixing GitHub issues, TerminalBench will focus on agent running in terminal, will focus on agent running in terminal, will focus on agent running in terminal, and Co- Bench will focus on computer use and Co- Bench will focus on computer use and Co- Bench will focus on computer use agent. agent. agent. In your case, you want your benchmark to In your case, you want your benchmark to In your case, you want your benchmark to be focused on your company's domain be focused on your company's domain be focused on your company's domain both from perspective of use cases and both from perspective of use cases and both from perspective of use cases and in terms of tooling that your agent has, in terms of tooling that your agent has, in terms of tooling that your agent has, whether it follows the policies that whether it follows the policies that whether it follows the policies that your company uses, and whether you get your company uses, and whether you get your company uses, and whether you get full production environment.

full production environment. full production environment. Basically, public benchmark is useful to Basically, public benchmark is useful to Basically, public benchmark is useful to orient and build your prior, but orient and build your prior, but orient and build your prior, but your private benchmark is useful to your private benchmark is useful to your private benchmark is useful to ship. ship. ship. And a lot of the time public benchmarks And a lot of the time public benchmarks And a lot of the time public benchmarks they're specifically focused on pass they're specifically focused on pass they're specifically focused on pass rate. Every time you see new model rate. Every time you see new model rate. Every time you see new model release, you see performance like pass release, you see performance like pass release, you see performance like pass rate on different benchmarks. rate on different benchmarks. rate on different benchmarks. Which makes sense because it tells us Which makes sense because it tells us Which makes sense because it tells us like like like about the frontier of how good is the about the frontier of how good is the about the frontier of how good is the like new model. like new model. like new model. But when you release agent to But when you release agent to But when you release agent to production, you also care about more production, you also care about more production, you also care about more metrics. You care about cost for solving metrics. You care about cost for solving metrics. You care about cost for solving the task. You care about latency. You the task. You care about latency. You the task. You care about latency. You care about number of retries. care about number of retries. care about number of retries. And by running And by running And by running evaluation offline with in simulations, evaluation offline with in simulations, evaluation offline with in simulations, you can effectively compare apples to you can effectively compare apples to you can effectively compare apples to apples and and iterate on agent. In this case, you can test the full In this case, you can test the full stack of your agent. You're not just stack of your agent. You're not just stack of your agent. You're not just checking, okay, is model checking, okay, is model checking, okay, is model one one one is one model performs better than the is one model performs better than the is one model performs better than the other model.

other model. other model. You check even like thinking talk like You check even like thinking talk like You check even like thinking talk like thinking level. You can change the thinking level. You can change the thinking level. You can change the prompt. You can tune the full hardness prompt. You can tune the full hardness prompt. You can tune the full hardness and skills tools available to agent and skills tools available to agent and skills tools available to agent because in your production, you don't because in your production, you don't because in your production, you don't care about the model, you care about the care about the model, you care about the care about the model, you care about the full system. full system. full system. And here you can configure and test the And here you can configure and test the And here you can configure and test the full system while keeping full system while keeping full system while keeping environment and evaluators the same environment and evaluators the same environment and evaluators the same between runs. And And that raise the point about like what that raise the point about like what that raise the point about like what benchmark is in this case. Because first benchmark is in this case. Because first benchmark is in this case. Because first of all, you can use it to first release of all, you can use it to first release of all, you can use it to first release the agent. You can make sure that it the agent. You can make sure that it the agent. You can make sure that it works. You can handle edge cases well. works. You can handle edge cases well. works. You can handle edge cases well. You can select the optimal model, and You can select the optimal model, and You can select the optimal model, and you can debug the traces. That's how you you can debug the traces. That's how you you can debug the traces. That's how you make sure that it works. make sure that it works. make sure that it works. Next, you can make it right. You can put Next, you can make it right. You can put Next, you can make it right. You can put it as a release gate for your agent and it as a release gate for your agent and it as a release gate for your agent and verify that verify that verify that any change to agent stack in didn't any change to agent stack in didn't any change to agent stack in didn't reduce regression suddenly. reduce regression suddenly. reduce regression suddenly. And iterate on hardness. And finally, And iterate on hardness. And finally, And iterate on hardness. And finally, you can optimize it. You can tune it for you can optimize it. You can tune it for you can optimize it. You can tune it for better cost or latency. Or you can use better cost or latency. Or you can use better cost or latency. Or you can use the traces to even do error training.

the traces to even do error training. the traces to even do error training. Uh later I will say I will share Uh later I will say I will share Uh later I will say I will share link to our website where we have link to our website where we have link to our website where we have example how we used simulation example how we used simulation example how we used simulation environments to fine-tune small plan environments to fine-tune small plan environments to fine-tune small plan model to match performance of large plan model to match performance of large plan model to match performance of large plan model for specific tasks. model for specific tasks. model for specific tasks. Here effectively, that becomes like Here effectively, that becomes like Here effectively, that becomes like trifecta of use cases. trifecta of use cases. trifecta of use cases. Like for you, benchmark becomes part Like for you, benchmark becomes part Like for you, benchmark becomes part part of agent evaluation, part of agent evaluation, part of agent evaluation, becomes part of integration test for becomes part of integration test for becomes part of integration test for agent for release, and it becomes also agent for release, and it becomes also agent for release, and it becomes also training set for agent to improve it. So, I hope I explained why you need So, I hope I explained why you need benchmark. So, now let's take a look benchmark. So, now let's take a look benchmark. So, now let's take a look like how can you actually construct it like how can you actually construct it like how can you actually construct it at scale for your company? Like what is at scale for your company? Like what is at scale for your company? Like what is the anatomy of the benchmark task? the anatomy of the benchmark task? the anatomy of the benchmark task? If you take a step back, like If you take a step back, like If you take a step back, like what is the sequence of running the what is the sequence of running the what is the sequence of running the benchmark? benchmark? benchmark? It's straightforward. Agent gets input It's straightforward. Agent gets input It's straightforward. Agent gets input prompt, it interacts with environment prompt, it interacts with environment prompt, it interacts with environment trying to solve the task with APIs, MCP trying to solve the task with APIs, MCP trying to solve the task with APIs, MCP tools, database, files. tools, database, files. tools, database, files. Then it produces the output, Then it produces the output, Then it produces the output, trace, trace, trace, final state of the environment, and final state of the environment, and final state of the environment, and artifacts, artifacts, artifacts, basically output files. Then we run basically output files. Then we run basically output files. Then we run verifiers, and we can produce the verifiers, and we can produce the verifiers, and we can produce the metrics how was agent able to solve the metrics how was agent able to solve the metrics how was agent able to solve the task, how well did it do, and so on.

task, how well did it do, and so on. task, how well did it do, and so on. The important second part of the task is The important second part of the task is The important second part of the task is Oracle. When Oracle solution runs, Oracle. When Oracle solution runs, Oracle. When Oracle solution runs, it runs through the whole sequence, but it runs through the whole sequence, but it runs through the whole sequence, but just instead of running the real agent, just instead of running the real agent, just instead of running the real agent, it runs Oracle, and it and when we it runs Oracle, and it and when we it runs Oracle, and it and when we construct Oracle ourselves to make sure construct Oracle ourselves to make sure construct Oracle ourselves to make sure that task is solvable in the first that task is solvable in the first that task is solvable in the first place. Because if it's not solvable, place. Because if it's not solvable, place. Because if it's not solvable, agent won't be able to solve it. agent won't be able to solve it. agent won't be able to solve it. So, Oracle is important part of the So, Oracle is important part of the So, Oracle is important part of the task. task. task. If we look at the anatomy of the like of If we look at the anatomy of the like of If we look at the anatomy of the like of the benchmark task, how it looks in the benchmark task, how it looks in the benchmark task, how it looks in terms of files, we can look at one of terms of files, we can look at one of terms of files, we can look at one of the most popular formats nowadays, the most popular formats nowadays, the most popular formats nowadays, Harbor format, which was done by the Harbor format, which was done by the Harbor format, which was done by the same team who same team who same team who uh maintains Terminal Bench. uh maintains Terminal Bench. uh maintains Terminal Bench. In this case, basically, In this case, basically, In this case, basically, it's just three set of files. So, it's just three set of files. So, it's just three set of files. So, basically, what agent sees and interacts basically, what agent sees and interacts basically, what agent sees and interacts with, with, with, instruction.md, classic markdown file, instruction.md, classic markdown file, instruction.md, classic markdown file, environment, which you can see environment, which you can see environment, which you can see Dockerfile, could be Docker Compose in Dockerfile, could be Docker Compose in Dockerfile, could be Docker Compose in case you have multiple Docker case you have multiple Docker case you have multiple Docker containers. containers. containers. You have something that agent doesn't You have something that agent doesn't You have something that agent doesn't see, which are which is Oracle solution see, which are which is Oracle solution see, which are which is Oracle solution and verifiers.

and verifiers. and verifiers. And finally, you have some metadata. And finally, you have some metadata. And finally, you have some metadata. It may uh look very straightforward. It may uh look very straightforward. It may uh look very straightforward. Yeah, I'm saying like, "Okay, simulation Yeah, I'm saying like, "Okay, simulation Yeah, I'm saying like, "Okay, simulation environment is just Dockerfile and a environment is just Dockerfile and a environment is just Dockerfile and a bunch of stuff." But, it is useful bunch of stuff." But, it is useful bunch of stuff." But, it is useful because now you have repeatable way of because now you have repeatable way of because now you have repeatable way of running experiments in agent simulation. running experiments in agent simulation. running experiments in agent simulation. So, now let's dive deeper into main So, now let's dive deeper into main So, now let's dive deeper into main parts of benchmark. First is parts of benchmark. First is parts of benchmark. First is environment. environment. environment. The main challenge with the environment The main challenge with the environment The main challenge with the environment is that effectively it has to be mini is that effectively it has to be mini is that effectively it has to be mini production, but you don't want to run production, but you don't want to run production, but you don't want to run full production for every experiment. full production for every experiment. full production for every experiment. So, you want to make sure that uh your So, you want to make sure that uh your So, you want to make sure that uh your database, API service, tools, and files database, API service, tools, and files database, API service, tools, and files match production, just as like previous match production, just as like previous match production, just as like previous speaker mentioned like speaker mentioned like speaker mentioned like that you don't want your agent to know that you don't want your agent to know that you don't want your agent to know that it's running within simulation. So, that it's running within simulation. So, that it's running within simulation. So, it has to be real. it has to be real. it has to be real. One thing, though, you cannot put real One thing, though, you cannot put real One thing, though, you cannot put real user in your simulation task. So, you user in your simulation task. So, you user in your simulation task. So, you can simulate the user. In this case, can simulate the user. In this case, can simulate the user. In this case, that becomes effectively LLM with its that becomes effectively LLM with its that becomes effectively LLM with its own prompt, which with additional own prompt, which with additional own prompt, which with additional context that can mimic human behavior context that can mimic human behavior context that can mimic human behavior and interaction with your system.

So, in the port system that all these So, in the port system that all these things exist in the environment, things exist in the environment, things exist in the environment, and verifiers just interact with the and verifiers just interact with the and verifiers just interact with the environment afterwards. environment afterwards. environment afterwards. There are certain patterns to make sure There are certain patterns to make sure There are certain patterns to make sure that you can that you can that you can organize environment this way. And think organize environment this way. And think organize environment this way. And think about it as like how you construct about it as like how you construct about it as like how you construct integration tests. Basically, how you integration tests. Basically, how you integration tests. Basically, how you give effectively diff environment to give effectively diff environment to give effectively diff environment to your agent. You don't run the full your agent. You don't run the full your agent. You don't run the full production database. You have a certain production database. You have a certain production database. You have a certain snapshot. You can run side containers snapshot. You can run side containers snapshot. You can run side containers sidecars in this case. So, your uh agent sidecars in this case. So, your uh agent sidecars in this case. So, your uh agent runs in one main uh environment, but runs in one main uh environment, but runs in one main uh environment, but there are other containers which contain there are other containers which contain there are other containers which contain API services, databases, MCP tools, and API services, databases, MCP tools, and API services, databases, MCP tools, and so on, available to your agent. so on, available to your agent. so on, available to your agent. You don't need to have like full You don't need to have like full You don't need to have like full production API services, you can mock production API services, you can mock production API services, you can mock them. I already mentioned simulated them. I already mentioned simulated them. I already mentioned simulated users. users. users. Uh and uh one important piece here is Uh and uh one important piece here is Uh and uh one important piece here is multi-step to handle long horizon task, multi-step to handle long horizon task, multi-step to handle long horizon task, if your agent needs to handle task that if your agent needs to handle task that if your agent needs to handle task that span hours. span hours. span hours. You want to ensure that you have You want to ensure that you have You want to ensure that you have intermediate like steps, and for each intermediate like steps, and for each intermediate like steps, and for each step you have separate prompt, separate step you have separate prompt, separate step you have separate prompt, separate verifiers, and uh you can uh finish verifiers, and uh you can uh finish verifiers, and uh you can uh finish simulation early if you see agent simulation early if you see agent simulation early if you see agent failing.

failing. failing. And basically that enables you to And basically that enables you to And basically that enables you to simulate long-running horizon tasks. simulate long-running horizon tasks. simulate long-running horizon tasks. So, next part is verifiers. So, next part is verifiers. So, next part is verifiers. Like Like Like in traditional sense, usually when in traditional sense, usually when in traditional sense, usually when people speak about verifiers, you just people speak about verifiers, you just people speak about verifiers, you just verify the output. You get agent output, verify the output. You get agent output, verify the output. You get agent output, you verify it, that's it. That's why, you verify it, that's it. That's why, you verify it, that's it. That's why, let's say, how coding works. It's like let's say, how coding works. It's like let's say, how coding works. It's like we just verify the output code. We test, we just verify the output code. We test, we just verify the output code. We test, and so on. and so on. and so on. Here Here Here it's more complex. The way agent it's more complex. The way agent it's more complex. The way agent interacts with the simulation, we get a interacts with the simulation, we get a interacts with the simulation, we get a lot of different data. lot of different data. lot of different data. We get the world, basically how We get the world, basically how We get the world, basically how environment changed in the final environment changed in the final environment changed in the final environment state. What is your database environment state. What is your database environment state. What is your database state? What are the API responses? What state? What are the API responses? What state? What are the API responses? What were user replies? And so on. were user replies? And so on. were user replies? And so on. And your verifier analyzes final state, And your verifier analyzes final state, And your verifier analyzes final state, trace, and artifacts. trace, and artifacts. trace, and artifacts. So, how can you analyze it? So, how can you analyze it? So, how can you analyze it? Effectively, there are multiple ways to Effectively, there are multiple ways to Effectively, there are multiple ways to do it. You can have deterministic do it. You can have deterministic do it. You can have deterministic checks. Basically, uh and that can work checks. Basically, uh and that can work checks. Basically, uh and that can work really well for things like final output really well for things like final output really well for things like final output or tool calls, where like it's very easy or tool calls, where like it's very easy or tool calls, where like it's very easy to check whether it was correct or not.

to check whether it was correct or not. to check whether it was correct or not. It Sometimes you can use LLM as a judge, It Sometimes you can use LLM as a judge, It Sometimes you can use LLM as a judge, or even harness as a judge, or agent as or even harness as a judge, or agent as or even harness as a judge, or agent as a judge a judge a judge to evaluate basically whether the uh to evaluate basically whether the uh to evaluate basically whether the uh trace quality was successful, whether um trace quality was successful, whether um trace quality was successful, whether um planning of the agent was correct, and planning of the agent was correct, and planning of the agent was correct, and so on. In this case, it really depends so on. In this case, it really depends so on. In this case, it really depends on the use case. So, you can use one or on the use case. So, you can use one or on the use case. So, you can use one or another or both another or both another or both depending on what's what works better. depending on what's what works better. depending on what's what works better. And finally, it's important to keep in And finally, it's important to keep in And finally, it's important to keep in mind that you can use sub- uh subject mind that you can use sub- uh subject mind that you can use sub- uh subject matter expert to review some of the matter expert to review some of the matter expert to review some of the traces, some of the outputs. Not for traces, some of the outputs. Not for traces, some of the outputs. Not for everything, everything, everything, but for cases where you see discrepancy but for cases where you see discrepancy but for cases where you see discrepancy in agent behavior and where you want in agent behavior and where you want in agent behavior and where you want human involvement. So, So, final piece, final piece, final piece, how can we kind of organize everything how can we kind of organize everything how can we kind of organize everything together as part of agent release and together as part of agent release and together as part of agent release and agent improvement process? agent improvement process? agent improvement process? So, So, So, can we just start? Not just yet. Can can we just start? Not just yet. Can can we just start? Not just yet. Can something go wrong with benchmark task? something go wrong with benchmark task? something go wrong with benchmark task? 100%. 100%. 100%. Agent can try to reward hack simulation Agent can try to reward hack simulation Agent can try to reward hack simulation environment because it can understand environment because it can understand environment because it can understand that it's in simulation and it can hack that it's in simulation and it can hack that it's in simulation and it can hack it.

it. it. Task could be too simple and like our Task could be too simple and like our Task could be too simple and like our verifiers could be too broad and in this verifiers could be too broad and in this verifiers could be too broad and in this case agent will always pass even if it case agent will always pass even if it case agent will always pass even if it does something incorrectly. It could be does something incorrectly. It could be does something incorrectly. It could be that agent always fail because verifiers that agent always fail because verifiers that agent always fail because verifiers are incorrect. are incorrect. are incorrect. Or it could be that agents not perform Or it could be that agents not perform Or it could be that agents not perform in a stable way and like you have high in a stable way and like you have high in a stable way and like you have high variation of agent success. variation of agent success. variation of agent success. So, all of these are effectively edge So, all of these are effectively edge So, all of these are effectively edge cases that you need to catch during your cases that you need to catch during your cases that you need to catch during your benchmark development because benchmark benchmark development because benchmark benchmark development because benchmark development is an art on its own. We saw development is an art on its own. We saw development is an art on its own. We saw already like hundreds of benchmarks already like hundreds of benchmarks already like hundreds of benchmarks appear over the course of last years, appear over the course of last years, appear over the course of last years, but this is something that but this is something that but this is something that the card culture and engineering the card culture and engineering the card culture and engineering discipline that needs to be built in discipline that needs to be built in discipline that needs to be built in each each each uh engineering team that needs to ship uh engineering team that needs to ship uh engineering team that needs to ship AI agents to production. Because as you saw, effectively Because as you saw, effectively benchmark is software. It's code. It's benchmark is software. It's code. It's benchmark is software. It's code. It's files. You need to treat it as such. You files. You need to treat it as such. You files. You need to treat it as such. You need to have a separate CI pipeline for need to have a separate CI pipeline for need to have a separate CI pipeline for it. it. it. E- E- E- And you can check pretty obvious things And you can check pretty obvious things And you can check pretty obvious things like for example, making sure that all like for example, making sure that all like for example, making sure that all dependencies are pinned or like your dependencies are pinned or like your dependencies are pinned or like your base image is correct or you don't have base image is correct or you don't have base image is correct or you don't have any missing fixtures, any missing fixtures, any missing fixtures, then you can run Oracle then you can run Oracle then you can run Oracle uh solution and make sure that it uh solution and make sure that it uh solution and make sure that it passes. Or like if you don't run Oracle passes. Or like if you don't run Oracle passes. Or like if you don't run Oracle that verifies fail, that verifies fail, that verifies fail, you can run several agent runs you can run several agent runs you can run several agent runs uh on the task and verify that okay, it uh on the task and verify that okay, it uh on the task and verify that okay, it is solvable and it is hard for agent.

is solvable and it is hard for agent. is solvable and it is hard for agent. You can tag the task whether it's You can tag the task whether it's You can tag the task whether it's simple, medium, or hard depending on how simple, medium, or hard depending on how simple, medium, or hard depending on how much time much time much time how many times the agent is successful. how many times the agent is successful. how many times the agent is successful. And finally, you can approve it to make And finally, you can approve it to make And finally, you can approve it to make it part of your benchmark. In this case, In this case, the process for improving agent becomes the process for improving agent becomes the process for improving agent becomes straightforward. You establish baseline straightforward. You establish baseline straightforward. You establish baseline in like on your benchmark, you run in like on your benchmark, you run in like on your benchmark, you run evaluation data set, you see the evaluation data set, you see the evaluation data set, you see the failures where it doesn't perform well, failures where it doesn't perform well, failures where it doesn't perform well, you change one thing, you change one thing, you change one thing, you rerun experiment and where you can you rerun experiment and where you can you rerun experiment and where you can use something like Arize to record your use something like Arize to record your use something like Arize to record your experiments. And then experiments. And then experiments. And then once you fix it, you rerun the full once you fix it, you rerun the full once you fix it, you rerun the full experimentation again. experimentation again. experimentation again. And then you can finally release to And then you can finally release to And then you can finally release to production. What it unlocks is to make sure that you What it unlocks is to make sure that you fix issues correctly. There is a bit of fix issues correctly. There is a bit of fix issues correctly. There is a bit of an anti-pattern in the industry where an anti-pattern in the industry where an anti-pattern in the industry where like folks try to fix things in the like folks try to fix things in the like folks try to fix things in the prompt. prompt. prompt. And they populate the prompt with things And they populate the prompt with things And they populate the prompt with things like, "Never do this or only do that. like, "Never do this or only do that. like, "Never do this or only do that. Never output this critical that." Never output this critical that." Never output this critical that." Which is one way of handling it for Which is one way of handling it for Which is one way of handling it for sure, sure, sure, but with simulation, you control the but with simulation, you control the but with simulation, you control the full stack. You can evaluate the full full stack. You can evaluate the full full stack. You can evaluate the full stack. And you can make sure that fix stack. And you can make sure that fix stack. And you can make sure that fix lives in the correct place. You don't lives in the correct place. You don't lives in the correct place. You don't push everything to the prompt.

push everything to the prompt. push everything to the prompt. You fix hardness if you want if you have You fix hardness if you want if you have You fix hardness if you want if you have context overload. Or you can if there is context overload. Or you can if there is context overload. Or you can if there is a missing procedure, you put it into a missing procedure, you put it into a missing procedure, you put it into skill. skill. skill. Or like if you need to have specific Or like if you need to have specific Or like if you need to have specific output schema, you put it as part of output schema, you put it as part of output schema, you put it as part of structured output. So, So, once agent is in production, once agent is in production, once agent is in production, effectively, you will have two loops. effectively, you will have two loops. effectively, you will have two loops. One loop for benchmark expansion. One loop for benchmark expansion. One loop for benchmark expansion. You take uh, observability traces uh, You take uh, observability traces uh, You take uh, observability traces uh, traces from observability basically traces from observability basically traces from observability basically using something like a rise. using something like a rise. using something like a rise. You record failures and use failures to You record failures and use failures to You record failures and use failures to build your benchmark further. build your benchmark further. build your benchmark further. Then you have simulation runner that Then you have simulation runner that Then you have simulation runner that will run experiments on your on extended will run experiments on your on extended will run experiments on your on extended benchmark on a new or on new agent benchmark on a new or on new agent benchmark on a new or on new agent config. config. config. Uh, we'll record these experiments. Uh, we'll record these experiments. Uh, we'll record these experiments. Then you can use it for as a release Then you can use it for as a release Then you can use it for as a release gate whether, gate whether, gate whether, uh, agent performs significantly well uh, agent performs significantly well uh, agent performs significantly well and finally release it to production. and finally release it to production. and finally release it to production. Uh, in this case it's important that you Uh, in this case it's important that you Uh, in this case it's important that you have system like it's important to have system like it's important to have system like it's important to under- under- under- to make sure that your observability to make sure that your observability to make sure that your observability piece and experimentation piece are piece and experimentation piece are piece and experimentation piece are connected.

connected. connected. Because they're part they're two sides Because they're part they're two sides Because they're part they're two sides of the same coin. Um, Um, so yeah, to summarize my talk, everyone so yeah, to summarize my talk, everyone so yeah, to summarize my talk, everyone needs every company needs a benchmark. needs every company needs a benchmark. needs every company needs a benchmark. Traces are useful for like the finding Traces are useful for like the finding Traces are useful for like the finding edge cases in production, but simulation edge cases in production, but simulation edge cases in production, but simulation helps to, uh, test what would happen. helps to, uh, test what would happen. helps to, uh, test what would happen. And finally, uh, you want, uh, your, um, And finally, uh, you want, uh, your, um, And finally, uh, you want, uh, your, um, benchmark to be part of your, uh, agent benchmark to be part of your, uh, agent benchmark to be part of your, uh, agent ops, uh, loop. ops, uh, loop. ops, uh, loop. Thank you very much for for your time Thank you very much for for your time Thank you very much for for your time and, uh, I think maybe I will have and, uh, I think maybe I will have and, uh, I think maybe I will have answer for one question, but will be answer for one question, but will be answer for one question, but will be happy to answer any questions outside happy to answer any questions outside happy to answer any questions outside and please check our booth, uh, if you and please check our booth, uh, if you and please check our booth, uh, if you have any questions about benchmarks. have any questions about benchmarks. have any questions about benchmarks. >> [applause] >> All right. >> All right. A couple questions. A couple questions. A couple questions. All right. All right. All right. >> A great talk. I have a question about >> A great talk. I have a question about >> A great talk. I have a question about structuring the benchmark. How many structuring the benchmark. How many structuring the benchmark. How many examples you should include ideally? And examples you should include ideally? And examples you should include ideally? And do you split them into, do you split them into, do you split them into, uh, like a train test split? If so, like uh, like a train test split? If so, like uh, like a train test split? If so, like how do you recommend structuring how do you recommend structuring how do you recommend structuring examples, examples, examples, um, um, um, across those two splits?

across those two splits? across those two splits? >> Great question. So, uh, the question was >> Great question. So, uh, the question was >> Great question. So, uh, the question was basically how can we structure to for basically how can we structure to for basically how can we structure to for training and validation? And just as training and validation? And just as training and validation? And just as previous speakers uh shared, basically previous speakers uh shared, basically previous speakers uh shared, basically they also had a pattern of like having they also had a pattern of like having they also had a pattern of like having train validation split. This is very train validation split. This is very train validation split. This is very close to traditional machine learning close to traditional machine learning close to traditional machine learning where we want to have standalone data where we want to have standalone data where we want to have standalone data set that agent didn't see. set that agent didn't see. set that agent didn't see. But basically where we can verify agent But basically where we can verify agent But basically where we can verify agent config. I think the classic approach config. I think the classic approach config. I think the classic approach applies when like you want to have applies when like you want to have applies when like you want to have 80/20%. 80/20%. 80/20%. Always depends on the use case. But uh Always depends on the use case. But uh Always depends on the use case. But uh you do want to have a standout data set you do want to have a standout data set you do want to have a standout data set that agent didn't see through the that agent didn't see through the that agent didn't see through the experimentation process. experimentation process. experimentation process. >> Awesome. Next question. >> Awesome. Next question. >> Awesome. Next question. >> Um so when you create the benchmarks, um >> Um so when you create the benchmarks, um >> Um so when you create the benchmarks, um what data do you include in the what data do you include in the what data do you include in the benchmark? Do you include production benchmark? Do you include production benchmark? Do you include production runs that you've handpicked into the runs that you've handpicked into the runs that you've handpicked into the benchmark? Or if your agent is not yet benchmark? Or if your agent is not yet benchmark? Or if your agent is not yet in production, do you create in production, do you create in production, do you create uh uh uh data sets or problems to solve for the data sets or problems to solve for the data sets or problems to solve for the agent in the benchmark? And if you're if agent in the benchmark? And if you're if agent in the benchmark? And if you're if you're like handpicking problems for the you're like handpicking problems for the you're like handpicking problems for the agent to solve, um agent to solve, um agent to solve, um how do you make sure you have like how do you make sure you have like how do you make sure you have like enough coverage uh similar to what you'd enough coverage uh similar to what you'd enough coverage uh similar to what you'd see in production?

see in production? see in production? >> Great question. So the question is how >> Great question. So the question is how >> Great question. So the question is how to ensure coverage and distribution of to ensure coverage and distribution of to ensure coverage and distribution of benchmark is a very important piece. So benchmark is a very important piece. So benchmark is a very important piece. So effectively you want to keep effectively you want to keep effectively you want to keep keep make sure like you have both. You keep make sure like you have both. You keep make sure like you have both. You have bread and butter use cases, have bread and butter use cases, have bread and butter use cases, basically covering all main use cases basically covering all main use cases basically covering all main use cases that work, but also making sure you have that work, but also making sure you have that work, but also making sure you have edge cases. Basically how you can make edge cases. Basically how you can make edge cases. Basically how you can make sure like your agent can handle edge sure like your agent can handle edge sure like your agent can handle edge case when tools fail or if like there is case when tools fail or if like there is case when tools fail or if like there is problem with database and so on. problem with database and so on. problem with database and so on. So you want to have both uh So you want to have both uh So you want to have both uh it basically think about it as it basically think about it as it basically think about it as integration test. You you have happy integration test. You you have happy integration test. You you have happy path, but you also have edge cases. path, but you also have edge cases. path, but you also have edge cases. >> Would you simulate agent with LLMs or >> Would you simulate agent with LLMs or >> Would you simulate agent with LLMs or would you create hand-crafted business? would you create hand-crafted business? would you create hand-crafted business? >> Yeah, uh >> Yeah, uh >> Yeah, uh great question. Basically similar to us great question. Basically similar to us great question. Basically similar to us handcrafting, uh I think basically I handcrafting, uh I think basically I handcrafting, uh I think basically I mean mean mean people use agents to write code. So a people use agents to write code. So a people use agents to write code. So a lot of things here can be automated. So lot of things here can be automated. So lot of things here can be automated. So the what is handcrafted kind of changes. the what is handcrafted kind of changes. the what is handcrafted kind of changes. Like the the most important piece that Like the the most important piece that Like the the most important piece that you can provide is like the build you can provide is like the build you can provide is like the build environment once and then build the environment once and then build the environment once and then build the context uh context uh context uh for basically that mimics your for basically that mimics your for basically that mimics your production. production. production. >> Um >> Um >> Um you mentioned sometimes you use you mentioned sometimes you use you mentioned sometimes you use LLM or sometimes use human expert to LLM or sometimes use human expert to LLM or sometimes use human expert to build verifiers.

build verifiers. build verifiers. Can you elaborate on that? What's the Can you elaborate on that? What's the Can you elaborate on that? What's the best practice over there? best practice over there? best practice over there? >> Great question. So, in our case >> Great question. So, in our case >> Great question. So, in our case basically, we have a lot of subject basically, we have a lot of subject basically, we have a lot of subject matter experts, so we do things at matter experts, so we do things at matter experts, so we do things at scale. So, the important piece here is scale. So, the important piece here is scale. So, the important piece here is to you don't need subject matter experts to you don't need subject matter experts to you don't need subject matter experts to review everything, but you to review everything, but you to review everything, but you specifically want to find cases where specifically want to find cases where specifically want to find cases where there's disagreement between there's disagreement between there's disagreement between agent and different verifiers. If you agent and different verifiers. If you agent and different verifiers. If you you think that basically task was you think that basically task was you think that basically task was supposed to be solved, but agent somehow supposed to be solved, but agent somehow supposed to be solved, but agent somehow marks it as like not solved correctly or marks it as like not solved correctly or marks it as like not solved correctly or like it marks that trace wasn't optimal like it marks that trace wasn't optimal like it marks that trace wasn't optimal enough. enough. enough. And this is where you want to have And this is where you want to have And this is where you want to have subject matter expert that can basically subject matter expert that can basically subject matter expert that can basically tune the agent that does the review.