Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

>> Hey everyone. >> Hey everyone. I'm Vic and I lead applied research at I'm Vic and I lead applied research at I'm Vic and I lead applied research at LangChain and I'm going to talk about LangChain and I'm going to talk about LangChain and I'm going to talk about something that I think is sexy, which is something that I think is sexy, which is something that I think is sexy, which is data mining, but it's not as sexy as data mining, but it's not as sexy as data mining, but it's not as sexy as LLM, so we're going to try to like make LLM, so we're going to try to like make LLM, so we're going to try to like make it sexy together. And the problem that it sexy together. And the problem that it sexy together. And the problem that we're going to talk about today is how we're going to talk about today is how we're going to talk about today is how do we continuously improve agents, but do we continuously improve agents, but do we continuously improve agents, but how do we do that via data? how do we do that via data? how do we do that via data? So, to start, I'm going to tell a little So, to start, I'm going to tell a little So, to start, I'm going to tell a little story that I think maybe a lot of us story that I think maybe a lot of us story that I think maybe a lot of us have felt before. Like, I ran my agent, have felt before. Like, I ran my agent, have felt before. Like, I ran my agent, it did a bunch of things, it did a bunch of things, it did a bunch of things, it made some mistakes. it made some mistakes. it made some mistakes. Now, I ask someone like, what do I Now, I ask someone like, what do I Now, I ask someone like, what do I actually do about that? Like, I have all actually do about that? Like, I have all actually do about that? Like, I have all this data, made some mistakes, what now? this data, made some mistakes, what now? this data, made some mistakes, what now? Basically, what we're going to do today Basically, what we're going to do today Basically, what we're going to do today is we're going to motivate a recipe for is we're going to motivate a recipe for is we're going to motivate a recipe for what we should do to continuously what we should do to continuously what we should do to continuously improve agents over time, and then I'm improve agents over time, and then I'm improve agents over time, and then I'm going to talk from some lived experience going to talk from some lived experience going to talk from some lived experience and like some stuff that we help and like some stuff that we help and like some stuff that we help customers do to run this over customers do to run this over customers do to run this over large-scale trace data. large-scale trace data. large-scale trace data. So, the first step in building a So, the first step in building a So, the first step in building a successful agent is shipping it. So, if successful agent is shipping it. So, if successful agent is shipping it. So, if you put it out into the real world, then you put it out into the real world, then you put it out into the real world, then it can operate in environments and then it can operate in environments and then it can operate in environments and then you can get feedback from what it's you can get feedback from what it's you can get feedback from what it's doing.

doing. doing. The second step is collect a ton of The second step is collect a ton of The second step is collect a ton of traces. So, agents operate in the traces. So, agents operate in the traces. So, agents operate in the environment every single time they environment every single time they environment every single time they operate, they do tool calls, they have operate, they do tool calls, they have operate, they do tool calls, they have output messages, they call APIs, they output messages, they call APIs, they output messages, they call APIs, they use CLIs. All of that generates data and use CLIs. All of that generates data and use CLIs. All of that generates data and we want to store all of that so we can we want to store all of that so we can we want to store all of that so we can like do stuff with it. like do stuff with it. like do stuff with it. The next thing is the data mining in The next thing is the data mining in The next thing is the data mining in this talk, which is once we have tons of this talk, which is once we have tons of this talk, which is once we have tons of trace data, maybe gigabytes, maybe trace data, maybe gigabytes, maybe trace data, maybe gigabytes, maybe terabytes, depending on like how many terabytes, depending on like how many terabytes, depending on like how many agents you're shipping, we're going to agents you're shipping, we're going to agents you're shipping, we're going to do data mining over that. And I promise do data mining over that. And I promise do data mining over that. And I promise I will tell you exactly what data mining I will tell you exactly what data mining I will tell you exactly what data mining we're going do, uh but we're going to do we're going do, uh but we're going to do we're going do, uh but we're going to do some over it. some over it. some over it. And then the fun part, which is I And then the fun part, which is I And then the fun part, which is I collected that data, I read it, I collected that data, I read it, I collected that data, I read it, I curated it, and now we actually need to curated it, and now we actually need to curated it, and now we actually need to run the experiments in a data-driven way run the experiments in a data-driven way run the experiments in a data-driven way to see, "Hey, is this new prompt, or is to see, "Hey, is this new prompt, or is to see, "Hey, is this new prompt, or is this new tool, or is this new this new tool, or is this new this new tool, or is this new orchestration, or is this new loop, is orchestration, or is this new loop, is orchestration, or is this new loop, is it actually improving things based on it actually improving things based on it actually improving things based on the previous traces that I've seen?" And this is maybe a bit of a hot take, And this is maybe a bit of a hot take, but continual learning is super hot but continual learning is super hot but continual learning is super hot right now. I'm talking about it, this right now. I'm talking about it, this right now. I'm talking about it, this whole room is going to hear about it for whole room is going to hear about it for whole room is going to hear about it for the next like 5-6 hours. Um but there's the next like 5-6 hours. Um but there's the next like 5-6 hours. Um but there's a very tight coupling between what a very tight coupling between what a very tight coupling between what observability is and what continual observability is and what continual observability is and what continual learning is. And the main reason for learning is. And the main reason for learning is. And the main reason for that is that that is that that is that agents that operate in environments, agents that operate in environments, agents that operate in environments, they produce trace data, and what they produce trace data, and what they produce trace data, and what continual learning for agents and continual learning for agents and continual learning for agents and continual learning for humans basically continual learning for humans basically continual learning for humans basically is is I do a bunch of stuff in the is is I do a bunch of stuff in the is is I do a bunch of stuff in the world, I think about what I did, and world, I think about what I did, and world, I think about what I did, and then I need to update my definition,

then I need to update my definition, then I need to update my definition, like my knowledge, stuff I write down, like my knowledge, stuff I write down, like my knowledge, stuff I write down, in order to respond to the feedback from in order to respond to the feedback from in order to respond to the feedback from the environment. And if you're continual the environment. And if you're continual the environment. And if you're continual learning company, you need traces, and learning company, you need traces, and learning company, you need traces, and if you have traces, then you can try to if you have traces, then you can try to if you have traces, then you can try to do continual learning over your agents. I had to put in a meme because I had to put in a meme because if you look at your data, then you can if you look at your data, then you can if you look at your data, then you can be like Will Hunting if anyone's seen be like Will Hunting if anyone's seen be like Will Hunting if anyone's seen the movie, where like everything is the movie, where like everything is the movie, where like everything is super super easy, and you can like super super easy, and you can like super super easy, and you can like improve over time, and I promised Emma I improve over time, and I promised Emma I improve over time, and I promised Emma I would put this in there, so putting it would put this in there, so putting it would put this in there, so putting it in there. in there. in there. Cool. So, why am I talking a bunch about Cool. So, why am I talking a bunch about Cool. So, why am I talking a bunch about traces anyway? So, traces anyway? So, traces anyway? So, I'm sure a ton of us were software I'm sure a ton of us were software I'm sure a ton of us were software engineers before, we're software engineers before, we're software engineers before, we're software engineers now, and on the left we have a engineers now, and on the left we have a engineers now, and on the left we have a code block, and we can sort of like read code block, and we can sort of like read code block, and we can sort of like read the code, and in my head, I can almost the code, and in my head, I can almost the code, and in my head, I can almost reason over what this code does. I can reason over what this code does. I can reason over what this code does. I can see the functions, I can see like how see the functions, I can see like how see the functions, I can see like how they call each other. they call each other. they call each other. I can roughly understand the logic in I can roughly understand the logic in I can roughly understand the logic in Python. Um that doesn't exactly exist in Python. Um that doesn't exactly exist in Python. Um that doesn't exactly exist in agent world because agents have prompts, agent world because agents have prompts, agent world because agents have prompts, they have tools, they have skills, they they have tools, they have skills, they they have tools, they have skills, they have hooks, they have middlewares, some have hooks, they have middlewares, some have hooks, they have middlewares, some agents call other agents and I agents call other agents and I agents call other agents and I orchestrate them in swarms. It's really orchestrate them in swarms. It's really orchestrate them in swarms. It's really really hard for humans to reason about really hard for humans to reason about really hard for humans to reason about how certain prompts that they change are how certain prompts that they change are how certain prompts that they change are actually going to affect agent behavior actually going to affect agent behavior actually going to affect agent behavior at scale. And this also varies between at scale. And this also varies between at scale. And this also varies between the different domains that you're doing the different domains that you're doing the different domains that you're doing it on. So, a prompt change and you're it on. So, a prompt change and you're it on. So, a prompt change and you're using for the medical domain is going to using for the medical domain is going to using for the medical domain is going to be like completely different than a be like completely different than a be like completely different than a prompt change that you want to do for prompt change that you want to do for prompt change that you want to do for the law domain.

the law domain. the law domain. And in general, over the last four years And in general, over the last four years And in general, over the last four years since the ChatGPT moment, we've started since the ChatGPT moment, we've started since the ChatGPT moment, we've started trading determinism for autonomy. And in trading determinism for autonomy. And in trading determinism for autonomy. And in that shift, sort of what we need to do that shift, sort of what we need to do that shift, sort of what we need to do is create tools and create systems to is create tools and create systems to is create tools and create systems to still understand agents when they're still understand agents when they're still understand agents when they're autonomously operating in environments. autonomously operating in environments. autonomously operating in environments. So, I talked about traces. Um So, I talked about traces. Um So, I talked about traces. Um why like why should you read them? And why like why should you read them? And why like why should you read them? And at LangChain, what do we actually do at LangChain, what do we actually do at LangChain, what do we actually do when we're reading traces? So, we when we're reading traces? So, we when we're reading traces? So, we centralize a bunch of our data, so we centralize a bunch of our data, so we centralize a bunch of our data, so we put everything in a tracing project and put everything in a tracing project and put everything in a tracing project and this is usually either like per agent or this is usually either like per agent or this is usually either like per agent or like centralized across all of our like centralized across all of our like centralized across all of our agents. And then what we do is we send agents. And then what we do is we send agents. And then what we do is we send agents to read traces from other agents, agents to read traces from other agents, agents to read traces from other agents, right? And then we look for a bunch of right? And then we look for a bunch of right? And then we look for a bunch of different things. And we might ask for, different things. And we might ask for, different things. And we might ask for, "Hey, like find a bunch of like good and "Hey, like find a bunch of like good and "Hey, like find a bunch of like good and bad interactions where like users got bad interactions where like users got bad interactions where like users got upset or like users are like really upset or like users are like really upset or like users are like really happy." happy." happy." Um Um Um another question I might ask is uh this another question I might ask is uh this another question I might ask is uh this is a technical question like "Agents now is a technical question like "Agents now is a technical question like "Agents now run for millions of tokens. Does the run for millions of tokens. Does the run for millions of tokens. Does the agent get really dumb after the first agent get really dumb after the first agent get really dumb after the first compaction? After the second compaction?

compaction? After the second compaction? compaction? After the second compaction? Does it never get dumb?" Like how do we Does it never get dumb?" Like how do we Does it never get dumb?" Like how do we actually answer these questions? We need actually answer these questions? We need actually answer these questions? We need to do it by actually looking at the to do it by actually looking at the to do it by actually looking at the traces. traces. traces. And then the the other thing is like if And then the the other thing is like if And then the the other thing is like if I look at the traces, then I can try to I look at the traces, then I can try to I look at the traces, then I can try to prove some counterfactuals, which is prove some counterfactuals, which is prove some counterfactuals, which is "Hey, like I ran GPT 5.5 for this and I "Hey, like I ran GPT 5.5 for this and I "Hey, like I ran GPT 5.5 for this and I heard like GLM is really good. What heard like GLM is really good. What heard like GLM is really good. What happens if I run GLM 5.2 for this task happens if I run GLM 5.2 for this task happens if I run GLM 5.2 for this task and how do I compare them? Metrics, and how do I compare them? Metrics, and how do I compare them? Metrics, awesome. The The trace level captures awesome. The The trace level captures awesome. The The trace level captures the actual like behavior that users see. the actual like behavior that users see. the actual like behavior that users see. So, that's also like very helpful for So, that's also like very helpful for So, that's also like very helpful for seeing behavior like fine grain scales. And the way that we sort of think about And the way that we sort of think about the data that's being generated by the data that's being generated by the data that's being generated by agents is that the data that we see agents is that the data that we see agents is that the data that we see today is going to be the smallest that today is going to be the smallest that today is going to be the smallest that humans have ever seen in their entire humans have ever seen in their entire humans have ever seen in their entire lives because we're in this massive lives because we're in this massive lives because we're in this massive exponential shift to our agents are exponential shift to our agents are exponential shift to our agents are doing more and more work in the economy. doing more and more work in the economy. doing more and more work in the economy. And what that means is like the amount And what that means is like the amount And what that means is like the amount of data that humans have produced in our of data that humans have produced in our of data that humans have produced in our entire lifetime will soon be eclipsed by entire lifetime will soon be eclipsed by entire lifetime will soon be eclipsed by agents running on like year scales and agents running on like year scales and agents running on like year scales and then 6-month scales and 3-month scales then 6-month scales and 3-month scales then 6-month scales and 3-month scales and then maybe every day, right? Um and and then maybe every day, right? Um and and then maybe every day, right? Um and to understand a ton of that data, to understand a ton of that data, to understand a ton of that data, roughly what we need to do is contend roughly what we need to do is contend roughly what we need to do is contend with a couple problems. There's more, with a couple problems. There's more, with a couple problems. There's more, but these are the two that I'm going to but these are the two that I'm going to but these are the two that I'm going to focus on. So, focus on. So, focus on. So, one, uh reading traces at scale is super one, uh reading traces at scale is super one, uh reading traces at scale is super expensive, uh especially if you have expensive, uh especially if you have expensive, uh especially if you have millions of traces and if you have millions of traces and if you have millions of traces and if you have millions of tokens per trace, right? Um millions of tokens per trace, right? Um millions of tokens per trace, right? Um think of it as like an input token cost.

think of it as like an input token cost. think of it as like an input token cost. You can like literally multiply the You can like literally multiply the You can like literally multiply the input token cost uh times the number of input token cost uh times the number of input token cost uh times the number of traces times like how big each trace is traces times like how big each trace is traces times like how big each trace is on average, right? Um the the other on average, right? Um the the other on average, right? Um the the other thing is [clears throat] if I have a thing is [clears throat] if I have a thing is [clears throat] if I have a super long interaction with a coding super long interaction with a coding super long interaction with a coding agent like Cloud Code or Codex or like agent like Cloud Code or Codex or like agent like Cloud Code or Codex or like deep agents, um deep agents, um deep agents, um I can't even read that trace with I can't even read that trace with I can't even read that trace with another agent because that that context another agent because that that context another agent because that that context like doesn't fit in memory, right? So, like doesn't fit in memory, right? So, like doesn't fit in memory, right? So, it's like we we need to develop systems it's like we we need to develop systems it's like we we need to develop systems so I can sort of treat that context as so I can sort of treat that context as so I can sort of treat that context as like an external object and then I can like an external object and then I can like an external object and then I can sort of query into it, right? So, we we sort of query into it, right? So, we we sort of query into it, right? So, we we need to build agents to efficiently mine need to build agents to efficiently mine need to build agents to efficiently mine data from other agents and it's it's no data from other agents and it's it's no data from other agents and it's it's no longer as simple as just like feeding longer as simple as just like feeding longer as simple as just like feeding the data into context and there's like the data into context and there's like the data into context and there's like tricks that we'll sort of talk about uh tricks that we'll sort of talk about uh tricks that we'll sort of talk about uh to to do that well. to to do that well. to to do that well. Great. So, one of the things that I Great. So, one of the things that I Great. So, one of the things that I think is really really cool in the last think is really really cool in the last think is really really cool in the last 6 months is that open models have 6 months is that open models have 6 months is that open models have basically hit an inflection point in basically hit an inflection point in basically hit an inflection point in intelligence that we at LangChain don't intelligence that we at LangChain don't intelligence that we at LangChain don't reach for the frontier models for every reach for the frontier models for every reach for the frontier models for every single use case. We're quite conscious single use case. We're quite conscious single use case. We're quite conscious about what is the minimum level of about what is the minimum level of about what is the minimum level of intelligence that I need to do any given intelligence that I need to do any given intelligence that I need to do any given task. And like practically speaking, task. And like practically speaking, task. And like practically speaking, honestly, yes, we start with Opus, we honestly, yes, we start with Opus, we honestly, yes, we start with Opus, we start with 55 because we just want to start with 55 because we just want to start with 55 because we just want to know if the task is even possible. But know if the task is even possible. But know if the task is even possible. But then once we reach that sort of like then once we reach that sort of like then once we reach that sort of like waterline, then we like look back at waterline, then we like look back at waterline, then we like look back at those traces and we see, "Hey, can we those traces and we see, "Hey, can we those traces and we see, "Hey, can we use an open model to do the same thing?"

use an open model to do the same thing?" use an open model to do the same thing?" So, this is a bunch of work that we did So, this is a bunch of work that we did So, this is a bunch of work that we did with Harvey and then their lab legal with Harvey and then their lab legal with Harvey and then their lab legal benchmark. Basically, what we're looking benchmark. Basically, what we're looking benchmark. Basically, what we're looking at is at is at is can I match the trace judging capability can I match the trace judging capability can I match the trace judging capability of Opus with an open cheaper model? And of Opus with an open cheaper model? And of Opus with an open cheaper model? And the answer is roughly yes at like an the answer is roughly yes at like an the answer is roughly yes at like an order or like two orders of magnitude order or like two orders of magnitude order or like two orders of magnitude cheaper. And like the way we do that is cheaper. And like the way we do that is cheaper. And like the way we do that is we try a bunch of models, we try a bunch of models, we try a bunch of models, we do a bunch of like harness we do a bunch of like harness we do a bunch of like harness engineering, and the harness engineering engineering, and the harness engineering engineering, and the harness engineering is informed by a bunch of the traces is informed by a bunch of the traces is informed by a bunch of the traces that we read. So, it's like, "Hey, like that we read. So, it's like, "Hey, like that we read. So, it's like, "Hey, like Opus reasons about things in this way. Opus reasons about things in this way. Opus reasons about things in this way. Maybe that's because of the prompt. Maybe that's because of the prompt. Maybe that's because of the prompt. Maybe Opus is just smarter, which it is, Maybe Opus is just smarter, which it is, Maybe Opus is just smarter, which it is, than a bunch of the open models, but than a bunch of the open models, but than a bunch of the open models, but that might mean I need to give it a that might mean I need to give it a that might mean I need to give it a little bit more guidance so it can reach little bit more guidance so it can reach little bit more guidance so it can reach the sort of same intelligence level at the sort of same intelligence level at the sort of same intelligence level at like a much like a much like a much much lower cost." much lower cost." much lower cost." And the the other thing that we sort of And the the other thing that we sort of And the the other thing that we sort of look at is like harness engineering is look at is like harness engineering is look at is like harness engineering is amazing. amazing. amazing. You get instant feedback and you can You get instant feedback and you can You get instant feedback and you can sort of like run on your evals, but sort of like run on your evals, but sort of like run on your evals, but eventually what we find is you hit a eventually what we find is you hit a eventually what we find is you hit a threshold of intelligence where it's threshold of intelligence where it's threshold of intelligence where it's like "If I keep tweaking this prompt, like "If I keep tweaking this prompt, like "If I keep tweaking this prompt, I'm not going to get too much more out I'm not going to get too much more out I'm not going to get too much more out of it." And once we reach that point, we of it." And once we reach that point, we of it." And once we reach that point, we sort of look at, "Okay, can I actually sort of look at, "Okay, can I actually sort of look at, "Okay, can I actually like fine-tune the model on my like fine-tune the model on my like fine-tune the model on my domain-specific task?" And can I like domain-specific task?" And can I like domain-specific task?" And can I like make it better on those tasks? And what make it better on those tasks? And what make it better on those tasks? And what we find is if we take like base models we find is if we take like base models we find is if we take like base models and we tune them on like very specific and we tune them on like very specific and we tune them on like very specific vertical tasks, which is what a lot of vertical tasks, which is what a lot of vertical tasks, which is what a lot of our customers do, they don't really care our customers do, they don't really care our customers do, they don't really care about the entire variance of tasks. like about the entire variance of tasks. like about the entire variance of tasks. like they care about what their customers they care about what their customers they care about what their customers care about. So, if we focus on that care about. So, if we focus on that care about. So, if we focus on that narrow set of tasks, then we can narrow set of tasks, then we can narrow set of tasks, then we can fine-tune base models to sort of like fine-tune base models to sort of like fine-tune base models to sort of like reach and then also go beyond frontier

reach and then also go beyond frontier reach and then also go beyond frontier performance. And I think one sort of performance. And I think one sort of performance. And I think one sort of like small thing I'll mention as a lot like small thing I'll mention as a lot like small thing I'll mention as a lot of people are getting into fine-tuning of people are getting into fine-tuning of people are getting into fine-tuning is that another sort of like economic is that another sort of like economic is that another sort of like economic decision is that you can move from token decision is that you can move from token decision is that you can move from token costs to hardware costs. And this is costs to hardware costs. And this is costs to hardware costs. And this is like can be a really big change, right? like can be a really big change, right? like can be a really big change, right? Cuz like you're very used to hey, like a Cuz like you're very used to hey, like a Cuz like you're very used to hey, like a million tokens cost this much, not as million tokens cost this much, not as million tokens cost this much, not as much like this cluster sort of costs much like this cluster sort of costs much like this cluster sort of costs this much. But for like very high this much. But for like very high this much. But for like very high inference workloads, we find it to be inference workloads, we find it to be inference workloads, we find it to be way cheaper just to like run a cluster way cheaper just to like run a cluster way cheaper just to like run a cluster and I get like unlimited inference on and I get like unlimited inference on and I get like unlimited inference on that cluster. I don't have to worry that cluster. I don't have to worry that cluster. I don't have to worry about tokens, but I can just do the about tokens, but I can just do the about tokens, but I can just do the calculation of like, hey, calculation of like, hey, calculation of like, hey, um this will end up being cheaper and um this will end up being cheaper and um this will end up being cheaper and then I can spin it down when I don't then I can spin it down when I don't then I can spin it down when I don't need it. Cool. And I said all of this um Cool. And I said all of this um so we obviously like built a product to so we obviously like built a product to so we obviously like built a product to do that. Uh I won't shill it too much, do that. Uh I won't shill it too much, do that. Uh I won't shill it too much, but it's LangSplat engine. Uh basically, but it's LangSplat engine. Uh basically, but it's LangSplat engine. Uh basically, this product is trying to automate this this product is trying to automate this this product is trying to automate this loop for you, which is if you have any loop for you, which is if you have any loop for you, which is if you have any volume of trace data and you're looking volume of trace data and you're looking volume of trace data and you're looking for something that trace data or you for something that trace data or you for something that trace data or you want to generate e-vals from that trace want to generate e-vals from that trace want to generate e-vals from that trace data or you want to like generate data or you want to like generate data or you want to like generate feedback for like humans to read from feedback for like humans to read from feedback for like humans to read from that trace data, it will go read all of that trace data, it will go read all of that trace data, it will go read all of it, it'll like find issues, it'll it, it'll like find issues, it'll it, it'll like find issues, it'll agentically search over it, and they can agentically search over it, and they can agentically search over it, and they can like prepare data sets for you to do like prepare data sets for you to do like prepare data sets for you to do something after. And a bit of a leader, something after. And a bit of a leader, something after. And a bit of a leader, um um um what that something basically is is the what that something basically is is the what that something basically is is the outputs of this trace mining exercise.

outputs of this trace mining exercise. outputs of this trace mining exercise. So, So, So, there's like three things that I there's like three things that I there's like three things that I mentioned here uh which we see a bunch mentioned here uh which we see a bunch mentioned here uh which we see a bunch and we kind of put into the product. So, and we kind of put into the product. So, and we kind of put into the product. So, one is distillation and fine-tuning, one is distillation and fine-tuning, one is distillation and fine-tuning, which is let's say I'm running GLM 5.2. which is let's say I'm running GLM 5.2. which is let's say I'm running GLM 5.2. It's doing great, but I think that I can It's doing great, but I think that I can It's doing great, but I think that I can run this task like way cheaper with like run this task like way cheaper with like run this task like way cheaper with like a 9B or 13B model. Then what I'll do is a 9B or 13B model. Then what I'll do is a 9B or 13B model. Then what I'll do is like I'll take the good traces and the like I'll take the good traces and the like I'll take the good traces and the good examples from the GLM 5.2 runs, good examples from the GLM 5.2 runs, good examples from the GLM 5.2 runs, I'll prepare them in a data set, and I'll prepare them in a data set, and I'll prepare them in a data set, and then I'll try to fine-tune a small model then I'll try to fine-tune a small model then I'll try to fine-tune a small model on that data set to like mimic behavior, on that data set to like mimic behavior, on that data set to like mimic behavior, essentially, right? And this is like essentially, right? And this is like essentially, right? And this is like distillation, SFT. The The other one is distillation, SFT. The The other one is distillation, SFT. The The other one is generating evals and environments. So, generating evals and environments. So, generating evals and environments. So, maybe another slightly hot take, I think maybe another slightly hot take, I think maybe another slightly hot take, I think you can basically define agent behavior you can basically define agent behavior you can basically define agent behavior by showing the evals that you ran on it, by showing the evals that you ran on it, by showing the evals that you ran on it, right? Like, if someone showed me all right? Like, if someone showed me all right? Like, if someone showed me all the things that they're trying to test the things that they're trying to test the things that they're trying to test their agent on, I think I would have a their agent on, I think I would have a their agent on, I think I would have a rough idea about how that agent is going rough idea about how that agent is going rough idea about how that agent is going to behave because it literally like hill to behave because it literally like hill to behave because it literally like hill climbs those evals, and you you alter climbs those evals, and you you alter climbs those evals, and you you alter the behavior of the agent to make the the behavior of the agent to make the the behavior of the agent to make the evals pass, right? Like, the purpose of evals pass, right? Like, the purpose of evals pass, right? Like, the purpose of evals is roughly to try to make them evals is roughly to try to make them evals is roughly to try to make them pass, right? So, I update my agent so pass, right? So, I update my agent so pass, right? So, I update my agent so that they essentially pass. And then the that they essentially pass. And then the that they essentially pass. And then the the other thing is um like humans are the other thing is um like humans are the other thing is um like humans are still in the loop. Like, I need to know still in the loop. Like, I need to know still in the loop. Like, I need to know that customers are happy. I also want to that customers are happy. I also want to that customers are happy. I also want to know what my agents are doing. I just know what my agents are doing. I just know what my agents are doing. I just don't have the bandwidth to read a bunch don't have the bandwidth to read a bunch don't have the bandwidth to read a bunch of traces. So, preparing content for of traces. So, preparing content for of traces. So, preparing content for humans is still like really, really humans is still like really, really humans is still like really, really valuable today, especially in like valuable today, especially in like valuable today, especially in like high-trust domains like legal and high-trust domains like legal and high-trust domains like legal and medical. Like, some human needs to

medical. Like, some human needs to medical. Like, some human needs to review this, um but they can't read it review this, um but they can't read it review this, um but they can't read it all, so we try to make it easy for them all, so we try to make it easy for them all, so we try to make it easy for them to process all that data. Great. This is um maybe a bit of a Great. This is um maybe a bit of a throwback. Like, how many people here throwback. Like, how many people here throwback. Like, how many people here know what like scikit-learn is? Uh maybe know what like scikit-learn is? Uh maybe know what like scikit-learn is? Uh maybe put your uh psych This crowd is just put your uh psych This crowd is just put your uh psych This crowd is just awesome. Um cool. So, uh when I was like awesome. Um cool. So, uh when I was like awesome. Um cool. So, uh when I was like first doing my PhD, uh first doing my PhD, uh first doing my PhD, uh my PhD was like kind of trying to do my PhD was like kind of trying to do my PhD was like kind of trying to do this, but like add new algorithms to this, but like add new algorithms to this, but like add new algorithms to scikit-learn. And like, what scikit-learn. And like, what scikit-learn. And like, what scikit-learn basically is uh an abstract scikit-learn basically is uh an abstract scikit-learn basically is uh an abstract level, it's a bunch of helpers to fit level, it's a bunch of helpers to fit level, it's a bunch of helpers to fit learning systems to data, right? And learning systems to data, right? And learning systems to data, right? And like, classical machine learning, I had like, classical machine learning, I had like, classical machine learning, I had like a data set, I tried to fit it to like a data set, I tried to fit it to like a data set, I tried to fit it to it, but I think the same principles that it, but I think the same principles that it, but I think the same principles that we use in modern in I got I call it we use in modern in I got I call it we use in modern in I got I call it classical machine learning, it's like 6 classical machine learning, it's like 6 classical machine learning, it's like 6 years ago. Um that we do in classical years ago. Um that we do in classical years ago. Um that we do in classical machine learning uh definitely still machine learning uh definitely still machine learning uh definitely still apply to this agent-first world. Um, apply to this agent-first world. Um, apply to this agent-first world. Um, the way that they apply is what I like the way that they apply is what I like the way that they apply is what I like to call model harness task fit. So, we to call model harness task fit. So, we to call model harness task fit. So, we still have this sort of like fit still have this sort of like fit still have this sort of like fit function that I'm going to try to like function that I'm going to try to like function that I'm going to try to like take my data, take a harness, take a take my data, take a harness, take a take my data, take a harness, take a model, and I'm going to try to fit it model, and I'm going to try to fit it model, and I'm going to try to fit it all together to make sure that all of my all together to make sure that all of my all together to make sure that all of my tasks pass, right? The algorithms look tasks pass, right? The algorithms look tasks pass, right? The algorithms look slightly different, uh but the overall slightly different, uh but the overall slightly different, uh but the overall process of machine learning doesn't process of machine learning doesn't process of machine learning doesn't really look that different, and we'll really look that different, and we'll really look that different, and we'll talk about maybe roughly what our job talk about maybe roughly what our job talk about maybe roughly what our job becomes in this data-first, agent-first, becomes in this data-first, agent-first, becomes in this data-first, agent-first, fit-first world. So, a couple of our

fit-first world. So, a couple of our fit-first world. So, a couple of our main jobs now are main jobs now are main jobs now are find good fit functions. So, these are find good fit functions. So, these are find good fit functions. So, these are like auto research. This is tons of like auto research. This is tons of like auto research. This is tons of great work that's being done in RL on great work that's being done in RL on great work that's being done in RL on different methods like OPD, OPSD, different methods like OPD, OPSD, different methods like OPD, OPSD, trySFT. trySFT. trySFT. And also find good data, right? So, if And also find good data, right? So, if And also find good data, right? So, if you put those two things together, then you put those two things together, then you put those two things together, then that is basically the applied or just that is basically the applied or just that is basically the applied or just overall research question that every overall research question that every overall research question that every team has to make their agents better. team has to make their agents better. team has to make their agents better. I like some some examples that we've I like some some examples that we've I like some some examples that we've seen that are like very popular that seen that are like very popular that seen that are like very popular that we're pretty bullish on are just we're pretty bullish on are just we're pretty bullish on are just generally auto research. So, if you have generally auto research. So, if you have generally auto research. So, if you have some sort of score that you can make some sort of score that you can make some sort of score that you can make number go up, uh number go up, uh number go up, uh agents are pretty good at making that agents are pretty good at making that agents are pretty good at making that number go up. They might cheat a little number go up. They might cheat a little number go up. They might cheat a little bit and you need to like check them on bit and you need to like check them on bit and you need to like check them on some stuff. Um, but this sort of like some stuff. Um, but this sort of like some stuff. Um, but this sort of like general feedback loop of do something, general feedback loop of do something, general feedback loop of do something, read the results, read the traces, and read the results, read the traces, and read the results, read the traces, and then do an update ends up being pretty then do an update ends up being pretty then do an update ends up being pretty useful. And then I talked about like useful. And then I talked about like useful. And then I talked about like model fine-tuning a bunch as well. model fine-tuning a bunch as well. model fine-tuning a bunch as well. Um, Um, Um, so we we just like went and did this. Uh so we we just like went and did this. Uh so we we just like went and did this. Uh this was I think even before the term this was I think even before the term this was I think even before the term auto research came out, but a lot of auto research came out, but a lot of auto research came out, but a lot of people were doing it, which is hey, like people were doing it, which is hey, like people were doing it, which is hey, like terminal benches like really hard. Uh terminal benches like really hard. Uh terminal benches like really hard. Uh what would happen if an agent just like what would happen if an agent just like what would happen if an agent just like read its traces, uh proposed read its traces, uh proposed read its traces, uh proposed experiments, and then tried to do fixes?

experiments, and then tried to do fixes? experiments, and then tried to do fixes? Um, I think one like really key thing Um, I think one like really key thing Um, I think one like really key thing here is uh here is uh here is uh giving agents dense feedback signals. giving agents dense feedback signals. giving agents dense feedback signals. So, like terminal bench, the output is So, like terminal bench, the output is So, like terminal bench, the output is just a number, right? Like, did you pass just a number, right? Like, did you pass just a number, right? Like, did you pass or did you not pass? Uh that's like kind or did you not pass? Uh that's like kind or did you not pass? Uh that's like kind of helpful, but if I give you like a of helpful, but if I give you like a of helpful, but if I give you like a super random task, like you just did a super random task, like you just did a super random task, like you just did a bunch of stuff, and then I just said bunch of stuff, and then I just said bunch of stuff, and then I just said like you failed or you passed, uh if you like you failed or you passed, uh if you like you failed or you passed, uh if you failed, like you wouldn't really have a failed, like you wouldn't really have a failed, like you wouldn't really have a good signal to figure out what you good signal to figure out what you good signal to figure out what you should do next, right? So, densifying should do next, right? So, densifying should do next, right? So, densifying feedback is uh really good way to feedback is uh really good way to feedback is uh really good way to improve agents, and like traces are the improve agents, and like traces are the improve agents, and like traces are the substrate that hold that feedback. And substrate that hold that feedback. And substrate that hold that feedback. And then agents are very good at like then agents are very good at like then agents are very good at like reading those uh those traces and then reading those uh those traces and then reading those uh those traces and then figuring out like what to do next. Um figuring out like what to do next. Um figuring out like what to do next. Um and then this sort of question always and then this sort of question always and then this sort of question always comes up, which is comes up, which is comes up, which is when should I like harness Enge? When when should I like harness Enge? When when should I like harness Enge? When should I fine-tune? Uh should I do more should I fine-tune? Uh should I do more should I fine-tune? Uh should I do more harness Enge after it? I'm like pretty harness Enge after it? I'm like pretty harness Enge after it? I'm like pretty bullish on the idea of if you need to do bullish on the idea of if you need to do bullish on the idea of if you need to do something for improving your agent, the something for improving your agent, the something for improving your agent, the best thing that you can do is collect best thing that you can do is collect best thing that you can do is collect feedback as quickly as possible, like feedback as quickly as possible, like feedback as quickly as possible, like either from humans labeling or just either from humans labeling or just either from humans labeling or just letting the agents run. So, like harness letting the agents run. So, like harness letting the agents run. So, like harness engineering gives you feedback in maybe engineering gives you feedback in maybe engineering gives you feedback in maybe 2 minutes. Um 2 minutes. Um 2 minutes. Um once you sort of saturate the harness once you sort of saturate the harness once you sort of saturate the harness engineering ceiling, right? Then you can engineering ceiling, right? Then you can engineering ceiling, right? Then you can maybe try to do like fine-tuning after maybe try to do like fine-tuning after maybe try to do like fine-tuning after that, but we find a lot of teams are that, but we find a lot of teams are that, but we find a lot of teams are happy with harness engineering and uh it happy with harness engineering and uh it happy with harness engineering and uh it solves their customer use case, so like solves their customer use case, so like solves their customer use case, so like we always sort of sort of recommend it.

we always sort of sort of recommend it. we always sort of sort of recommend it. And then we have this like sort of And then we have this like sort of And then we have this like sort of sandwich, which is like try harness sandwich, which is like try harness sandwich, which is like try harness engineering, try to do fine-tuning to engineering, try to do fine-tuning to engineering, try to do fine-tuning to sort of like break through that ceiling, sort of like break through that ceiling, sort of like break through that ceiling, and then do more harness engineering and then do more harness engineering and then do more harness engineering again if you need to. again if you need to. again if you need to. And then I'll sort of end on the the And then I'll sort of end on the the And then I'll sort of end on the the idea generally of continual learning is idea generally of continual learning is idea generally of continual learning is that there's an agent taking actions in that there's an agent taking actions in that there's an agent taking actions in the environment, and then it needs to the environment, and then it needs to the environment, and then it needs to use that information, sorry guys, needs use that information, sorry guys, needs use that information, sorry guys, needs to use that information to update to use that information to update to use that information to update information about itself, right? So, information about itself, right? So, information about itself, right? So, it's like I did a bunch of these tasks, it's like I did a bunch of these tasks, it's like I did a bunch of these tasks, and like I need to update my prompts to and like I need to update my prompts to and like I need to update my prompts to make sure I do them more efficiently. Or make sure I do them more efficiently. Or make sure I do them more efficiently. Or users are users keep asking to search users are users keep asking to search users are users keep asking to search for these types of things, I should for these types of things, I should for these types of things, I should maybe tell like tell my creator that maybe tell like tell my creator that maybe tell like tell my creator that like they're they're doing this sort of like they're they're doing this sort of like they're they're doing this sort of stuff, right? It's like taking action in stuff, right? It's like taking action in stuff, right? It's like taking action in the environment kind of like humans do the environment kind of like humans do the environment kind of like humans do and updating ourselves. What that looks and updating ourselves. What that looks and updating ourselves. What that looks like today, slightly unclear, but we like today, slightly unclear, but we like today, slightly unclear, but we think that you're going to have to do it think that you're going to have to do it think that you're going to have to do it across all three axes, which is one, across all three axes, which is one, across all three axes, which is one, collect a bunch of training data, which collect a bunch of training data, which collect a bunch of training data, which is like observational data from agents is like observational data from agents is like observational data from agents taking actions. The The other one is taking actions. The The other one is taking actions. The The other one is like harness updates generally, like like harness updates generally, like like harness updates generally, like you know, the the Codex harness and the you know, the the Codex harness and the you know, the the Codex harness and the Cloud Code harness and like our harness Cloud Code harness and like our harness Cloud Code harness and like our harness and everyone's harness, like they look a and everyone's harness, like they look a and everyone's harness, like they look a certain way because like models are are certain way because like models are are certain way because like models are are trained in them and they look a certain trained in them and they look a certain trained in them and they look a certain way because of the tasks that they do in way because of the tasks that they do in way because of the tasks that they do in the real world and we think like the real world and we think like the real world and we think like evolving those over time is going to be evolving those over time is going to be evolving those over time is going to be super important in in order to make them super important in in order to make them super important in in order to make them work. And work. And work. And the the last thing is like memory. So, the the last thing is like memory. So, the the last thing is like memory. So, uh uh uh we humans are like really good at like

we humans are like really good at like we humans are like really good at like remembering stuff over time, but we are remembering stuff over time, but we are remembering stuff over time, but we are not append-only logs of information. And not append-only logs of information. And not append-only logs of information. And if agents are going to be working with if agents are going to be working with if agents are going to be working with us over like year, 5-year, decade, us over like year, 5-year, decade, us over like year, 5-year, decade, lifetime time scales, we cannot just lifetime time scales, we cannot just lifetime time scales, we cannot just append everything to like a really big append everything to like a really big append everything to like a really big file and then search over it. There's a file and then search over it. There's a file and then search over it. There's a ton of stuff that needs to happen with ton of stuff that needs to happen with ton of stuff that needs to happen with like updating those files over time and like updating those files over time and like updating those files over time and then just making memory like really then just making memory like really then just making memory like really efficient. But, we think a lot of that efficient. But, we think a lot of that efficient. But, we think a lot of that actually comes from this idea of scaling actually comes from this idea of scaling actually comes from this idea of scaling sleep time compute and and dreaming sleep time compute and and dreaming sleep time compute and and dreaming generally. So, it's like read all of the generally. So, it's like read all of the generally. So, it's like read all of the traces over the entire agent life cycle traces over the entire agent life cycle traces over the entire agent life cycle and then like do [music] things to and then like do [music] things to and then like do [music] things to update agent state. update agent state. update agent state. Awesome. So, like quick quick takeaways, Awesome. So, like quick quick takeaways, Awesome. So, like quick quick takeaways, uh mining traces gives you signals to uh mining traces gives you signals to uh mining traces gives you signals to hill climb on. Uh I would say like if hill climb on. Uh I would say like if hill climb on. Uh I would say like if you have an agent, just turn on tracing you have an agent, just turn on tracing you have an agent, just turn on tracing and point an agent at it and that's like and point an agent at it and that's like and point an agent at it and that's like the easiest thing that you can do to see the easiest thing that you can do to see the easiest thing that you can do to see like to basically understand what your like to basically understand what your like to basically understand what your agents are doing. agents are doing. agents are doing. Uh we're very excited about open models. Uh we're very excited about open models. Uh we're very excited about open models. We want to help you fine-tune open We want to help you fine-tune open We want to help you fine-tune open models. Um we provide them as a service models. Um we provide them as a service models. Um we provide them as a service as well. So, if you're interested in as well. So, if you're interested in as well. So, if you're interested in that, would would love to chat how you that, would would love to chat how you that, would would love to chat how you can use open models to make everything can use open models to make everything can use open models to make everything smarter and cheaper. Um, smarter and cheaper. Um, smarter and cheaper. Um, continual learning is about operating continual learning is about operating continual learning is about operating environments and then integrating that environments and then integrating that environments and then integrating that data back into agent state. And then data back into agent state. And then data back into agent state. And then finally, I think this is so cool that finally, I think this is so cool that finally, I think this is so cool that like we have systems that's going to like we have systems that's going to like we have systems that's going to pre- that are going to produce more data pre- that are going to produce more data pre- that are going to produce more data than we ever have before.

than we ever have before. than we ever have before. Uh, we need to all come up with like Uh, we need to all come up with like Uh, we need to all come up with like interesting research directions to learn interesting research directions to learn interesting research directions to learn how to like manage that at scale and how to like manage that at scale and how to like manage that at scale and like make all of our agents better. like make all of our agents better. like make all of our agents better. And with that, And with that, And with that, thank you. Thank you all for coming. thank you. Thank you all for coming. thank you. Thank you all for coming. >> [applause]