Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic

Good morning. We're super excited to be Good morning. We're super excited to be here at AI Engineer with all of you. I'm here at AI Engineer with all of you. I'm here at AI Engineer with all of you. I'm Caitlyn and I lead platform engineering Caitlyn and I lead platform engineering Caitlyn and I lead platform engineering at Anthropic. at Anthropic. at Anthropic. >> And I'm Angela. I lead platform product >> And I'm Angela. I lead platform product >> And I'm Angela. I lead platform product at Anthropic. And today we want to talk at Anthropic. And today we want to talk at Anthropic. And today we want to talk to you about a concept that we've been to you about a concept that we've been to you about a concept that we've been spending a lot of time thinking about spending a lot of time thinking about spending a lot of time thinking about and working on with our team, which is and working on with our team, which is and working on with our team, which is this idea that we think that tokens this idea that we think that tokens this idea that we think that tokens should have jobs. should have jobs. should have jobs. So if you're building an agentic system So if you're building an agentic system So if you're building an agentic system and you're trying to accomplish some and you're trying to accomplish some and you're trying to accomplish some specific outcome, you're trying to get specific outcome, you're trying to get specific outcome, you're trying to get something done with agents, there's one something done with agents, there's one something done with agents, there's one lever that everybody pulls in order to lever that everybody pulls in order to lever that everybody pulls in order to get a better outcome, and that's usually get a better outcome, and that's usually get a better outcome, and that's usually increasing your budget, which means you increasing your budget, which means you increasing your budget, which means you spend more tokens or you spend more spend more tokens or you spend more spend more tokens or you spend more expensive tokens. expensive tokens. expensive tokens. But we've been wondering is that all But we've been wondering is that all But we've been wondering is that all there is underlying this assumption of there is underlying this assumption of there is underlying this assumption of uh using the budget is this kind of uh using the budget is this kind of uh using the budget is this kind of implicit perspective that every single implicit perspective that every single implicit perspective that every single token is basically fungeable. And we've token is basically fungeable. And we've token is basically fungeable. And we've been wondering is that actually true? been wondering is that actually true? been wondering is that actually true? Are all these tokens actually fungeible? Are all these tokens actually fungeible? Are all these tokens actually fungeible? And to test that, we've been thinking, And to test that, we've been thinking, And to test that, we've been thinking, what if we gave tokens jobs? what if we gave tokens jobs? what if we gave tokens jobs? So, if you think about the way that you So, if you think about the way that you So, if you think about the way that you would normally set up an agent to go would normally set up an agent to go would normally set up an agent to go accomplish a task, you give it that accomplish a task, you give it that accomplish a task, you give it that task, you give it this token budget, and task, you give it this token budget, and task, you give it this token budget, and then all the tokens that are being spent then all the tokens that are being spent then all the tokens that are being spent are basically indiscriminate in the are basically indiscriminate in the are basically indiscriminate in the sense that they're all doing one job.

sense that they're all doing one job. sense that they're all doing one job. They're just executing. They're just executing. They're just executing. But what if you take some of those But what if you take some of those But what if you take some of those tokens and they're not just executing, tokens and they're not just executing, tokens and they're not just executing, they're doing some other job. So, for they're doing some other job. So, for they're doing some other job. So, for example, maybe you take some of your example, maybe you take some of your example, maybe you take some of your tokens and they're advising the tokens tokens and they're advising the tokens tokens and they're advising the tokens that are executing. Or maybe the tokens that are executing. Or maybe the tokens that are executing. Or maybe the tokens that are executing try to get something that are executing try to get something that are executing try to get something done well and you take some other tokens done well and you take some other tokens done well and you take some other tokens and you actually grade how well the and you actually grade how well the and you actually grade how well the executor is doing so that it can iterate executor is doing so that it can iterate executor is doing so that it can iterate and try again. Or maybe you have tokens and try again. Or maybe you have tokens and try again. Or maybe you have tokens that are dreaming. They're reflecting that are dreaming. They're reflecting that are dreaming. They're reflecting back on the job that other executors back on the job that other executors back on the job that other executors have done and writing learnings to have done and writing learnings to have done and writing learnings to memory so that they can do it again. And memory so that they can do it again. And memory so that they can do it again. And what we call each of these if you take what we call each of these if you take what we call each of these if you take some tokens that are executing and some some tokens that are executing and some some tokens that are executing and some tokens that are doing some other job. tokens that are doing some other job. tokens that are doing some other job. Let's call this a strategy. Let's call this a strategy. Let's call this a strategy. So let's go take a look at the first So let's go take a look at the first So let's go take a look at the first strategy, the advising strategy. Here strategy, the advising strategy. Here strategy, the advising strategy. Here we're splitting up an executor and an we're splitting up an executor and an we're splitting up an executor and an adviser. The executor obviously adviser. The executor obviously adviser. The executor obviously executes, but crucially they can call executes, but crucially they can call executes, but crucially they can call out to an adviser for advice. And then out to an adviser for advice. And then out to an adviser for advice. And then they can take this advice and figure out they can take this advice and figure out they can take this advice and figure out if they're doing the next step if they're doing the next step if they're doing the next step correctly.

correctly. correctly. This is really helpful in use cases. For This is really helpful in use cases. For This is really helpful in use cases. For example, if you're building a sales example, if you're building a sales example, if you're building a sales agent, in an ideal world, you'd have agent, in an ideal world, you'd have agent, in an ideal world, you'd have that sales agent be able to actually that sales agent be able to actually that sales agent be able to actually help the sales rep flag when a follow-up help the sales rep flag when a follow-up help the sales rep flag when a follow-up is overdue or deal is stalling. In this is overdue or deal is stalling. In this is overdue or deal is stalling. In this construct, having an adviser to be able construct, having an adviser to be able construct, having an adviser to be able to kind of make sure that all the to kind of make sure that all the to kind of make sure that all the different pieces are actually working is different pieces are actually working is different pieces are actually working is really helpful. really helpful. really helpful. So another example is grading. Let's say So another example is grading. Let's say So another example is grading. Let's say you're executing and you kind of know you're executing and you kind of know you're executing and you kind of know exactly what good really does look like. exactly what good really does look like. exactly what good really does look like. You can define this in a rubric and then You can define this in a rubric and then You can define this in a rubric and then each time an executor tries to each time an executor tries to each time an executor tries to accomplish that outcome, you can have a accomplish that outcome, you can have a accomplish that outcome, you can have a grader provisioned that grades how well grader provisioned that grades how well grader provisioned that grades how well the executor did while looking at that the executor did while looking at that the executor did while looking at that rubric. And if the executor did a good rubric. And if the executor did a good rubric. And if the executor did a good job, then great, it can be done. But if job, then great, it can be done. But if job, then great, it can be done. But if it didn't do such a great job, you can it didn't do such a great job, you can it didn't do such a great job, you can iterate again until you get that good iterate again until you get that good iterate again until you get that good outcome. outcome. outcome. So an example in practice of when you So an example in practice of when you So an example in practice of when you might want to use this is let's say you might want to use this is let's say you might want to use this is let's say you have a customer service agent and you're have a customer service agent and you're have a customer service agent and you're running a store and your customers are running a store and your customers are running a store and your customers are writing in and they're saying, "H, I writing in and they're saying, "H, I writing in and they're saying, "H, I should get a refund for this thing." And should get a refund for this thing." And should get a refund for this thing." And your customer service agent needs to be your customer service agent needs to be your customer service agent needs to be able to respond. You probably have some able to respond. You probably have some able to respond. You probably have some like pretty specific criteria on when like pretty specific criteria on when like pretty specific criteria on when you would give somebody a refund. And so you would give somebody a refund. And so you would give somebody a refund. And so what you can do is define a rubric that what you can do is define a rubric that what you can do is define a rubric that uses that criteria. You can have a uses that criteria. You can have a uses that criteria. You can have a grader that goes and looks at the work grader that goes and looks at the work grader that goes and looks at the work that the customer service agent is doing that the customer service agent is doing that the customer service agent is doing and decide is it getting it right and is and decide is it getting it right and is and decide is it getting it right and is it coming to the right outcome.

it coming to the right outcome. it coming to the right outcome. And the last strategy we have is And the last strategy we have is And the last strategy we have is dreaming. So in dreaming there's an dreaming. So in dreaming there's an dreaming. So in dreaming there's an executor who naturally executes and then executor who naturally executes and then executor who naturally executes and then there's a dreamer. The dreamer is there's a dreamer. The dreamer is there's a dreamer. The dreamer is actually able to inspect the work and actually able to inspect the work and actually able to inspect the work and the transcripts of the executor and then the transcripts of the executor and then the transcripts of the executor and then it takes any of the findings that it has it takes any of the findings that it has it takes any of the findings that it has and it writes them to memory. This and it writes them to memory. This and it writes them to memory. This memory is repicked up by the executor memory is repicked up by the executor memory is repicked up by the executor for the next round. So ideally would for the next round. So ideally would for the next round. So ideally would have improved. have improved. have improved. A great use case for this is if you're A great use case for this is if you're A great use case for this is if you're building a recruiting agent. Now building a recruiting agent. Now building a recruiting agent. Now recruiting requires a lot of interaction recruiting requires a lot of interaction recruiting requires a lot of interaction with feedback on whether or not a with feedback on whether or not a with feedback on whether or not a candidate does or doesn't make sense and candidate does or doesn't make sense and candidate does or doesn't make sense and if it's a good fit between both parties. if it's a good fit between both parties. if it's a good fit between both parties. And so by taking all this type of data, And so by taking all this type of data, And so by taking all this type of data, if you build a dreaming type of strategy if you build a dreaming type of strategy if you build a dreaming type of strategy on this agent, it's actually able to on this agent, it's actually able to on this agent, it's actually able to kind of sharpen the next round so that kind of sharpen the next round so that kind of sharpen the next round so that it's more and more increasingly useful. it's more and more increasingly useful. it's more and more increasingly useful. So let's make this concrete with some So let's make this concrete with some So let's make this concrete with some experiments. So what we did was we experiments. So what we did was we experiments. So what we did was we created a bench of a bunch of tasks created a bench of a bunch of tasks created a bench of a bunch of tasks related to financial an analysis. And related to financial an analysis. And related to financial an analysis. And what we were doing with each of these what we were doing with each of these what we were doing with each of these tasks is trying to replicate in the real tasks is trying to replicate in the real tasks is trying to replicate in the real world a expert human financial analyst. world a expert human financial analyst. world a expert human financial analyst. How well would they do on each of these How well would they do on each of these How well would they do on each of these various tasks? And so what we did was we various tasks? And so what we did was we various tasks? And so what we did was we start with a control that's just start with a control that's just start with a control that's just executing. Let's try each of these tasks executing. Let's try each of these tasks executing. Let's try each of these tasks and we'll eval them when we're literally and we'll eval them when we're literally and we'll eval them when we're literally just executing. But then we can just executing. But then we can just executing. But then we can experiment with each of our strategies experiment with each of our strategies experiment with each of our strategies and see how well we perform.

So, we start with a super basic So, we start with a super basic experiment. Let's just oneshot it. Let's experiment. Let's just oneshot it. Let's experiment. Let's just oneshot it. Let's take each of our strategies and we'll go take each of our strategies and we'll go take each of our strategies and we'll go and just make an attempt to accomplish and just make an attempt to accomplish and just make an attempt to accomplish these tasks and we'll see how accurate these tasks and we'll see how accurate these tasks and we'll see how accurate we are. And so, you can see here with we are. And so, you can see here with we are. And so, you can see here with executing um it didn't do so well. 15% executing um it didn't do so well. 15% executing um it didn't do so well. 15% accuracy, but because it was just a accuracy, but because it was just a accuracy, but because it was just a oneshot, the strategy got to choose how oneshot, the strategy got to choose how oneshot, the strategy got to choose how many tokens it would actually spend on many tokens it would actually spend on many tokens it would actually spend on its own. And so, you can actually see its own. And so, you can actually see its own. And so, you can actually see that execute decided not to spend that that execute decided not to spend that that execute decided not to spend that many tokens, only 39,000. And as we go many tokens, only 39,000. And as we go many tokens, only 39,000. And as we go into our larger strategies, our more into our larger strategies, our more into our larger strategies, our more complex strategies, we did choose to complex strategies, we did choose to complex strategies, we did choose to spend more tokens, but we did a better spend more tokens, but we did a better spend more tokens, but we did a better job. So, this isn't really telling us job. So, this isn't really telling us job. So, this isn't really telling us much because sure, Drain did really, much because sure, Drain did really, much because sure, Drain did really, really well, but it used a whopping really well, but it used a whopping really well, but it used a whopping 600,000 tokens to get there. That's 600,000 tokens to get there. That's 600,000 tokens to get there. That's right. So, in order to actually figure right. So, in order to actually figure right. So, in order to actually figure out if varying the jobs produces any out if varying the jobs produces any out if varying the jobs produces any alpha, what we need to do is hold the alpha, what we need to do is hold the alpha, what we need to do is hold the budget constant. And to do this, we're budget constant. And to do this, we're budget constant. And to do this, we're going to take Dreaming's budget, that going to take Dreaming's budget, that going to take Dreaming's budget, that 600,000 or so, as the maximum budget 600,000 or so, as the maximum budget 600,000 or so, as the maximum budget that is fixed across the board. And we that is fixed across the board. And we that is fixed across the board. And we give every single strategy this budget give every single strategy this budget give every single strategy this budget in order to analyze how well it's in order to analyze how well it's in order to analyze how well it's performing. And as expected again, if performing. And as expected again, if performing. And as expected again, if you give a lot of strategies more you give a lot of strategies more you give a lot of strategies more budget, you are going to see performance budget, you are going to see performance budget, you are going to see performance increase across the board. So execute increase across the board. So execute increase across the board. So execute went from 0.15 to 76. Advise and grade went from 0.15 to 76. Advise and grade went from 0.15 to 76. Advise and grade went from the 60s to closer to the 90s.

went from the 60s to closer to the 90s. went from the 60s to closer to the 90s. And that's again expected given the fact And that's again expected given the fact And that's again expected given the fact that if you give things more test time that if you give things more test time that if you give things more test time compute, they should generally perform compute, they should generally perform compute, they should generally perform better. But if that was the only thing better. But if that was the only thing better. But if that was the only thing that mattered, we should actually expect that mattered, we should actually expect that mattered, we should actually expect to see execute, advise, grade, dream to see execute, advise, grade, dream to see execute, advise, grade, dream actually all be at the exact same level actually all be at the exact same level actually all be at the exact same level given the exact same token budget. But given the exact same token budget. But given the exact same token budget. But what we're actually seeing is that there what we're actually seeing is that there what we're actually seeing is that there is an alpha or there is a difference and is an alpha or there is a difference and is an alpha or there is a difference and therefore an alpha for us to exploit. If therefore an alpha for us to exploit. If therefore an alpha for us to exploit. If you look at execute at this exact same you look at execute at this exact same you look at execute at this exact same budget level, it gets to 76 but advise budget level, it gets to 76 but advise budget level, it gets to 76 but advise is at 89. So while a minimal, it does is at 89. So while a minimal, it does is at 89. So while a minimal, it does exist and so there is alpha for us to exist and so there is alpha for us to exist and so there is alpha for us to take a look at. take a look at. take a look at. Now, we decided to take a look at this Now, we decided to take a look at this Now, we decided to take a look at this analysis from a completely different analysis from a completely different analysis from a completely different lens. And as Kayla mentioned, you know, lens. And as Kayla mentioned, you know, lens. And as Kayla mentioned, you know, we're doing this bench for a very we're doing this bench for a very we're doing this bench for a very complex set of financial tasks in the complex set of financial tasks in the complex set of financial tasks in the real world. And we wanted to analyze the real world. And we wanted to analyze the real world. And we wanted to analyze the usage of agents with actual experts. So, usage of agents with actual experts. So, usage of agents with actual experts. So, if we look at a financial analyst if we look at a financial analyst if we look at a financial analyst expert, right, the kind of task that expert, right, the kind of task that expert, right, the kind of task that they need to do with an agent is that they need to do with an agent is that they need to do with an agent is that they're giving it something very they're giving it something very they're giving it something very concrete like let's say make a P&L and concrete like let's say make a P&L and concrete like let's say make a P&L and then they're getting the result back.

then they're getting the result back. then they're getting the result back. Now if that result is 80% accurate on a Now if that result is 80% accurate on a Now if that result is 80% accurate on a bench that sounds great but in reality bench that sounds great but in reality bench that sounds great but in reality what that means for that expert is they what that means for that expert is they what that means for that expert is they have to go back and recomputee that P&L have to go back and recomputee that P&L have to go back and recomputee that P&L themselves or alter or alternatively themselves or alter or alternatively themselves or alter or alternatively send it through another run and that's send it through another run and that's send it through another run and that's because in this kind of domain for this because in this kind of domain for this because in this kind of domain for this kind of task if you're not 100% accurate kind of task if you're not 100% accurate kind of task if you're not 100% accurate it's actually not useful. it's actually not useful. it's actually not useful. You cannot make up an income number or You cannot make up an income number or You cannot make up an income number or you can't make up a cost number right you can't make up a cost number right you can't make up a cost number right you have to make sure that it's 100% you have to make sure that it's 100% you have to make sure that it's 100% accurate. So with this lens of the real accurate. So with this lens of the real accurate. So with this lens of the real world consequence associated with this world consequence associated with this world consequence associated with this domain, we needed to recomputee our domain, we needed to recomputee our domain, we needed to recomputee our experiments and score them a bit experiments and score them a bit experiments and score them a bit differently. Crucially, we needed to differently. Crucially, we needed to differently. Crucially, we needed to make sure that our experiment had this make sure that our experiment had this make sure that our experiment had this kind of construct where if it was scored kind of construct where if it was scored kind of construct where if it was scored perfectly, we'd actually give it a pass. perfectly, we'd actually give it a pass. perfectly, we'd actually give it a pass. And if it scored anything less than 100% And if it scored anything less than 100% And if it scored anything less than 100% on that kind of task, we would actually on that kind of task, we would actually on that kind of task, we would actually mark it as a failure. mark it as a failure. mark it as a failure. So let's look at a different cut of our So let's look at a different cut of our So let's look at a different cut of our data from our experiments with this lens data from our experiments with this lens data from our experiments with this lens where we're looking for this perfect run where we're looking for this perfect run where we're looking for this perfect run 100% accuracy pass. And let's look at 100% accuracy pass. And let's look at 100% accuracy pass. And let's look at what percent of the time each of these what percent of the time each of these what percent of the time each of these strategies was able to achieve a pass. strategies was able to achieve a pass. strategies was able to achieve a pass. Um so we we've got executes um down at Um so we we've got executes um down at Um so we we've got executes um down at 42% and we've got our more complex 42% and we've got our more complex 42% and we've got our more complex strategies doing a bit better up to 75% strategies doing a bit better up to 75% strategies doing a bit better up to 75% accuracy. Um and again this doesn't accuracy. Um and again this doesn't accuracy. Um and again this doesn't necessarily tell us a ton because um you necessarily tell us a ton because um you necessarily tell us a ton because um you know each of these strategies might um know each of these strategies might um know each of these strategies might um choose to use different budgets over choose to use different budgets over choose to use different budgets over time, right? So what we did here was we time, right? So what we did here was we time, right? So what we did here was we fixed the budget and we said within a fixed the budget and we said within a fixed the budget and we said within a fixed budget, how well do each of these fixed budget, how well do each of these fixed budget, how well do each of these strategies perform?

strategies perform? strategies perform? And so what really matters to us And so what really matters to us And so what really matters to us actually is if you're trying to get this actually is if you're trying to get this actually is if you're trying to get this perfect answer and you're in the real perfect answer and you're in the real perfect answer and you're in the real world, you're running a business, what world, you're running a business, what world, you're running a business, what matters to you is the cost to you to get matters to you is the cost to you to get matters to you is the cost to you to get to that perfect answer. And so one way to that perfect answer. And so one way to that perfect answer. And so one way we can think about this is we had our we can think about this is we had our we can think about this is we had our execute strategy for example. The execute strategy for example. The execute strategy for example. The execute strategy around 40% of the time execute strategy around 40% of the time execute strategy around 40% of the time will give you that perfect answer. So on will give you that perfect answer. So on will give you that perfect answer. So on average, you can expect to have to run average, you can expect to have to run average, you can expect to have to run it three times and you should hopefully it three times and you should hopefully it three times and you should hopefully sometime in those three runs get a sometime in those three runs get a sometime in those three runs get a perfect answer. And as we talked about perfect answer. And as we talked about perfect answer. And as we talked about earlier, we fixed our budget to that earlier, we fixed our budget to that earlier, we fixed our budget to that highest token budget strategy, which was highest token budget strategy, which was highest token budget strategy, which was 600,000 tokens. So if you spend 600,000 600,000 tokens. So if you spend 600,000 600,000 tokens. So if you spend 600,000 tokens in each individual run, you have tokens in each individual run, you have tokens in each individual run, you have to run approximately three times. You to run approximately three times. You to run approximately three times. You can expect on average to have to spend can expect on average to have to spend can expect on average to have to spend 1.8 million tokens with the execution 1.8 million tokens with the execution 1.8 million tokens with the execution strategy to get to your perfect answer. strategy to get to your perfect answer. strategy to get to your perfect answer. And so if we take this analysis and run And so if we take this analysis and run And so if we take this analysis and run it across the board against all these it across the board against all these it across the board against all these strategies, this is actually the true strategies, this is actually the true strategies, this is actually the true cost it took in this domain for that cost it took in this domain for that cost it took in this domain for that agent to be useful for that strategy. So agent to be useful for that strategy. So agent to be useful for that strategy. So as Caitlyn mentioned for execute, which as Caitlyn mentioned for execute, which as Caitlyn mentioned for execute, which is our baseline, this is going to be 1.8 is our baseline, this is going to be 1.8 is our baseline, this is going to be 1.8 million true total token cost for you.

million true total token cost for you. million true total token cost for you. But advise, grade, and dream are showing But advise, grade, and dream are showing But advise, grade, and dream are showing us a bit of difference. Crucially, us a bit of difference. Crucially, us a bit of difference. Crucially, advise and grade are actually quite advise and grade are actually quite advise and grade are actually quite token efficient when you think about the token efficient when you think about the token efficient when you think about the actual usage of the end output of each actual usage of the end output of each actual usage of the end output of each of these agents. of these agents. of these agents. So what does this mean for you as a So what does this mean for you as a So what does this mean for you as a business? Well, it actually really business? Well, it actually really business? Well, it actually really depends on what kind of thing you want depends on what kind of thing you want depends on what kind of thing you want to optimize for and it's going to vary, to optimize for and it's going to vary, to optimize for and it's going to vary, right? There's going to be businesses right? There's going to be businesses right? There's going to be businesses who say, "Actually, for me, the most who say, "Actually, for me, the most who say, "Actually, for me, the most important thing is to be really token important thing is to be really token important thing is to be really token efficient. In that case, you should efficient. In that case, you should efficient. In that case, you should probably pick the advised type of probably pick the advised type of probably pick the advised type of strategy in order to solve for that strategy in order to solve for that strategy in order to solve for that particular domain in which you want to particular domain in which you want to particular domain in which you want to optimize that." There's going to be optimize that." There's going to be optimize that." There's going to be other areas or other businesses where other areas or other businesses where other areas or other businesses where you're going to say, I'm not going to you're going to say, I'm not going to you're going to say, I'm not going to care so much about token efficiency care so much about token efficiency care so much about token efficiency because what I really care about is because what I really care about is because what I really care about is reliability of that answer and so I need reliability of that answer and so I need reliability of that answer and so I need to maximize the percentage of runs in to maximize the percentage of runs in to maximize the percentage of runs in which I get that perfect answer. In which I get that perfect answer. In which I get that perfect answer. In which case, you would actually pick which case, you would actually pick which case, you would actually pick completely different strategies. You completely different strategies. You completely different strategies. You probably lean towards grade or dream. probably lean towards grade or dream. probably lean towards grade or dream. So if you take away one thing, the thing So if you take away one thing, the thing So if you take away one thing, the thing we want everyone to think about is this we want everyone to think about is this we want everyone to think about is this idea that tokens are not fungeible. You idea that tokens are not fungeible. You idea that tokens are not fungeible. You can use your tokens to execute. You can can use your tokens to execute. You can can use your tokens to execute. You can brute force your way through your task brute force your way through your task brute force your way through your task and you can throw more budget at it. But and you can throw more budget at it. But and you can throw more budget at it. But if you get really smart about having if you get really smart about having if you get really smart about having your tokens do these different jobs and your tokens do these different jobs and your tokens do these different jobs and try these different strategies, you're try these different strategies, you're try these different strategies, you're very very likely to be able to get a very very likely to be able to get a very very likely to be able to get a better outcome for the task at hand better outcome for the task at hand better outcome for the task at hand within a fixed budget.

within a fixed budget. within a fixed budget. And so let's talk a little bit about how And so let's talk a little bit about how And so let's talk a little bit about how we actually build strategies and how we we actually build strategies and how we we actually build strategies and how we bring this to life. Um so we've done a bring this to life. Um so we've done a bring this to life. Um so we've done a lot of work to create a really excellent lot of work to create a really excellent lot of work to create a really excellent harness for individual agents. Um and if harness for individual agents. Um and if harness for individual agents. Um and if you see this uh picture at the bottom you see this uh picture at the bottom you see this uh picture at the bottom here, this is actually the architecture here, this is actually the architecture here, this is actually the architecture that we've used for cloud managed agents that we've used for cloud managed agents that we've used for cloud managed agents um which is our Aentic solution that we um which is our Aentic solution that we um which is our Aentic solution that we give to you within the cloud platform. give to you within the cloud platform. give to you within the cloud platform. And what we do on top of this is we And what we do on top of this is we And what we do on top of this is we start to get into the meta harness level start to get into the meta harness level start to get into the meta harness level like the multi- aent orchestration and like the multi- aent orchestration and like the multi- aent orchestration and execution level where this strategy can execution level where this strategy can execution level where this strategy can go and be um coordinated between our go and be um coordinated between our go and be um coordinated between our executor and our adviser or the other executor and our adviser or the other executor and our adviser or the other agents within our strategy. And some of agents within our strategy. And some of agents within our strategy. And some of these um like dreaming and outcomes we these um like dreaming and outcomes we these um like dreaming and outcomes we actually give to you out of the box actually give to you out of the box actually give to you out of the box within cloud manage agents. within cloud manage agents. within cloud manage agents. So with those set of primitives it's So with those set of primitives it's So with those set of primitives it's actually relatively trivial for us to actually relatively trivial for us to actually relatively trivial for us to construct this kind of you know construct this kind of you know construct this kind of you know architecture where we're able to combine architecture where we're able to combine architecture where we're able to combine these different types of strategies and these different types of strategies and these different types of strategies and figure out how to they should work figure out how to they should work figure out how to they should work together. So for example it's relatively together. So for example it's relatively together. So for example it's relatively trivial for us to say okay now with this trivial for us to say okay now with this trivial for us to say okay now with this I can take a task and I should be able I can take a task and I should be able I can take a task and I should be able to execute it but also allow it to to execute it but also allow it to to execute it but also allow it to advise and fable is back online. So we advise and fable is back online. So we advise and fable is back online. So we could actually say Fable is the one could actually say Fable is the one could actually say Fable is the one that's actually advising uh the that's actually advising uh the that's actually advising uh the executor. And then I can take all these executor. And then I can take all these executor. And then I can take all these results and say send them to a greater results and say send them to a greater results and say send them to a greater so that I can make sure that this is so that I can make sure that this is so that I can make sure that this is verifying in a loop that makes sense.

verifying in a loop that makes sense. verifying in a loop that makes sense. And if it passes, that's awesome. I want And if it passes, that's awesome. I want And if it passes, that's awesome. I want to send all of that stuff to Dreaming to send all of that stuff to Dreaming to send all of that stuff to Dreaming and make sure that my next run is better and make sure that my next run is better and make sure that my next run is better than ever. than ever. than ever. And of course, you don't have to stop And of course, you don't have to stop And of course, you don't have to stop there, right? If the right primitives there, right? If the right primitives there, right? If the right primitives are there and the right coordination is are there and the right coordination is are there and the right coordination is there, then you can actually construct there, then you can actually construct there, then you can actually construct really complex setups that fit for all really complex setups that fit for all really complex setups that fit for all the different types of dynamic problems the different types of dynamic problems the different types of dynamic problems that you have. You can invent these that you have. You can invent these that you have. You can invent these kinds of large-scale architectures, kinds of large-scale architectures, kinds of large-scale architectures, again, very triv. And you could also again, very triv. And you could also again, very triv. And you could also invent completely new jobs, not just the invent completely new jobs, not just the invent completely new jobs, not just the ones of the pieces that Caitlyn and I ones of the pieces that Caitlyn and I ones of the pieces that Caitlyn and I have presented in this conversation. have presented in this conversation. have presented in this conversation. So, a big goal that we have over time is So, a big goal that we have over time is So, a big goal that we have over time is to get our models better and better and to get our models better and better and to get our models better and better and our platform better and better at our platform better and better at our platform better and better at dynamically constructing these dynamically constructing these dynamically constructing these strategies for you as you're doing work. strategies for you as you're doing work. strategies for you as you're doing work. But in the meantime, as we're working But in the meantime, as we're working But in the meantime, as we're working our way there, we would love for you to our way there, we would love for you to our way there, we would love for you to continue to think about this idea that continue to think about this idea that continue to think about this idea that you should give your tokens jobs and you you should give your tokens jobs and you you should give your tokens jobs and you should use different novel strategies by should use different novel strategies by should use different novel strategies by combining these primitives in order to combining these primitives in order to combining these primitives in order to get the outcomes that you want for your get the outcomes that you want for your get the outcomes that you want for your tasks. tasks. tasks. >> Thanks for joining us.