Give the Agent a Budget, Not a Token — Sachin Malhotra, Anthropic

Hello. All right. Uh, let's get started. Hello. All right. Uh, let's get started. So, um, well, welcome to the talk. Give So, um, well, welcome to the talk. Give So, um, well, welcome to the talk. Give the agent a budget, not a not a token. the agent a budget, not a not a token. the agent a budget, not a not a token. Um, quick intro about me. My name is Um, quick intro about me. My name is Um, quick intro about me. My name is Sachin. Um, I'm an engineer on the CI Sachin. Um, I'm an engineer on the CI Sachin. Um, I'm an engineer on the CI team at Entropic. Uh we basically build team at Entropic. Uh we basically build team at Entropic. Uh we basically build and manage all of the test machinery for and manage all of the test machinery for and manage all of the test machinery for all the code that people write. Uh so all the code that people write. Uh so all the code that people write. Uh so everything from test quarantining to everything from test quarantining to everything from test quarantining to merge automation to CI autoscaling to merge automation to CI autoscaling to merge automation to CI autoscaling to merge cues, everything and anything in merge cues, everything and anything in merge cues, everything and anything in between. Um it's it's basically the between. Um it's it's basically the between. Um it's it's basically the plumbing that allows a few thousand plumbing that allows a few thousand plumbing that allows a few thousand engineers every day to ship code safely. engineers every day to ship code safely. engineers every day to ship code safely. Uh I've been at the company for a little Uh I've been at the company for a little Uh I've been at the company for a little over 10 months. I'm based out of over 10 months. I'm based out of over 10 months. I'm based out of Seattle. Um and outside of work, I love Seattle. Um and outside of work, I love Seattle. Um and outside of work, I love dialing my espressos and climbing some dialing my espressos and climbing some dialing my espressos and climbing some crazy mountains. crazy mountains. crazy mountains. So a lot of cool agent demos they they So a lot of cool agent demos they they So a lot of cool agent demos they they start with the same way really. So start with the same way really. So start with the same way really. So someone gives an agent a god token um someone gives an agent a god token um someone gives an agent a god token um and gives it an access to some sort of and gives it an access to some sort of and gives it an access to some sort of tool list and just watch as it goes. So tool list and just watch as it goes. So tool list and just watch as it goes. So and it does go like in in this and it does go like in in this and it does go like in in this particular example it it'll spin up like particular example it it'll spin up like particular example it it'll spin up like a cute little coffee website. It'll a cute little coffee website. It'll a cute little coffee website. It'll build all the pages, deploy them, uh build all the pages, deploy them, uh build all the pages, deploy them, uh bring it up live and it's done in 3 bring it up live and it's done in 3 bring it up live and it's done in 3 seconds. Uh people will nod people will seconds. Uh people will nod people will seconds. Uh people will nod people will clap and someone will ship it into clap and someone will ship it into clap and someone will ship it into production. And this this talk is production. And this this talk is production. And this this talk is basically about what happens after that basically about what happens after that basically about what happens after that after the demo ships once the agent is after the demo ships once the agent is after the demo ships once the agent is starting to do some like real work in starting to do some like real work in starting to do some like real work in production. [snorts] production. [snorts] production. [snorts] So here's what after looked like in in So here's what after looked like in in So here's what after looked like in in our one of our scenarios. So what you're our one of our scenarios. So what you're our one of our scenarios. So what you're looking at is like a real command. Um looking at is like a real command. Um looking at is like a real command. Um the agent was trying to clean up after the agent was trying to clean up after the agent was trying to clean up after itself. Um it it was basically listing a itself. Um it it was basically listing a itself. Um it it was basically listing a bunch of workloads that were no longer bunch of workloads that were no longer bunch of workloads that were no longer interesting or useful to itself. Um and interesting or useful to itself. Um and interesting or useful to itself. Um and then it found them and deleted them. Now then it found them and deleted them. Now then it found them and deleted them. Now that's that's completely reasonable

that's that's completely reasonable that's that's completely reasonable except one stage in the pipeline except one stage in the pipeline except one stage in the pipeline basically evaluated to nothing and the basically evaluated to nothing and the basically evaluated to nothing and the filter dropped out and now the selector filter dropped out and now the selector filter dropped out and now the selector matched everything. matched everything. matched everything. So you can see it took out about 200 So you can see it took out about 200 So you can see it took out about 200 workloads uh which ended up impacting workloads uh which ended up impacting workloads uh which ended up impacting about 20 engineers worth of stuff and about 20 engineers worth of stuff and about 20 engineers worth of stuff and all of that was gone in 90 seconds. all of that was gone in 90 seconds. all of that was gone in 90 seconds. Nobody was being malicious in this case Nobody was being malicious in this case Nobody was being malicious in this case like the agent genuinely thought it was like the agent genuinely thought it was like the agent genuinely thought it was tidying up after itself. uh some of tidying up after itself. uh some of tidying up after itself. uh some of these workloads were long running these workloads were long running these workloads were long running training jobs and stuff. Maybe some of training jobs and stuff. Maybe some of training jobs and stuff. Maybe some of these were not even checkpointed and it these were not even checkpointed and it these were not even checkpointed and it was just like hours of progress that was was just like hours of progress that was was just like hours of progress that was gone poof in like 90 seconds and stuff, gone poof in like 90 seconds and stuff, gone poof in like 90 seconds and stuff, right? Um the the problem in in this right? Um the the problem in in this right? Um the the problem in in this case is that this idea of like here's a case is that this idea of like here's a case is that this idea of like here's a token and here's a tool list that is token and here's a tool list that is token and here's a tool list that is just like not enough like it just just like not enough like it just just like not enough like it just doesn't scale when the agent is starting doesn't scale when the agent is starting doesn't scale when the agent is starting to do some some like real work in to do some some like real work in to do some some like real work in production. production. production. >> So here's just a brief summary of like >> So here's just a brief summary of like >> So here's just a brief summary of like what we're going to talk about today. Uh what we're going to talk about today. Uh what we're going to talk about today. Uh I'm going to walk through three I'm going to walk through three I'm going to walk through three primitives um and asymmetric verbs. We primitives um and asymmetric verbs. We primitives um and asymmetric verbs. We we're going to look at some like rate we're going to look at some like rate we're going to look at some like rate limits. We're going to look at this idea limits. We're going to look at this idea limits. We're going to look at this idea of like trip wires over allow lists. And of like trip wires over allow lists. And of like trip wires over allow lists. And then I'm going to talk about this one then I'm going to talk about this one then I'm going to talk about this one lens that which I call the undo test lens that which I call the undo test lens that which I call the undo test that you can sort of use to size up the that you can sort of use to size up the that you can sort of use to size up the other three primitives. Uh so three other three primitives. Uh so three other three primitives. Uh so three things that you basically enforce and things that you basically enforce and things that you basically enforce and then one question you sort of ask about then one question you sort of ask about then one question you sort of ask about all of them.

So here's basically what bugged me uh So here's basically what bugged me uh after the cleanup was done for the after the cleanup was done for the after the cleanup was done for the incident that I was just showing you. Uh incident that I was just showing you. Uh incident that I was just showing you. Uh the the agent technically hadn't done the the agent technically hadn't done the the agent technically hadn't done anything that I couldn't have done. Uh anything that I couldn't have done. Uh anything that I couldn't have done. Uh it was using my token after all. The the it was using my token after all. The the it was using my token after all. The the failure wasn't the model itself. The failure wasn't the model itself. The failure wasn't the model itself. The failure was that I was giving the agent failure was that I was giving the agent failure was that I was giving the agent unbounded amount of power to do unbounded amount of power to do unbounded amount of power to do something that I wasn't watching super something that I wasn't watching super something that I wasn't watching super intently. Um and it was we've basically intently. Um and it was we've basically intently. Um and it was we've basically solved this the same kind of problem solved this the same kind of problem solved this the same kind of problem except not for agents but like the first except not for agents but like the first except not for agents but like the first time you onboarded like a junior time you onboarded like a junior time you onboarded like a junior engineer on onto your own teams, right? engineer on onto your own teams, right? engineer on onto your own teams, right? So just think about it like we don't we So just think about it like we don't we So just think about it like we don't we don't basically sit around watching um don't basically sit around watching um don't basically sit around watching um every engineer or new boarded person every engineer or new boarded person every engineer or new boarded person just like type out every keystroke on just like type out every keystroke on just like type out every keystroke on their keyboard, right? Like we're not their keyboard, right? Like we're not their keyboard, right? Like we're not behind their chairs just watching behind their chairs just watching behind their chairs just watching everything that they're doing. Uh there everything that they're doing. Uh there everything that they're doing. Uh there is always a path on whenever a new is always a path on whenever a new is always a path on whenever a new engineer wants to ask for something. Um engineer wants to ask for something. Um engineer wants to ask for something. Um there is always an escalation path. uh there is always an escalation path. uh there is always an escalation path. uh the catastrophic stuff is just the catastrophic stuff is just the catastrophic stuff is just structurally out of reach for them. So structurally out of reach for them. So structurally out of reach for them. So the the other side of the coin is that the the other side of the coin is that the the other side of the coin is that the agents are very different. They they the agents are very different. They they the agents are very different. They they never get tired. They never sleep. And never get tired. They never sleep. And never get tired. They never sleep. And every so often they're just like very every so often they're just like very every so often they're just like very confidently wrong. Um and if I'm being confidently wrong. Um and if I'm being confidently wrong. Um and if I'm being honest, that's that's maybe most of us honest, that's that's maybe most of us honest, that's that's maybe most of us on the first month of our jobs anyways.

on the first month of our jobs anyways. on the first month of our jobs anyways. And so what what we're going to talk And so what what we're going to talk And so what what we're going to talk about in a little bit in is just this about in a little bit in is just this about in a little bit in is just this idea of like your onboarding checklist, idea of like your onboarding checklist, idea of like your onboarding checklist, but written down as policy for for but written down as policy for for but written down as policy for for agents. agents. agents. So that and that brings me to the word So that and that brings me to the word So that and that brings me to the word in the title of my my talk here is that in the title of my my talk here is that in the title of my my talk here is that the the the standard fix for an incident the the the standard fix for an incident the the the standard fix for an incident like that is basically that you narrow like that is basically that you narrow like that is basically that you narrow the token scope, right? Like you just the token scope, right? Like you just the token scope, right? Like you just take the deletes away effectively. You take the deletes away effectively. You take the deletes away effectively. You would technically never do that for like would technically never do that for like would technically never do that for like a new hire, right? Like you would you a new hire, right? Like you would you a new hire, right? Like you would you wouldn't take the whole verb away from wouldn't take the whole verb away from wouldn't take the whole verb away from them. And it also doesn't work for an them. And it also doesn't work for an them. And it also doesn't work for an agent either. Maybe it works for about a agent either. Maybe it works for about a agent either. Maybe it works for about a week, maybe two, but then you eventually week, maybe two, but then you eventually week, maybe two, but then you eventually end up in a situation where the agent is end up in a situation where the agent is end up in a situation where the agent is genuinely trying to delete something genuinely trying to delete something genuinely trying to delete something that it feels like is just not in the that it feels like is just not in the that it feels like is just not in the infrastructure and you will just be infrastructure and you will just be infrastructure and you will just be there sitting and pressing enter by hand there sitting and pressing enter by hand there sitting and pressing enter by hand all over again. The the core concept all over again. The the core concept all over again. The the core concept with a token that I feel like is wrong with a token that I feel like is wrong with a token that I feel like is wrong is that a token is a boolean. It's just is that a token is a boolean. It's just is that a token is a boolean. It's just a yes or no. It's a static list of a yes or no. It's a static list of a yes or no. It's a static list of scopes. Um you either have it or you scopes. Um you either have it or you scopes. Um you either have it or you don't have it. So if the token list is don't have it. So if the token list is don't have it. So if the token list is too tight, then your agent is too tight, then your agent is too tight, then your agent is effectively useless. If the token list effectively useless. If the token list effectively useless. If the token list is too wide, then you're maybe writing is too wide, then you're maybe writing is too wide, then you're maybe writing writing like a postmortm. A budget is a writing like a postmortm. A budget is a writing like a postmortm. A budget is a very different shape. A budget is is um very different shape. A budget is is um very different shape. A budget is is um not just like one number. It just has not just like one number. It just has not just like one number. It just has like four different four different like four different four different like four different four different dimensions, right? How much can the dimensions, right? How much can the dimensions, right? How much can the agent do? Uh how fast can it do it? What agent do? Uh how fast can it do it? What agent do? Uh how fast can it do it? What can it undo on its own? And then who's can it undo on its own? And then who's can it undo on its own? And then who's noticing while it's actually taking noticing while it's actually taking noticing while it's actually taking those actions? And that's roughly the those actions? And that's roughly the those actions? And that's roughly the umbrella for like my four primitives. Uh umbrella for like my four primitives. Uh umbrella for like my four primitives. Uh is that each of the things that follow, is that each of the things that follow, is that each of the things that follow, they're basically being um one of the they're basically being um one of the they're basically being um one of the things along these dimensions. and

things along these dimensions. and things along these dimensions. and they're going to replace a yes or no they're going to replace a yes or no they're going to replace a yes or no question with a budget. question with a budget. question with a budget. So, I'm going to start off with this So, I'm going to start off with this So, I'm going to start off with this first verb which I call asymmetric first verb which I call asymmetric first verb which I call asymmetric verbs. Uh, and by verbs I simply means verbs. Uh, and by verbs I simply means verbs. Uh, and by verbs I simply means operations or actions that an agent can operations or actions that an agent can operations or actions that an agent can take. So, like it could be API calls, it take. So, like it could be API calls, it take. So, like it could be API calls, it could be uh CLI commands, it could be could be uh CLI commands, it could be could be uh CLI commands, it could be really anything. And they're asymmetric really anything. And they're asymmetric really anything. And they're asymmetric because the same sized action even because the same sized action even because the same sized action even though it looks same size, they're they though it looks same size, they're they though it looks same size, they're they can have very different blast radius in can have very different blast radius in can have very different blast radius in actuality depending on which direction actuality depending on which direction actuality depending on which direction it goes. So the the core point is that it goes. So the the core point is that it goes. So the the core point is that you need to stop thinking about you need to stop thinking about you need to stop thinking about resources for a second and think about resources for a second and think about resources for a second and think about verbs. So specifically think about what verbs. So specifically think about what verbs. So specifically think about what happens when one of the verbs goes happens when one of the verbs goes happens when one of the verbs goes wrong. So some verbs they fail out loud. wrong. So some verbs they fail out loud. wrong. So some verbs they fail out loud. So in this particular example, let's say So in this particular example, let's say So in this particular example, let's say if an agent decides to unskip a test and if an agent decides to unskip a test and if an agent decides to unskip a test and say it's the wrong call, the worst that say it's the wrong call, the worst that say it's the wrong call, the worst that would happen is CI would go red uh for a would happen is CI would go red uh for a would happen is CI would go red uh for a bunch of people. Um same with paging. bunch of people. Um same with paging. bunch of people. Um same with paging. Like if an agent decides to page a human Like if an agent decides to page a human Like if an agent decides to page a human and if it's the wrong call, the worst and if it's the wrong call, the worst and if it's the wrong call, the worst that's happening is that it's a nuisance that's happening is that it's a nuisance that's happening is that it's a nuisance for the on call. But there's always a for the on call. But there's always a for the on call. But there's always a human to correct it. There are other human to correct it. There are other human to correct it. There are other verbs that fail silently, right? So if verbs that fail silently, right? So if verbs that fail silently, right? So if the agent decides to skip a test uh due the agent decides to skip a test uh due the agent decides to skip a test uh due to whatever reason, it shouldn't have.

to whatever reason, it shouldn't have. to whatever reason, it shouldn't have. Nothing technically turns red. A real Nothing technically turns red. A real Nothing technically turns red. A real bug can actually walk into production bug can actually walk into production bug can actually walk into production with green checks and nobody would with green checks and nobody would with green checks and nobody would notice it until much later. So unskip notice it until much later. So unskip notice it until much later. So unskip and skip in this example, they're and skip in this example, they're and skip in this example, they're effectively the they're the same kind of effectively the they're the same kind of effectively the they're the same kind of action, but the difference is which of action, but the difference is which of action, but the difference is which of the failures would show up on a the failures would show up on a the failures would show up on a dashboard and which one wouldn't. And so dashboard and which one wouldn't. And so dashboard and which one wouldn't. And so the core idea is that you give um access the core idea is that you give um access the core idea is that you give um access to verbs that can fail out loud loudly to verbs that can fail out loud loudly to verbs that can fail out loud loudly on a dashboard to your agent and for the on a dashboard to your agent and for the on a dashboard to your agent and for the other ones just involve a human. other ones just involve a human. other ones just involve a human. Uh just a bit of context on CI stuff and Uh just a bit of context on CI stuff and Uh just a bit of context on CI stuff and how this plays out for us is that we how this plays out for us is that we how this plays out for us is that we have a test quarantining service behind have a test quarantining service behind have a test quarantining service behind the scenes and it basically holds a list the scenes and it basically holds a list the scenes and it basically holds a list of um all the tests that are currently of um all the tests that are currently of um all the tests that are currently skipped because an on call decided that skipped because an on call decided that skipped because an on call decided that it had to uh they had to like break it had to uh they had to like break it had to uh they had to like break glass a certain situation whenever there glass a certain situation whenever there glass a certain situation whenever there was like an incident or something. Now was like an incident or something. Now was like an incident or something. Now the agent has the ability to reenable the agent has the ability to reenable the agent has the ability to reenable any one of these own depending on like any one of these own depending on like any one of these own depending on like when it evaluates when the tests are uh when it evaluates when the tests are uh when it evaluates when the tests are uh working fine and stuff. Again if it's working fine and stuff. Again if it's working fine and stuff. Again if it's the wrong call the worst that would the wrong call the worst that would the wrong call the worst that would happen is a bunch of different builds happen is a bunch of different builds happen is a bunch of different builds and tests would just like start showing and tests would just like start showing and tests would just like start showing up red and then a human can actually put up red and then a human can actually put up red and then a human can actually put them back very cheaply. The skip is a them back very cheaply. The skip is a them back very cheaply. The skip is a break glass verb itself right like it's break glass verb itself right like it's break glass verb itself right like it's basically as I mentioned what our on basically as I mentioned what our on basically as I mentioned what our on call would reach for uh during a very call would reach for uh during a very call would reach for uh during a very critical situation during under pressure critical situation during under pressure critical situation during under pressure and stuff. And so in that scenario if and stuff. And so in that scenario if and stuff. And so in that scenario if like uh an agent actually gets it wrong like uh an agent actually gets it wrong like uh an agent actually gets it wrong a real production bug can actually a a real production bug can actually a a real production bug can actually a real bug can walk into production. So real bug can walk into production. So real bug can walk into production. So this needs a human and it will always this needs a human and it will always this needs a human and it will always leave an audit trail. Uh and the key leave an audit trail. Uh and the key leave an audit trail. Uh and the key detail is that the agent itself is not detail is that the agent itself is not detail is that the agent itself is not responsible for writing the row or the responsible for writing the row or the responsible for writing the row or the audit trail itself. There is a proxy in audit trail itself. There is a proxy in audit trail itself. There is a proxy in the middle that I've highlighted that the middle that I've highlighted that the middle that I've highlighted that I'm going to talk about in a little bit.

I'm going to talk about in a little bit. I'm going to talk about in a little bit. Uh which is responsible for stamping the Uh which is responsible for stamping the Uh which is responsible for stamping the caller's main identity on every call caller's main identity on every call caller's main identity on every call whether it's a skip or an unskip. um the whether it's a skip or an unskip. um the whether it's a skip or an unskip. um the agent technically never holds uh the pen agent technically never holds uh the pen agent technically never holds uh the pen on its own provenence. on its own provenence. on its own provenence. The second primitive is about rate The second primitive is about rate The second primitive is about rate limits. Uh it's pretty standard concept limits. Uh it's pretty standard concept limits. Uh it's pretty standard concept but and this is the most concrete form but and this is the most concrete form but and this is the most concrete form of like the budget idea as a whole. So a of like the budget idea as a whole. So a of like the budget idea as a whole. So a ceiling that refills. So every caller ceiling that refills. So every caller ceiling that refills. So every caller gets a small amount of disruptive gets a small amount of disruptive gets a small amount of disruptive actions uh per time window. So you can actions uh per time window. So you can actions uh per time window. So you can spend them however you want. Uh there's spend them however you want. Uh there's spend them however you want. Uh there's no approval, there's no waiting. And if no approval, there's no waiting. And if no approval, there's no waiting. And if you cross the line, the request simply you cross the line, the request simply you cross the line, the request simply bounces back with a count saying that bounces back with a count saying that bounces back with a count saying that you're actually exceeding your count um you're actually exceeding your count um you're actually exceeding your count um your budget. You wait a bit and then the your budget. You wait a bit and then the your budget. You wait a bit and then the limit essentially refills. And that's limit essentially refills. And that's limit essentially refills. And that's that's the whole thing. So the agent that's the whole thing. So the agent that's the whole thing. So the agent gets full autonomy within the limit. And gets full autonomy within the limit. And gets full autonomy within the limit. And there is a hard ceiling on how bad a there is a hard ceiling on how bad a there is a hard ceiling on how bad a single loop can get. And every write single loop can get. And every write single loop can get. And every write effectively gets a rate limit. There are effectively gets a rate limit. There are effectively gets a rate limit. There are no exceptions to that. What changes is no exceptions to that. What changes is no exceptions to that. What changes is the size of the rate limit. So if if if the size of the rate limit. So if if if the size of the rate limit. So if if if I'm if I'm trying to delete a bunch of I'm if I'm trying to delete a bunch of I'm if I'm trying to delete a bunch of workloads in my own name space like my workloads in my own name space like my workloads in my own name space like my my rate limit might be higher but if I'm my rate limit might be higher but if I'm my rate limit might be higher but if I'm trying to touch resources in a shared trying to touch resources in a shared trying to touch resources in a shared namespace my budget or my rate limits namespace my budget or my rate limits namespace my budget or my rate limits might be smaller.

So this is uh effectively the So this is uh effectively the solutioning for the incident that I was solutioning for the incident that I was solutioning for the incident that I was showing earlier. Um after the incident showing earlier. Um after the incident showing earlier. Um after the incident the team sort of that sits next to mine the team sort of that sits next to mine the team sort of that sits next to mine they they built an admission web hook of they they built an admission web hook of they they built an admission web hook of sorts uh whose sole job is to cap the sorts uh whose sole job is to cap the sorts uh whose sole job is to cap the number of deletes at a fixed number per number of deletes at a fixed number per number of deletes at a fixed number per hour per resource kind per name space. hour per resource kind per name space. hour per resource kind per name space. There is always a bypass flag because There is always a bypass flag because There is always a bypass flag because sometimes you genuinely want to delete sometimes you genuinely want to delete sometimes you genuinely want to delete more than you're allowed for and maybe more than you're allowed for and maybe more than you're allowed for and maybe some on call scenario or whatever. And some on call scenario or whatever. And some on call scenario or whatever. And the part that I absolutely love in this the part that I absolutely love in this the part that I absolutely love in this case is that inside a cloud code session case is that inside a cloud code session case is that inside a cloud code session or inside an agent session effectively or inside an agent session effectively or inside an agent session effectively the the bypass flag simply refuses to do the the bypass flag simply refuses to do the the bypass flag simply refuses to do anything. All it's going to do is tell anything. All it's going to do is tell anything. All it's going to do is tell the agent to ask the human to run the the agent to ask the human to run the the agent to ask the human to run the command itself. So the agent effectively command itself. So the agent effectively command itself. So the agent effectively gets the rate limit and the human keeps gets the rate limit and the human keeps gets the rate limit and the human keeps the override and nobody effectively has the override and nobody effectively has the override and nobody effectively has to file a ticket for uh the limit to file a ticket for uh the limit to file a ticket for uh the limit because it just like refills. because it just like refills. because it just like refills. The third primitive is about this The third primitive is about this The third primitive is about this concept of trip wires over allow lists. concept of trip wires over allow lists. concept of trip wires over allow lists. So in my mind like an allow list is So in my mind like an allow list is So in my mind like an allow list is effectively a guess that you're making effectively a guess that you're making effectively a guess that you're making up front about what the agent needs or up front about what the agent needs or up front about what the agent needs or about model behavior or agent behavior about model behavior or agent behavior about model behavior or agent behavior itself. uh I feel like it's pretty itself. uh I feel like it's pretty itself. uh I feel like it's pretty static and you write it up front before static and you write it up front before static and you write it up front before before you have any data on like how the before you have any data on like how the before you have any data on like how the agent is behaving in different agent is behaving in different agent is behaving in different situations. A trip wire on the other situations. A trip wire on the other situations. A trip wire on the other hand is how you get that data like after hand is how you get that data like after hand is how you get that data like after the fact. So for cheap actions you let the fact. So for cheap actions you let the fact. So for cheap actions you let the agent act and every action gets the agent act and every action gets the agent act and every action gets recorded with the actor sort of stamp recorded with the actor sort of stamp recorded with the actor sort of stamp identity stamp and these two kind of go identity stamp and these two kind of go identity stamp and these two kind of go well together like rate limits are the well together like rate limits are the well together like rate limits are the enforcement um they they put a hard enforcement um they they put a hard enforcement um they they put a hard limit on uh the rate itself. Trip wires limit on uh the rate itself. Trip wires limit on uh the rate itself. Trip wires are how you find out what actually are how you find out what actually are how you find out what actually happened so you've got something to happened so you've got something to happened so you've got something to react to. So you effectively watch the react to. So you effectively watch the react to. So you effectively watch the aggregate and not like individual calls.

aggregate and not like individual calls. aggregate and not like individual calls. And usually when a trip wire goes off And usually when a trip wire goes off And usually when a trip wire goes off the fix is like maybe one or two lines the fix is like maybe one or two lines the fix is like maybe one or two lines in the agents context and not really a in the agents context and not really a in the agents context and not really a big code change. Uh the core point is big code change. Uh the core point is big code change. Uh the core point is that allow lists don't really get better that allow lists don't really get better that allow lists don't really get better over time. They can get stale but trip over time. They can get stale but trip over time. They can get stale but trip wires do get better over time. wires do get better over time. wires do get better over time. So here's like a loopin practice for us. So here's like a loopin practice for us. So here's like a loopin practice for us. So like we track one number uh which is So like we track one number uh which is So like we track one number uh which is the number of investigation threads that the number of investigation threads that the number of investigation threads that our agent is launching per hour for a our agent is launching per hour for a our agent is launching per hour for a given test job failure. So one morning given test job failure. So one morning given test job failure. So one morning say the number was way above the say the number was way above the say the number was way above the baseline and the trip fire page is on baseline and the trip fire page is on baseline and the trip fire page is on call and that part's important because a call and that part's important because a call and that part's important because a trip wire that nobody sees is trip wire that nobody sees is trip wire that nobody sees is practically useless. Uh it pages after practically useless. Uh it pages after practically useless. Uh it pages after the right has already happened after the the right has already happened after the the right has already happened after the the limit has been crossed not before. the limit has been crossed not before. the limit has been crossed not before. It's effectively the smoke detector not It's effectively the smoke detector not It's effectively the smoke detector not the lock on the door. Uh the agent has the lock on the door. Uh the agent has the lock on the door. Uh the agent has spun up in this particular scenario a spun up in this particular scenario a spun up in this particular scenario a bunch of like investigation threads for bunch of like investigation threads for bunch of like investigation threads for dozens of job that were all failing with dozens of job that were all failing with dozens of job that were all failing with the same kind of error signature. Each the same kind of error signature. Each the same kind of error signature. Each thread effectively looked reasonable on thread effectively looked reasonable on thread effectively looked reasonable on its own, but like if you took them in its own, but like if you took them in its own, but like if you took them in aggregate, you would realize that it was aggregate, you would realize that it was aggregate, you would realize that it was actually an infrastructure failure that actually an infrastructure failure that actually an infrastructure failure that was causing the same test failure was causing the same test failure was causing the same test failure signatures across the board. So the fix signatures across the board. So the fix signatures across the board. So the fix in this case was as simply as telling in this case was as simply as telling in this case was as simply as telling the agent on like, hey, the next time the agent on like, hey, the next time the agent on like, hey, the next time you encounter something like this, maybe you encounter something like this, maybe you encounter something like this, maybe try to correlate a bunch of different try to correlate a bunch of different try to correlate a bunch of different failures and test jobs and stuff before failures and test jobs and stuff before failures and test jobs and stuff before launching a separate investigation launching a separate investigation launching a separate investigation thread. So that correlation is thread. So that correlation is thread. So that correlation is important. Um and the next time when important. Um and the next time when important. Um and the next time when this happened, uh it did exactly that this happened, uh it did exactly that this happened, uh it did exactly that because that that example and that line because that that example and that line because that that example and that line of like how it should react and like how of like how it should react and like how of like how it should react and like how it should like uh debug things was was it should like uh debug things was was it should like uh debug things was was right there to guide it.

right there to guide it. right there to guide it. So that's three rough primitives. Now So that's three rough primitives. Now So that's three rough primitives. Now the lens I mentioned at the start is the lens I mentioned at the start is the lens I mentioned at the start is this idea of undo test. And this one's this idea of undo test. And this one's this idea of undo test. And this one's slightly different. It's it's not slightly different. It's it's not slightly different. It's it's not something you effectively enforce in something you effectively enforce in something you effectively enforce in code. It's the question you ask when code. It's the question you ask when code. It's the question you ask when you're sizing the other three. So it's you're sizing the other three. So it's you're sizing the other three. So it's two questions really. One, can the agent two questions really. One, can the agent two questions really. One, can the agent put it back by itself? and how bad would put it back by itself? and how bad would put it back by itself? and how bad would the impact be if it actually got it the impact be if it actually got it the impact be if it actually got it wrong. And this this this sounds like wrong. And this this this sounds like wrong. And this this this sounds like asymmetric verbs in a sense, but it's asymmetric verbs in a sense, but it's asymmetric verbs in a sense, but it's kind of difference because the verbs ask kind of difference because the verbs ask kind of difference because the verbs ask whether you would notice the failure and whether you would notice the failure and whether you would notice the failure and undo asks whether you can recover from undo asks whether you can recover from undo asks whether you can recover from it. So if if if the agent can it. So if if if the agent can it. So if if if the agent can effectively roll back its own change and effectively roll back its own change and effectively roll back its own change and the blast radius is acceptable to you, the blast radius is acceptable to you, the blast radius is acceptable to you, you effectively log it and you let it you effectively log it and you let it you effectively log it and you let it go. If either of the answers is no, then go. If either of the answers is no, then go. If either of the answers is no, then you effectively need a second key and you effectively need a second key and you effectively need a second key and the second key is not something that the the second key is not something that the the second key is not something that the agent holds itself. It has to be someone agent holds itself. It has to be someone agent holds itself. It has to be someone else. Um, and there has to be an audit else. Um, and there has to be an audit else. Um, and there has to be an audit record so that you can you can track on record so that you can you can track on record so that you can you can track on like what happened, why the second key like what happened, why the second key like what happened, why the second key was involved and stuff like that. was involved and stuff like that. was involved and stuff like that. Let me let me just show you this another Let me let me just show you this another Let me let me just show you this another example of like how this works outside example of like how this works outside example of like how this works outside of like the CI domain specifically. So of like the CI domain specifically. So of like the CI domain specifically. So our agent has a key for one of our our agent has a key for one of our our agent has a key for one of our feature flag services. So on the canary feature flag services. So on the canary feature flag services. So on the canary side which is basically our staging side which is basically our staging side which is basically our staging traffic and a bunch of like dog fooding traffic and a bunch of like dog fooding traffic and a bunch of like dog fooding customers the agent effectively has the customers the agent effectively has the customers the agent effectively has the full dial. It has the ability to um ramp full dial. It has the ability to um ramp full dial. It has the ability to um ramp up a specific feature flag roll it out up a specific feature flag roll it out up a specific feature flag roll it out to uh all of the canary traffic and it to uh all of the canary traffic and it to uh all of the canary traffic and it can take it all the way from zero to 100 can take it all the way from zero to 100 can take it all the way from zero to 100 and it also has the ability to sort of and it also has the ability to sort of and it also has the ability to sort of uh look at any bugs that are being filed uh look at any bugs that are being filed uh look at any bugs that are being filed and toggle it back off and on. What the and toggle it back off and on. What the and toggle it back off and on. What the agents key is not scoped to do is for agents key is not scoped to do is for agents key is not scoped to do is for promoting the flag to real production.

promoting the flag to real production. promoting the flag to real production. The best that an agent can do for now in The best that an agent can do for now in The best that an agent can do for now in a lot of cases is that it can propose uh a lot of cases is that it can propose uh a lot of cases is that it can propose uh that an that someone actually promote that an that someone actually promote that an that someone actually promote the feature flag in production because the feature flag in production because the feature flag in production because it's been tested out in Canary. But it's been tested out in Canary. But it's been tested out in Canary. But that's pretty much it. Um the second key that's pretty much it. Um the second key that's pretty much it. Um the second key in this scenario is not necessarily a in this scenario is not necessarily a in this scenario is not necessarily a new O system. It's a scoped key for new O system. It's a scoped key for new O system. It's a scoped key for production and a scoped key for Canary. So this is what it looks like day-to-day So this is what it looks like day-to-day for for me personally. is like I I ask for for me personally. is like I I ask for for me personally. is like I I ask cla tag in our slack channel to to own cla tag in our slack channel to to own cla tag in our slack channel to to own the loop the entire loop uh on on the loop the entire loop uh on on the loop the entire loop uh on on rolling out a flag. It can it can rolling out a flag. It can it can rolling out a flag. It can it can basically ask me a clarifying question basically ask me a clarifying question basically ask me a clarifying question on whether this is for internal dog on whether this is for internal dog on whether this is for internal dog fooding group or early access or fooding group or early access or fooding group or early access or whatever. Uh and the important part is whatever. Uh and the important part is whatever. Uh and the important part is that I'm not in the middle of any of that I'm not in the middle of any of that I'm not in the middle of any of these things. Every action that the these things. Every action that the these things. Every action that the agent is taking is stamped with its own agent is taking is stamped with its own agent is taking is stamped with its own identity and not mine. And that stamp is identity and not mine. And that stamp is identity and not mine. And that stamp is basically what ties the whole thing the basically what ties the whole thing the basically what ties the whole thing the whole thing together. whole thing together. whole thing together. So the these these primitives they So the these these primitives they So the these these primitives they compose together nicely because each one compose together nicely because each one compose together nicely because each one asks us a different question right the asks us a different question right the asks us a different question right the verbs ask what the agent is touching the verbs ask what the agent is touching the verbs ask what the agent is touching the rate limit asks how often and the trip rate limit asks how often and the trip rate limit asks how often and the trip wires catch what happened afterwards and wires catch what happened afterwards and wires catch what happened afterwards and the undo test is sort of the length that the undo test is sort of the length that the undo test is sort of the length that I personally used to to size all these I personally used to to size all these I personally used to to size all these three now the original sort of cold open three now the original sort of cold open three now the original sort of cold open delete that I was showing you the the delete that I was showing you the the delete that I was showing you the the incident itself the rate limit itself incident itself the rate limit itself incident itself the rate limit itself would have capped it at a few couple of would have capped it at a few couple of would have capped it at a few couple of tens of workloads itself the undo test tens of workloads itself the undo test tens of workloads itself the undo test if you would be thinking about it it if you would be thinking about it it if you would be thinking about it it will basically tell you that you can't will basically tell you that you can't will basically tell you that you can't unddelete a running job in someone unddelete a running job in someone unddelete a running job in someone else's namespace. Um so so anything past else's namespace. Um so so anything past else's namespace. Um so so anything past the cap basically need a human with the the cap basically need a human with the the cap basically need a human with the with a with a second key. So the the with a with a second key. So the the with a with a second key. So the the important part is that you don't need important part is that you don't need important part is that you don't need all the checks in every right scenario.

all the checks in every right scenario. all the checks in every right scenario. Only some of these might be relevant for Only some of these might be relevant for Only some of these might be relevant for like whatever kind of action you're like whatever kind of action you're like whatever kind of action you're trying to evaluate. Uh and if any of trying to evaluate. Uh and if any of trying to evaluate. Uh and if any of these really sound familiar, it it these really sound familiar, it it these really sound familiar, it it should because this is effectively some should because this is effectively some should because this is effectively some sort of onboarding checklist for your sort of onboarding checklist for your sort of onboarding checklist for your engineers like what can a new engineer engineers like what can a new engineer engineers like what can a new engineer touch, how much rope do they get, who touch, how much rope do they get, who touch, how much rope do they get, who signs off on their operations, and how signs off on their operations, and how signs off on their operations, and how do we know it's effectively working? We do we know it's effectively working? We do we know it's effectively working? We just wrote it for people. It's now the just wrote it for people. It's now the just wrote it for people. It's now the same checklist that we want for agents. same checklist that we want for agents. same checklist that we want for agents. So where does all the policy that I've So where does all the policy that I've So where does all the policy that I've been talking about really live? Uh it's been talking about really live? Uh it's been talking about really live? Uh it's two places and I feel like you need two places and I feel like you need two places and I feel like you need both. The first one is text. Uh that's both. The first one is text. Uh that's both. The first one is text. Uh that's prompts. That's your context files prompts. That's your context files prompts. That's your context files markdown the agent reads before it can markdown the agent reads before it can markdown the agent reads before it can act. And this is where you can explain act. And this is where you can explain act. And this is where you can explain your why the the intent. We've we got your why the the intent. We've we got your why the the intent. We've we got exactly the same sentence that I was exactly the same sentence that I was exactly the same sentence that I was showing earlier um written down in a showing earlier um written down in a showing earlier um written down in a markdown file. It works about 80% of the markdown file. It works about 80% of the markdown file. It works about 80% of the time. Uh the the upside is that it's time. Uh the the upside is that it's time. Uh the the upside is that it's very cheap to change and you can explain very cheap to change and you can explain very cheap to change and you can explain the reasoning. The downside is you have the reasoning. The downside is you have the reasoning. The downside is you have to garden it because the the files can to garden it because the the files can to garden it because the the files can grow over time and at the end of the day grow over time and at the end of the day grow over time and at the end of the day it's just advice. Uh text can shape the it's just advice. Uh text can shape the it's just advice. Uh text can shape the intent but there is no enforcement intent but there is no enforcement intent but there is no enforcement anywhere. The the second place is anywhere. The the second place is anywhere. The the second place is infrastructure uh which is the proxy infrastructure uh which is the proxy infrastructure uh which is the proxy layer for us. So the proxy is is not layer for us. So the proxy is is not layer for us. So the proxy is is not reading the prompt. It doesn't know why reading the prompt. It doesn't know why reading the prompt. It doesn't know why the agent wants to do something and it the agent wants to do something and it the agent wants to do something and it doesn't really care. It will see a doesn't really care. It will see a doesn't really care. It will see a delete happening. It'll see like a delete happening. It'll see like a delete happening. It'll see like a budget being crossed and it'll simply budget being crossed and it'll simply budget being crossed and it'll simply return a 403 and that's the whole return a 403 and that's the whole return a 403 and that's the whole conversation really. It's narrow. It's conversation really. It's narrow. It's conversation really. It's narrow. It's deterministic. It it counts, compares, deterministic. It it counts, compares, deterministic. It it counts, compares, it can allow a delete or deny. What it it can allow a delete or deny. What it it can allow a delete or deny. What it can't do is explain the why and a clever can't do is explain the why and a clever can't do is explain the why and a clever like prompt injection cannot really talk like prompt injection cannot really talk like prompt injection cannot really talk it out of the rule itself. So you need it out of the rule itself. So you need it out of the rule itself. So you need both. Uh the text shapes what an agent

both. Uh the text shapes what an agent both. Uh the text shapes what an agent is trying to do and infra is bounding is trying to do and infra is bounding is trying to do and infra is bounding how wrong can it go. how wrong can it go. how wrong can it go. So zooming out, this is where uh the the So zooming out, this is where uh the the So zooming out, this is where uh the the infrastructure sort of really lives for infrastructure sort of really lives for infrastructure sort of really lives for us. So like every agent session has its us. So like every agent session has its us. So like every agent session has its own proxy running right next to it. The own proxy running right next to it. The own proxy running right next to it. The agent starts off by reading its own agent starts off by reading its own agent starts off by reading its own context file. The the markdown files context file. The the markdown files context file. The the markdown files that I was showing earlier that's that's that I was showing earlier that's that's that I was showing earlier that's that's the text layer and every outbound call the text layer and every outbound call the text layer and every outbound call goes through the proxy after that point goes through the proxy after that point goes through the proxy after that point which is the infrastructure layer. So which is the infrastructure layer. So which is the infrastructure layer. So the proxy is not like whatever the agent the proxy is not like whatever the agent the proxy is not like whatever the agent is trying to do the the proxy is only is trying to do the the proxy is only is trying to do the the proxy is only responsible for stamping that action responsible for stamping that action responsible for stamping that action with the agent's identity. Uh like if with the agent's identity. Uh like if with the agent's identity. Uh like if the if the if the agent is trying to the if the if the agent is trying to the if the if the agent is trying to launch a bunch of jobs in a Kubernetes launch a bunch of jobs in a Kubernetes launch a bunch of jobs in a Kubernetes cluster the the proxy is not following cluster the the proxy is not following cluster the the proxy is not following it. what is following is the stamp it. what is following is the stamp it. what is following is the stamp itself, the identity itself. So the in itself, the identity itself. So the in itself, the identity itself. So the in this example, the cluster would write this example, the cluster would write this example, the cluster would write the stamp onto the job as a label. Um the stamp onto the job as a label. Um the stamp onto the job as a label. Um and every child job or anything that's and every child job or anything that's and every child job or anything that's happening afterwards simply inherits the happening afterwards simply inherits the happening afterwards simply inherits the same identity and the every safeguard same identity and the every safeguard same identity and the every safeguard that is there in the rest of our systems that is there in the rest of our systems that is there in the rest of our systems that that are simply they're simply that that are simply they're simply that that are simply they're simply reading that like one label. Uh whether reading that like one label. Uh whether reading that like one label. Uh whether it's ownership, whether it's kotaas, it's ownership, whether it's kotaas, it's ownership, whether it's kotaas, rate limits, approvals, trip wires, rate limits, approvals, trip wires, rate limits, approvals, trip wires, whatever it is, they're all keyed on the whatever it is, they're all keyed on the whatever it is, they're all keyed on the same stamp and the agent never got to same stamp and the agent never got to same stamp and the agent never got to touch it.

touch it. touch it. So that's roughly the shape of it. But So that's roughly the shape of it. But So that's roughly the shape of it. But before I wrap up, I want to spend like before I wrap up, I want to spend like before I wrap up, I want to spend like 30 seconds on why it has to be a proxy 30 seconds on why it has to be a proxy 30 seconds on why it has to be a proxy layer, like a different layer that is layer, like a different layer that is layer, like a different layer that is doing the stamping on this identity and doing the stamping on this identity and doing the stamping on this identity and not the caller itself. Because without not the caller itself. Because without not the caller itself. Because without it, the agent can roughly pick a it, the agent can roughly pick a it, the agent can roughly pick a different name. Um, and every limit that different name. Um, and every limit that different name. Um, and every limit that you've basically set resets. you've basically set resets. you've basically set resets. So say the agent has the ability to set So say the agent has the ability to set So say the agent has the ability to set its own identity in a header, right? So its own identity in a header, right? So its own identity in a header, right? So let's say it hits a specific limit and let's say it hits a specific limit and let's say it hits a specific limit and what's the easiest fix from an agent's what's the easiest fix from an agent's what's the easiest fix from an agent's point of view that it can do? it'll just point of view that it can do? it'll just point of view that it can do? it'll just change the header. In this case, it'll change the header. In this case, it'll change the header. In this case, it'll just say instead of such in such in two just say instead of such in such in two just say instead of such in such in two and voila, you just have a fresh budget and voila, you just have a fresh budget and voila, you just have a fresh budget to work with. Now, in this case, you to work with. Now, in this case, you to work with. Now, in this case, you technically don't have a rate limit. You technically don't have a rate limit. You technically don't have a rate limit. You just have a suggestion. Uh with the just have a suggestion. Uh with the just have a suggestion. Uh with the proxy in the path, the agent never gets proxy in the path, the agent never gets proxy in the path, the agent never gets to say who it is. The proxy already to say who it is. The proxy already to say who it is. The proxy already knows. It's the thing that's holding knows. It's the thing that's holding knows. It's the thing that's holding real credentials and it stamps every real credentials and it stamps every real credentials and it stamps every call with the identity that it already call with the identity that it already call with the identity that it already knows, not the one that agent claims. knows, not the one that agent claims. knows, not the one that agent claims. And because proxy is the one that is And because proxy is the one that is And because proxy is the one that is stamping, you also get this per session stamping, you also get this per session stamping, you also get this per session ID so that you're able to differentiate ID so that you're able to differentiate ID so that you're able to differentiate different sessions that are running all different sessions that are running all different sessions that are running all for you and see which one's overreacting for you and see which one's overreacting for you and see which one's overreacting or which one's not acting as it's or which one's not acting as it's or which one's not acting as it's supposed to be.

supposed to be. supposed to be. U so again just stuffs that I want you U so again just stuffs that I want you U so again just stuffs that I want you to take home with is give the agent to take home with is give the agent to take home with is give the agent access to verbs that would fail out loud access to verbs that would fail out loud access to verbs that would fail out loud uh and keep the human on the ones that uh and keep the human on the ones that uh and keep the human on the ones that can fail out quietly. Two, you you put a can fail out quietly. Two, you you put a can fail out quietly. Two, you you put a ceiling on every right and that has the ceiling on every right and that has the ceiling on every right and that has the ability to refill on its own so nobody's ability to refill on its own so nobody's ability to refill on its own so nobody's effectively filing tickets. Three, you effectively filing tickets. Three, you effectively filing tickets. Three, you watch the aggregate and not the watch the aggregate and not the watch the aggregate and not the individual calls that help you individual calls that help you individual calls that help you understand the agents behavior. Um, and understand the agents behavior. Um, and understand the agents behavior. Um, and then you fix whatever you can with the then you fix whatever you can with the then you fix whatever you can with the sentence. And four, you use the undo sentence. And four, you use the undo sentence. And four, you use the undo test to sort of size all of these three test to sort of size all of these three test to sort of size all of these three or any other u primitive that you have or any other u primitive that you have or any other u primitive that you have for your own write operations. And for your own write operations. And for your own write operations. And underneath all of this uh is is this underneath all of this uh is is this underneath all of this uh is is this concept of identity which has to come concept of identity which has to come concept of identity which has to come from the infrastructure, not from the from the infrastructure, not from the from the infrastructure, not from the request. I feel like that's one rule. If request. I feel like that's one rule. If request. I feel like that's one rule. If you get that one rule right, everything you get that one rule right, everything you get that one rule right, everything else is uh just just tuning. else is uh just just tuning. else is uh just just tuning. Thank you. Thank you. Thank you. [applause]