We Gave an Agent Production Code Access and Then Tried to Sleep at Night — Moritz Johner, Form3

>> Thanks everyone for joining in. >> Thanks everyone for joining in. Um thanks for the great intro, by the Um thanks for the great intro, by the Um thanks for the great intro, by the way. Um way. Um way. Um So, yeah. So, yeah. So, yeah. My talk today is about um so titles we My talk today is about um so titles we My talk today is about um so titles we give in Asia production code access give in Asia production code access give in Asia production code access and then try to sleep at night. Um so, and then try to sleep at night. Um so, and then try to sleep at night. Um so, it's mostly around dependency dependency it's mostly around dependency dependency it's mostly around dependency dependency patching, which is probably the most patching, which is probably the most patching, which is probably the most glamorous problem in software glamorous problem in software glamorous problem in software engineering. I guess everyone did it engineering. I guess everyone did it engineering. I guess everyone did it here at some point in the past. here at some point in the past. here at some point in the past. Um it really is like vacuuming. Like Um it really is like vacuuming. Like Um it really is like vacuuming. Like everyone loves it, can't wait to do do everyone loves it, can't wait to do do everyone loves it, can't wait to do do it again next week. Now, at our scale, it again next week. Now, at our scale, it again next week. Now, at our scale, we have thousands of repositories and it we have thousands of repositories and it we have thousands of repositories and it really is a backlog that never empties really is a backlog that never empties really is a backlog that never empties and and and you close 10 issues today and you know you close 10 issues today and you know you close 10 issues today and you know next week 20 more will arrive and you next week 20 more will arrive and you next week 20 more will arrive and you have to deal with them. have to deal with them. have to deal with them. So, naturally you think, "Sure. So, naturally you think, "Sure. So, naturally you think, "Sure. Let's just automate it. Let's just automate it. Let's just automate it. Dependabot exists. Dependabot exists. Dependabot exists. Renovate exists. Problem solved, right?" Renovate exists. Problem solved, right?" Renovate exists. Problem solved, right?" It isn't. It isn't. It isn't. Cuz these tools were really built for a Cuz these tools were really built for a Cuz these tools were really built for a world where fixing a CVE world where fixing a CVE world where fixing a CVE means um means um means um um looking at the manifest in a um looking at the manifest in a um looking at the manifest in a repository and just now repository and just now repository and just now bumping a version to the next version bumping a version to the next version bumping a version to the next version and that's it. Um but our world is much and that's it. Um but our world is much and that's it. Um but our world is much more complicated than that, at least more complicated than that, at least more complicated than that, at least nowadays in our world since we have nowadays in our world since we have nowadays in our world since we have containers.

containers. containers. So, the first problem is that the So, the first problem is that the So, the first problem is that the vulnerable thing isn't necessarily the vulnerable thing isn't necessarily the vulnerable thing isn't necessarily the thing that these tools can see. thing that these tools can see. thing that these tools can see. For instance, the CVE might live in an For instance, the CVE might live in an For instance, the CVE might live in an OS package that you use in your base OS package that you use in your base OS package that you use in your base image. image. image. It's not in your Dockerfile. It's just It's not in your Dockerfile. It's just It's not in your Dockerfile. It's just in the base image. in the base image. in the base image. Or if your Dockerfile pulls down a Or if your Dockerfile pulls down a Or if your Dockerfile pulls down a binary during the build process, binary during the build process, binary during the build process, it may see the URL, which might contain it may see the URL, which might contain it may see the URL, which might contain a version number, but it has no idea how a version number, but it has no idea how a version number, but it has no idea how to act on it. So, they can only look at to act on it. So, they can only look at to act on it. So, they can only look at the manifest, patch the manifest, but the manifest, patch the manifest, but the manifest, patch the manifest, but everything else is just simply everything else is just simply everything else is just simply invisible. invisible. invisible. Now, the second problem is patches never Now, the second problem is patches never Now, the second problem is patches never happen in isolation. happen in isolation. happen in isolation. So, when we look at the Go code base, So, when we look at the Go code base, So, when we look at the Go code base, sometimes you want to bump the Go sometimes you want to bump the Go sometimes you want to bump the Go runtime to the next minor version. runtime to the next minor version. runtime to the next minor version. When you do that, sometimes you might When you do that, sometimes you might When you do that, sometimes you might have to also bump the Go linter cuz the have to also bump the Go linter cuz the have to also bump the Go linter cuz the version are just intertwined. You have version are just intertwined. You have version are just intertwined. You have to bump both of them at at the same to bump both of them at at the same to bump both of them at at the same time. time. time. So, when you bump the linter, that So, when you bump the linter, that So, when you bump the linter, that sometimes introduces new linting rules, sometimes introduces new linting rules, sometimes introduces new linting rules, which then invalidates the code base, which then invalidates the code base, which then invalidates the code base, and you're left with this and you're left with this and you're left with this big mess. Now, these old tools, quote big mess. Now, these old tools, quote big mess. Now, these old tools, quote unquote old tools, Dependabot, unquote old tools, Dependabot, unquote old tools, Dependabot, well, they just bump the Go runtime well, they just bump the Go runtime well, they just bump the Go runtime version, create a PR, and walk away, and version, create a PR, and walk away, and version, create a PR, and walk away, and you're just left with this mess, and you you're just left with this mess, and you you're just left with this mess, and you need to figure out need to figure out need to figure out how to fix that.

So, you don't really have a like a So, you don't really have a like a patching problem, you also have like a patching problem, you also have like a patching problem, you also have like a reasoning problem that you reasoning problem that you reasoning problem that you um need to address here. um need to address here. um need to address here. You really want to have a tool that You really want to have a tool that You really want to have a tool that looks at the CVE and understands the looks at the CVE and understands the looks at the CVE and understands the full surface of it. Where does the CVE full surface of it. Where does the CVE full surface of it. Where does the CVE actually live? actually live? actually live? And needs to figure out what else needs And needs to figure out what else needs And needs to figure out what else needs to move in order to um get to a green to move in order to um get to a green to move in order to um get to a green CI. So, that's basically what we did. Um we So, that's basically what we did. Um we built Patch Pilot. built Patch Pilot. built Patch Pilot. We pushed to production, and We pushed to production, and We pushed to production, and eventually Infosec um came around the eventually Infosec um came around the eventually Infosec um came around the corner ask a very reasonable question, corner ask a very reasonable question, corner ask a very reasonable question, is this automation, or is it a supply is this automation, or is it a supply is this automation, or is it a supply chain incident waiting to happen? A useful coding agent A useful coding agent is a supply chain actor, whether you is a supply chain actor, whether you is a supply chain actor, whether you plan for that or not. That's the thesis plan for that or not. That's the thesis plan for that or not. That's the thesis of this talk, basically. of this talk, basically. of this talk, basically. It's not agents are dangerous, or agents It's not agents are dangerous, or agents It's not agents are dangerous, or agents are fine. It's the moment where you give are fine. It's the moment where you give are fine. It's the moment where you give an agent um an agent um an agent um production credentials in order to like production credentials in order to like production credentials in order to like be useful, be useful, be useful, it really becomes a supply chain actor, it really becomes a supply chain actor, it really becomes a supply chain actor, just like an engineer in your just like an engineer in your just like an engineer in your department. And you should apply similar department. And you should apply similar department. And you should apply similar or the same guardrails uh for the agent or the same guardrails uh for the agent or the same guardrails uh for the agent just as you do for engineers.

just as you do for engineers. just as you do for engineers. Now, this is a case study. We built Now, this is a case study. We built Now, this is a case study. We built this, we ran it in production. Infosec this, we ran it in production. Infosec this, we ran it in production. Infosec pushed back in several places. They were pushed back in several places. They were pushed back in several places. They were right. right. right. Um I just want to present to you what we Um I just want to present to you what we Um I just want to present to you what we came up with in the end. Um it's nothing came up with in the end. Um it's nothing came up with in the end. Um it's nothing new, it's nothing fancy. Um I just want new, it's nothing fancy. Um I just want new, it's nothing fancy. Um I just want to share to share to share share what we did and then hopefully share what we did and then hopefully share what we did and then hopefully have a discussion afterwards to see how have a discussion afterwards to see how have a discussion afterwards to see how we all figured this out together. Now, patch pilot has two layers. One, Now, patch pilot has two layers. One, it's a simple go application that is it's a simple go application that is it's a simple go application that is deterministic. It's boring. It just does deterministic. It's boring. It just does deterministic. It's boring. It just does orchestration. It discovers vulnerable orchestration. It discovers vulnerable orchestration. It discovers vulnerable artifacts. So, artifacts that's our OCI artifacts. So, artifacts that's our OCI artifacts. So, artifacts that's our OCI images. images. images. That's what we have in production. That's what we have in production. That's what we have in production. That's what we ship to our customers and That's what we ship to our customers and That's what we ship to our customers and that's the thing that we really care that's the thing that we really care that's the thing that we really care about. about. about. So, the go app discovers those, scans So, the go app discovers those, scans So, the go app discovers those, scans the images, the images, the images, and figures out which image is built by and figures out which image is built by and figures out which image is built by which repository. which repository. which repository. Because we need that link in order to Because we need that link in order to Because we need that link in order to later then work on the repositories. And this deterministic part is very And this deterministic part is very boring on purpose. It's very simple. boring on purpose. It's very simple. boring on purpose. It's very simple. And inside that, we spawn agents.

And inside that, we spawn agents. And inside that, we spawn agents. Now, these agents are there for the for Now, these agents are there for the for Now, these agents are there for the for the reasoning. For instance, when we see the reasoning. For instance, when we see the reasoning. For instance, when we see a CI failure, we just tell the agent, a CI failure, we just tell the agent, a CI failure, we just tell the agent, "Look, that's a CI failure. Please "Look, that's a CI failure. Please "Look, that's a CI failure. Please figure out what else to do in this figure out what else to do in this figure out what else to do in this case." case." case." It could be that CI failed because a It could be that CI failed because a It could be that CI failed because a previous change that we did. It could previous change that we did. It could previous change that we did. It could also be also be also be a flaky CI. It could be an a flaky CI. It could be an a flaky CI. It could be an infrastructure failure, just a simple infrastructure failure, just a simple infrastructure failure, just a simple timeout that just, timeout that just, timeout that just, you know, caused the CI failure. And you know, caused the CI failure. And you know, caused the CI failure. And then then we would just need to retry then then we would just need to retry then then we would just need to retry and do nothing else in that case. and do nothing else in that case. and do nothing else in that case. Now, let's have a look at it how it Now, let's have a look at it how it Now, let's have a look at it how it looks end-to-end. looks end-to-end. looks end-to-end. So, first and foremost, a little bit of So, first and foremost, a little bit of So, first and foremost, a little bit of color coding. I hope you can see this. color coding. I hope you can see this. color coding. I hope you can see this. So, the green bits, that's our So, the green bits, that's our So, the green bits, that's our deterministic part, our go application. deterministic part, our go application. deterministic part, our go application. And the yellow ones, that's the agent. And the yellow ones, that's the agent. And the yellow ones, that's the agent. Um Um Um So, first we start with the So, first we start with the So, first we start with the with the assessment. That's just a with the assessment. That's just a with the assessment. That's just a simple manifest, a simple JSON file simple manifest, a simple JSON file simple manifest, a simple JSON file which contains a list of images and the which contains a list of images and the which contains a list of images and the CVEs that it found for these particular CVEs that it found for these particular CVEs that it found for these particular images. images. images. Um Um Um It will then moves on and downloads the It will then moves on and downloads the It will then moves on and downloads the repository it's supposed to work on, repository it's supposed to work on, repository it's supposed to work on, um clones it into a local directory and um clones it into a local directory and um clones it into a local directory and creates another directory inside it creates another directory inside it creates another directory inside it where it then shoves where it then shoves where it then shoves all the context for the agent inside all the context for the agent inside all the context for the agent inside there. That's a prompt with which has there. That's a prompt with which has there. That's a prompt with which has like 2,000 words or so.

like 2,000 words or so. like 2,000 words or so. Um it's the assessment manifest, um a Um it's the assessment manifest, um a Um it's the assessment manifest, um a couple of other things for like couple of other things for like couple of other things for like communicating with the agent. And then simply we simply invoke the CVE And then simply we simply invoke the CVE remediation agent, which is like the remediation agent, which is like the remediation agent, which is like the first bit. first bit. first bit. Um the first agentic part that we Um the first agentic part that we Um the first agentic part that we invoke, which is supposed to do the the invoke, which is supposed to do the the invoke, which is supposed to do the the smallest effective change set. smallest effective change set. smallest effective change set. And it should only fix the CVEs which And it should only fix the CVEs which And it should only fix the CVEs which are tied to the findings. It shouldn't are tied to the findings. It shouldn't are tied to the findings. It shouldn't just, you know, bump the dependencies to just, you know, bump the dependencies to just, you know, bump the dependencies to the latest and greatest version. That's the latest and greatest version. That's the latest and greatest version. That's just just just that interest introduces unnecessary that interest introduces unnecessary that interest introduces unnecessary risk, which we want to avoid. So just risk, which we want to avoid. So just risk, which we want to avoid. So just bump to the version that actually fixes bump to the version that actually fixes bump to the version that actually fixes the CVE and um nothing more. the CVE and um nothing more. the CVE and um nothing more. And at the very end, the agent then of And at the very end, the agent then of And at the very end, the agent then of course should um just verify its own course should um just verify its own course should um just verify its own work. Should ensure that the Dockerfile work. Should ensure that the Dockerfile work. Should ensure that the Dockerfile is able to build, should um rescan the is able to build, should um rescan the is able to build, should um rescan the image that it has built, and all of that image that it has built, and all of that image that it has built, and all of that stuff. Um so the CVE remediation agent actually Um so the CVE remediation agent actually just modifies files on the file system. just modifies files on the file system. just modifies files on the file system. It doesn't commit, it doesn't push, it It doesn't commit, it doesn't push, it It doesn't commit, it doesn't push, it doesn't create a PR, it doesn't watch doesn't create a PR, it doesn't watch doesn't create a PR, it doesn't watch the CI itself. It just modifies files on the CI itself. It just modifies files on the CI itself. It just modifies files on the file system.

the file system. the file system. And once it's done, it hands back And once it's done, it hands back And once it's done, it hands back control to the controller, to the control to the controller, to the control to the controller, to the deterministic bit, deterministic bit, deterministic bit, which then which then which then quickly vets the changes for, you know, quickly vets the changes for, you know, quickly vets the changes for, you know, some nonsense in there in there. For some nonsense in there in there. For some nonsense in there in there. For instance, like instance, like instance, like um empty files or some binaries that the um empty files or some binaries that the um empty files or some binaries that the agent might create because agents are agent might create because agents are agent might create because agents are really dumb. They mess up, you know, really dumb. They mess up, you know, really dumb. They mess up, you know, some bash commands that are piped some bash commands that are piped some bash commands that are piped together, which then could um create together, which then could um create together, which then could um create empty files. empty files. empty files. Or they just, you know, compile the Or they just, you know, compile the Or they just, you know, compile the application and leave the binary inside application and leave the binary inside application and leave the binary inside the repository, which is nonsense. So we the repository, which is nonsense. So we the repository, which is nonsense. So we got to take care of that. And then we got to take care of that. And then we got to take care of that. And then we commit it, we push it, we create a PR, commit it, we push it, we create a PR, commit it, we push it, we create a PR, um and then we watch CI. um and then we watch CI. um and then we watch CI. And once CI is green, cool, we just send And once CI is green, cool, we just send And once CI is green, cool, we just send over Slack notifications and we're done. over Slack notifications and we're done. over Slack notifications and we're done. Um in case there's a CI failure, we just Um in case there's a CI failure, we just Um in case there's a CI failure, we just move to the next agentic bit, which move to the next agentic bit, which move to the next agentic bit, which um should then just remediate the PR um should then just remediate the PR um should then just remediate the PR failure. failure. failure. To do that, we just give it all the To do that, we just give it all the To do that, we just give it all the context that it needs. So, CI logs, context that it needs. So, CI logs, context that it needs. So, CI logs, workflow metadata that we need, and workflow metadata that we need, and workflow metadata that we need, and everything else that is needed to work everything else that is needed to work everything else that is needed to work on this on this on this on the CI failure.

on the CI failure. on the CI failure. And then we invoke it. And this PR And then we invoke it. And this PR And then we invoke it. And this PR remediation agent just simply should do remediation agent just simply should do remediation agent just simply should do the minimum effort fixes. It kind of the minimum effort fixes. It kind of the minimum effort fixes. It kind of LLMs LLMs kind of tend to just revert LLMs LLMs kind of tend to just revert LLMs LLMs kind of tend to just revert the previous changes that it did. So, we the previous changes that it did. So, we the previous changes that it did. So, we got to tell it to not do this. got to tell it to not do this. got to tell it to not do this. And again, it should just verify its own And again, it should just verify its own And again, it should just verify its own work. work. work. And this then goes on in loop, create And this then goes on in loop, create And this then goes on in loop, create and watch and watch and watch the PR, watch CI, fixes the CI failure, the PR, watch CI, fixes the CI failure, the PR, watch CI, fixes the CI failure, and so on and so forth, until we hit and so on and so forth, until we hit and so on and so forth, until we hit like a like a like a a maximum retry. a maximum retry. a maximum retry. And then we send that over to a to a And then we send that over to a to a And then we send that over to a to a human in case there's like human in case there's like human in case there's like the limit is reached. We then just the limit is reached. We then just the limit is reached. We then just simply tell the tell the human to take a simply tell the tell the human to take a simply tell the tell the human to take a look at this particular issue. look at this particular issue. look at this particular issue. Cuz then that needs manual remediation. There's one more thing that I want to There's one more thing that I want to share here, which is that at the end of share here, which is that at the end of share here, which is that at the end of every agent invocation, we ask the agent every agent invocation, we ask the agent every agent invocation, we ask the agent to do a very short and simple to do a very short and simple to do a very short and simple retrospective. What went well, what went retrospective. What went well, what went retrospective. What went well, what went wrong, what tools are missing, and what wrong, what tools are missing, and what wrong, what tools are missing, and what kind of context would help the next time kind of context would help the next time kind of context would help the next time it would be invoked. it would be invoked. it would be invoked. And this really helps us to understand And this really helps us to understand And this really helps us to understand what the agent is doing and what is what what the agent is doing and what is what what the agent is doing and what is what is missing out.

is missing out. is missing out. Um Um Um Now, the observability for the agent in Now, the observability for the agent in Now, the observability for the agent in bits, that's still like an open issue bits, that's still like an open issue bits, that's still like an open issue and being built by the community at the and being built by the community at the and being built by the community at the moment. That's just how we deal moment. That's just how we deal moment. That's just how we deal dealt with this with this particular dealt with this with this particular dealt with this with this particular issue. issue. issue. Um Um Um So, yeah. That really helps us to So, yeah. That really helps us to So, yeah. That really helps us to understand this at scale because then we understand this at scale because then we understand this at scale because then we can just aggregate all this all that can just aggregate all this all that can just aggregate all this all that information across the PRs, condense it information across the PRs, condense it information across the PRs, condense it down, and then we can see um what we down, and then we can see um what we down, and then we can see um what we need to fix. We usually saw like two need to fix. We usually saw like two need to fix. We usually saw like two kinds of issues. One is an kinds of issues. One is an kinds of issues. One is an infrastructure-related issue. It could infrastructure-related issue. It could infrastructure-related issue. It could be um be um be um network failures, or that the agent network failures, or that the agent network failures, or that the agent didn't have enough permissions in order didn't have enough permissions in order didn't have enough permissions in order to like clone a repository, um or it to like clone a repository, um or it to like clone a repository, um or it could be some complexity issue related could be some complexity issue related could be some complexity issue related with the repository. Some repositories with the repository. Some repositories with the repository. Some repositories are just are just are just hard to deal with and hard to reason hard to deal with and hard to reason hard to deal with and hard to reason about without having about without having about without having the the context that the agent needs. the the context that the agent needs. the the context that the agent needs. Then we just need to either modify the Then we just need to either modify the Then we just need to either modify the system prompt of the agent or we then system prompt of the agent or we then system prompt of the agent or we then have repository-specific instructions um have repository-specific instructions um have repository-specific instructions um that we then just feed that we then just feed that we then just feed um to our agent to work on this issue. I brought a screenshot here with me. Um I brought a screenshot here with me. Um I had to redact a lot of stuff cuz it's I had to redact a lot of stuff cuz it's I had to redact a lot of stuff cuz it's an actual like production kind of an actual like production kind of an actual like production kind of screenshot and screenshot and screenshot and you know, in finance you don't want to you know, in finance you don't want to you know, in finance you don't want to share this kind of information, but um share this kind of information, but um share this kind of information, but um yeah, that's what I what I got out.

yeah, that's what I what I got out. yeah, that's what I what I got out. So, here at the top where the arrow is, So, here at the top where the arrow is, So, here at the top where the arrow is, that's a very simple bit. We just bump that's a very simple bit. We just bump that's a very simple bit. We just bump the the base image that we're using from the the base image that we're using from the the base image that we're using from the Go line that simply a dependabot can the Go line that simply a dependabot can the Go line that simply a dependabot can do. That's boring. At the bottom we can do. That's boring. At the bottom we can do. That's boring. At the bottom we can see that the agent uh updated libcrypto3 see that the agent uh updated libcrypto3 see that the agent uh updated libcrypto3 and libssl3. and libssl3. and libssl3. So, um So, um So, um yeah. yeah. yeah. I also have like a short diff um that I I also have like a short diff um that I I also have like a short diff um that I want to quickly showcase what the agent want to quickly showcase what the agent want to quickly showcase what the agent did. did. did. Um you can see here on the left-hand Um you can see here on the left-hand Um you can see here on the left-hand side that it side that it side that it that these um bottom two that these um bottom two that these um bottom two Can I just go there? Here. Can I just go there? Here. Can I just go there? Here. That we have these two packages here That we have these two packages here That we have these two packages here that it actually removed in the process that it actually removed in the process that it actually removed in the process because these packages um have been because these packages um have been because these packages um have been pinned previously. It then just removed pinned previously. It then just removed pinned previously. It then just removed those because they have been fixed by those because they have been fixed by those because they have been fixed by the actual base image. So, the actual base image. So, the actual base image. So, to be honest, if I were tasked with this to be honest, if I were tasked with this to be honest, if I were tasked with this kind of task, I probably would have kind of task, I probably would have kind of task, I probably would have missed that. So, I'm like glad that the missed that. So, I'm like glad that the missed that. So, I'm like glad that the agent um fixed that and is a good good agent um fixed that and is a good good agent um fixed that and is a good good engineer here. engineer here. engineer here. So, and that's it. So, we started with a So, and that's it. So, we started with a So, and that's it. So, we started with a bunch of CVEs. We scanned artifacts, um bunch of CVEs. We scanned artifacts, um bunch of CVEs. We scanned artifacts, um remediated remediated this the CVEs and remediated remediated this the CVEs and remediated remediated this the CVEs and the findings, and now we have a proper the findings, and now we have a proper the findings, and now we have a proper working um working um working um um PR that then just needs to be um PR that then just needs to be um PR that then just needs to be reviewed by a human and approved and reviewed by a human and approved and reviewed by a human and approved and merged.

merged. merged. So, to make that work, um So, to make that work, um So, to make that work, um we gave Patch Pilot a couple of things. we gave Patch Pilot a couple of things. we gave Patch Pilot a couple of things. We gave it GitHub access, um read and We gave it GitHub access, um read and We gave it GitHub access, um read and write access to clone the repository, to write access to clone the repository, to write access to clone the repository, to commit, and to push changes, to open up commit, and to push changes, to open up commit, and to push changes, to open up PR, to download the CI logs, and trigger PR, to download the CI logs, and trigger PR, to download the CI logs, and trigger CI. We also gave it OCR registry CI. We also gave it OCR registry CI. We also gave it OCR registry credentials in order to list images in credentials in order to list images in credentials in order to list images in the in the registry to figure out, you the in the registry to figure out, you the in the registry to figure out, you know, what images are available, which know, what images are available, which know, what images are available, which are latest. Um we gave it tools like a are latest. Um we gave it tools like a are latest. Um we gave it tools like a go run time, python run time, a bunch of go run time, python run time, a bunch of go run time, python run time, a bunch of linters, static libs, linters, static libs, linters, static libs, a bash shell, network access, and all of a bash shell, network access, and all of a bash shell, network access, and all of that. that. that. Um Um Um So, this is what we gave the whole So, this is what we gave the whole So, this is what we gave the whole application. But again, we had like two application. But again, we had like two application. But again, we had like two layers, the deterministic part and the layers, the deterministic part and the layers, the deterministic part and the agentic part. And we also applied the agentic part. And we also applied the agentic part. And we also applied the split for the credentials also for the split for the credentials also for the split for the credentials also for the two different layers. two different layers. two different layers. So, So, So, here's the thing about the capability here's the thing about the capability here's the thing about the capability list. The dangerous ones, the get up list. The dangerous ones, the get up list. The dangerous ones, the get up right access, um and trigger UCI is right access, um and trigger UCI is right access, um and trigger UCI is something that we did not give the something that we did not give the something that we did not give the agent. Instead, we pushed um that agent. Instead, we pushed um that agent. Instead, we pushed um that functionality out to the deterministic functionality out to the deterministic functionality out to the deterministic part cuz that's the thing that we can part cuz that's the thing that we can part cuz that's the thing that we can reason about and we can rely on that um reason about and we can rely on that um reason about and we can rely on that um you know, it just does these kind of you know, it just does these kind of you know, it just does these kind of kind of actions and we do not give um kind of actions and we do not give um kind of actions and we do not give um the agent these kinds of credentials cuz the agent these kinds of credentials cuz the agent these kinds of credentials cuz that then fundamentally fundamentally that then fundamentally fundamentally that then fundamentally fundamentally limits the blast radius of limits the blast radius of limits the blast radius of when in case the agent gets um prompt when in case the agent gets um prompt when in case the agent gets um prompt injected prompt injected.

injected prompt injected. injected prompt injected. So, that boundary really matters for the So, that boundary really matters for the So, that boundary really matters for the prompt injection case because you prompt injection case because you prompt injection case because you probably didn't see it in the screenshot probably didn't see it in the screenshot probably didn't see it in the screenshot earlier, but there was like 70,000 lines earlier, but there was like 70,000 lines earlier, but there was like 70,000 lines of code that were changed in that small of code that were changed in that small of code that were changed in that small PR. PR. PR. Um that's really like a lot of changes Um that's really like a lot of changes Um that's really like a lot of changes that come in just by bumping a couple of that come in just by bumping a couple of that come in just by bumping a couple of dependencies. dependencies. dependencies. Um and the attack surface is really Um and the attack surface is really Um and the attack surface is really really really wide. Um really really wide. Um really really wide. Um So, what we did to mitigate that, I So, what we did to mitigate that, I So, what we did to mitigate that, I guess like prompt injection itself isn't guess like prompt injection itself isn't guess like prompt injection itself isn't solved and we cannot really solve it. solved and we cannot really solve it. solved and we cannot really solve it. All we can do is just to limit the blast All we can do is just to limit the blast All we can do is just to limit the blast radius in case that happens. radius in case that happens. radius in case that happens. Um what we did is to do a little bit of Um what we did is to do a little bit of Um what we did is to do a little bit of prompt steering because we know what prompt steering because we know what prompt steering because we know what kind of directories or files contain kind of directories or files contain kind of directories or files contain untrusted um un untrusted um untrusted um un untrusted um untrusted um un untrusted um information, untrusted context. information, untrusted context. information, untrusted context. We just tell the agent, look, We just tell the agent, look, We just tell the agent, look, the vendor directory, just don't trust the vendor directory, just don't trust the vendor directory, just don't trust that. Or the CI logs which live in that that. Or the CI logs which live in that that. Or the CI logs which live in that file or in that subdirectory, you know, file or in that subdirectory, you know, file or in that subdirectory, you know, be sure that you don't, you know, be an be sure that you don't, you know, be an be sure that you don't, you know, be an idiot. idiot. idiot. Um there's another thing that we did um, Um there's another thing that we did um, Um there's another thing that we did um, which worked quite well, which is that which worked quite well, which is that which worked quite well, which is that we, um, we, um, we, um, we essentially implemented an end-to-end we essentially implemented an end-to-end we essentially implemented an end-to-end test where we created a repository and test where we created a repository and test where we created a repository and sent patch patch pilot added to just sent patch patch pilot added to just sent patch patch pilot added to just work on it.

work on it. work on it. Um, Um, Um, I guess that's what people call eval's I guess that's what people call eval's I guess that's what people call eval's today and then we just, um, ensure that today and then we just, um, ensure that today and then we just, um, ensure that it isn't, um, it isn't, um, it isn't, um, prompt injected. So, we have like a prompt injected. So, we have like a prompt injected. So, we have like a deprecated function in there in the in deprecated function in there in the in deprecated function in there in the in our crafted repository which tries to our crafted repository which tries to our crafted repository which tries to recruit the agent to do some malicious recruit the agent to do some malicious recruit the agent to do some malicious stuff or we have a migration guide where stuff or we have a migration guide where stuff or we have a migration guide where it deprecated function, um, it deprecated function, um, it deprecated function, um, points at and all of that. So, we try to points at and all of that. So, we try to points at and all of that. So, we try to like, you know, remediate these kinds like, you know, remediate these kinds like, you know, remediate these kinds of, um, issues that we know about. of, um, issues that we know about. of, um, issues that we know about. But still there um, unknown, um, But still there um, unknown, um, But still there um, unknown, um, injection vectors which we aren't aware injection vectors which we aren't aware injection vectors which we aren't aware of yet. Um, so that's why, you know, we of yet. Um, so that's why, you know, we of yet. Um, so that's why, you know, we still have to pray a little bit. still have to pray a little bit. still have to pray a little bit. But at least we don't like build the But at least we don't like build the But at least we don't like build the whole system on on hope. Now, that's the bit that kept me awake Now, that's the bit that kept me awake at night. Um, Now, sandboxes look great on a slide. Now, sandboxes look great on a slide. You just draw a box, put the agent in it You just draw a box, put the agent in it You just draw a box, put the agent in it and you feel secure, right? So, and you feel secure, right? So, and you feel secure, right? So, the problem is that at some point the the problem is that at some point the the problem is that at some point the agent really wants to, um, agent really wants to, um, agent really wants to, um, verify its own work. When it works with verify its own work. When it works with verify its own work. When it works with Docker files, it wants to build a Docker Docker files, it wants to build a Docker Docker files, it wants to build a Docker container. It might also want to run a container. It might also want to run a container. It might also want to run a Docker container in order to figure out Docker container in order to figure out Docker container in order to figure out what package versions are available and what package versions are available and what package versions are available and so on.

so on. so on. So, naturally you give it that Docker So, naturally you give it that Docker So, naturally you give it that Docker socket. socket. socket. At that point, it's more or less game At that point, it's more or less game At that point, it's more or less game over for you, um, because the agent can over for you, um, because the agent can over for you, um, because the agent can then simply just spawn a privileged then simply just spawn a privileged then simply just spawn a privileged container, escape out of it and then, container, escape out of it and then, container, escape out of it and then, you know, read environment variables of you know, read environment variables of you know, read environment variables of other processes, read the memory of other processes, read the memory of other processes, read the memory of other processes, can plant SSH keys, other processes, can plant SSH keys, other processes, can plant SSH keys, it's game over for you essentially at it's game over for you essentially at it's game over for you essentially at this point. We run it like that in production at We run it like that in production at some point. Um, it didn't feel good. We some point. Um, it didn't feel good. We some point. Um, it didn't feel good. We moved off of that, um, moved off of that, um, moved off of that, um, and we evaluated all the other obvious and we evaluated all the other obvious and we evaluated all the other obvious options in like the Linux sphere Linux options in like the Linux sphere Linux options in like the Linux sphere Linux bubble when it comes to like sandboxing. bubble when it comes to like sandboxing. bubble when it comes to like sandboxing. There's a lot of technologies out there There's a lot of technologies out there There's a lot of technologies out there like landlock, bubble wrap, seccomp, u like landlock, bubble wrap, seccomp, u like landlock, bubble wrap, seccomp, u notify, fence, and a lot of functions notify, fence, and a lot of functions notify, fence, and a lot of functions that we have for unprivileged Docker that we have for unprivileged Docker that we have for unprivileged Docker builds, um Kaniko, uh buildkit, and and builds, um Kaniko, uh buildkit, and and builds, um Kaniko, uh buildkit, and and what else. what else. what else. But, they don't really compose well with But, they don't really compose well with But, they don't really compose well with containers, and none of them really can containers, and none of them really can containers, and none of them really can contain a Docker socket or a Docker um contain a Docker socket or a Docker um contain a Docker socket or a Docker um demon that runs on a host. So, let me share a design that we came So, let me share a design that we came up with, which is still like in its in up with, which is still like in its in up with, which is still like in its in its infancy. Um its infancy. Um its infancy. Um It's the same pattern. You just draw a It's the same pattern. You just draw a It's the same pattern. You just draw a box, but instead of calling it a box, but instead of calling it a box, but instead of calling it a sandbox, it's just a micro VM. In our sandbox, it's just a micro VM. In our sandbox, it's just a micro VM. In our case, we're using um using case, we're using um using case, we're using um using [clears throat] Firecracker to have a [clears throat] Firecracker to have a [clears throat] Firecracker to have a like a proper isolation mechanism.

like a proper isolation mechanism. like a proper isolation mechanism. And then we put the agent in, we put the And then we put the agent in, we put the And then we put the agent in, we put the Docker socket in, and then the Docker Docker socket in, and then the Docker Docker socket in, and then the Docker socket is powered by its own kernel in socket is powered by its own kernel in socket is powered by its own kernel in this case. this case. this case. Um which is good because that really Um which is good because that really Um which is good because that really solves the issue with solves the issue with solves the issue with with the um Docker daemon or the agent with the um Docker daemon or the agent with the um Docker daemon or the agent trying to escape the sandbox. In this trying to escape the sandbox. In this trying to escape the sandbox. In this case, everything is just contained case, everything is just contained case, everything is just contained within that micro VM, which is good. within that micro VM, which is good. within that micro VM, which is good. There's another thing that is um There's another thing that is um There's another thing that is um worth mentioning in this case. Again, we worth mentioning in this case. Again, we worth mentioning in this case. Again, we have this two-tier architecture, this have this two-tier architecture, this have this two-tier architecture, this two layers, the agentic bit and the two layers, the agentic bit and the two layers, the agentic bit and the deterministic bit. deterministic bit. deterministic bit. And we also want to apply network And we also want to apply network And we also want to apply network policies to these two worlds separately. policies to these two worlds separately. policies to these two worlds separately. For the deterministic bit, we just know For the deterministic bit, we just know For the deterministic bit, we just know what kind of um what kind of um what kind of um um network access it requires in order um network access it requires in order um network access it requires in order to function properly. It needs GitHub to function properly. It needs GitHub to function properly. It needs GitHub access and what else. For the agent, we access and what else. For the agent, we access and what else. For the agent, we don't really know in advance. It kind of don't really know in advance. It kind of don't really know in advance. It kind of depends off on the kind of repository or depends off on the kind of repository or depends off on the kind of repository or language um it works on. When it uses language um it works on. When it uses language um it works on. When it uses Java, it needs like a completely Java, it needs like a completely Java, it needs like a completely different ecosystem it works with different ecosystem it works with different ecosystem it works with compared to Python or Go. compared to Python or Go. compared to Python or Go. Um so, that's why we really want to need Um so, that's why we really want to need Um so, that's why we really want to need to apply different network policies. Um to apply different network policies. Um to apply different network policies. Um so, that's what you can do. You can just so, that's what you can do. You can just so, that's what you can do. You can just set up a DNS and TCP forwarder inside set up a DNS and TCP forwarder inside set up a DNS and TCP forwarder inside that micro VM.

that micro VM. that micro VM. But, other than that, just, you know, But, other than that, just, you know, But, other than that, just, you know, cut it completely off from the host cut it completely off from the host cut it completely off from the host system. And then everything has to flow system. And then everything has to flow system. And then everything has to flow through this um through this Vsock, through this um through this Vsock, through this um through this Vsock, which then um which then um which then um hands all the network packets to a host hands all the network packets to a host hands all the network packets to a host process which then applies the network process which then applies the network process which then applies the network policies to policies to policies to um to all the outgoing packets. And then um to all the outgoing packets. And then um to all the outgoing packets. And then you can apply, you know, rules based on you can apply, you know, rules based on you can apply, you know, rules based on host names, based on target ports, side host names, based on target ports, side host names, based on target ports, side ranges, whatever you like. ranges, whatever you like. ranges, whatever you like. If you really want to go wild, you can If you really want to go wild, you can If you really want to go wild, you can just, you know, set up a custom CA, just, you know, set up a custom CA, just, you know, set up a custom CA, plant it into the micro VM, and then do plant it into the micro VM, and then do plant it into the micro VM, and then do all sorts of like TLS man-in-the-middle all sorts of like TLS man-in-the-middle all sorts of like TLS man-in-the-middle control. control. control. Which is really hard to do, especially Which is really hard to do, especially Which is really hard to do, especially when it comes to the Docker socket, but when it comes to the Docker socket, but when it comes to the Docker socket, but um um um we're not going to dive into this one we're not going to dive into this one we're not going to dive into this one yet. yet. yet. So, my take on agent sandboxing is this, So, my take on agent sandboxing is this, So, my take on agent sandboxing is this, that that that the existing agent that we have today the existing agent that we have today the existing agent that we have today with Codex and Cloud, they come with with Codex and Cloud, they come with with Codex and Cloud, they come with their own sandbox, but in my opinion, their own sandbox, but in my opinion, their own sandbox, but in my opinion, it's worthless, especially when you give it's worthless, especially when you give it's worthless, especially when you give it um a it um a it um a a Docker socket access. It's just really a Docker socket access. It's just really a Docker socket access. It's just really yeah, it's not going to be able to yeah, it's not going to be able to yeah, it's not going to be able to contain that.

contain that. contain that. Other agents like Open Code or Pi, they Other agents like Open Code or Pi, they Other agents like Open Code or Pi, they just don't have a sandbox because it's just don't have a sandbox because it's just don't have a sandbox because it's not implemented not implemented not implemented um or it's just not there by design. um or it's just not there by design. um or it's just not there by design. So, my take is that you really have to So, my take is that you really have to So, my take is that you really have to create a environment for an agent where create a environment for an agent where create a environment for an agent where you can just put it in and give it you can just put it in and give it you can just put it in and give it everything that it needs, um so it has everything that it needs, um so it has everything that it needs, um so it has like all the freedom to work on a like all the freedom to work on a like all the freedom to work on a particular issue that it's supposed to particular issue that it's supposed to particular issue that it's supposed to do. do. do. Um but there's still a gap. So, the diagram but there's still a gap. So, the diagram that I've just shown you that I've just shown you that I've just shown you isn't something unique. It's just a VM isn't something unique. It's just a VM isn't something unique. It's just a VM with a little bit of plumbing in there. with a little bit of plumbing in there. with a little bit of plumbing in there. You can just wipe code, it's not not a You can just wipe code, it's not not a You can just wipe code, it's not not a big deal, honestly. big deal, honestly. big deal, honestly. Um you can even go down sta- downstairs Um you can even go down sta- downstairs Um you can even go down sta- downstairs to the vendor booths and just, you know, to the vendor booths and just, you know, to the vendor booths and just, you know, talk to the vendors there. You have like talk to the vendors there. You have like talk to the vendors there. You have like a lot of sandbox as a service that you a lot of sandbox as a service that you a lot of sandbox as a service that you can easily consume. can easily consume. can easily consume. Some of which really lack in Some of which really lack in Some of which really lack in functionality, especially in that functionality, especially in that functionality, especially in that regards to like containing a Docker regards to like containing a Docker regards to like containing a Docker socket or having a proper network access socket or having a proper network access socket or having a proper network access controls. That's still lacking, but it controls. That's still lacking, but it controls. That's still lacking, but it differs from vendor to vendor. differs from vendor to vendor. differs from vendor to vendor. I also recently saw the um micro sandbox I also recently saw the um micro sandbox I also recently saw the um micro sandbox project, um project, um project, um which has now run for, I think, like which has now run for, I think, like which has now run for, I think, like three or four months or so, maybe even three or four months or so, maybe even three or four months or so, maybe even longer, um which really is would be my longer, um which really is would be my longer, um which really is would be my choice if I would build patch pilot choice if I would build patch pilot choice if I would build patch pilot today again because it comes with all today again because it comes with all today again because it comes with all the batteries included, network access the batteries included, network access the batteries included, network access controls, um all of that. So, keep that controls, um all of that. So, keep that controls, um all of that. So, keep that in mind if you really want to build in mind if you really want to build in mind if you really want to build this. Um this. Um this. Um it's an open source project. It has It it's an open source project. It has It it's an open source project. It has It has a community. It's currently funded has a community. It's currently funded has a community. It's currently funded by YC. Um so, we really don't really by YC. Um so, we really don't really by YC. Um so, we really don't really know know know how it how it goes in the next couple of how it how it goes in the next couple of how it how it goes in the next couple of months. We're going to see um if they're months. We're going to see um if they're months. We're going to see um if they're able to gather enough traction to to

able to gather enough traction to to able to gather enough traction to to keep the open source project afloat. keep the open source project afloat. keep the open source project afloat. Let's see. Let's see. Let's see. So, the gap isn't the tool doesn't So, the gap isn't the tool doesn't So, the gap isn't the tool doesn't exist. exist. exist. All the tools do exist, but most of them All the tools do exist, but most of them All the tools do exist, but most of them are still in the beta phase. are still in the beta phase. are still in the beta phase. And there's still a big gap to, you And there's still a big gap to, you And there's still a big gap to, you know, getting actual enterprise traction know, getting actual enterprise traction know, getting actual enterprise traction and getting all the features, all the and getting all the features, all the and getting all the features, all the bells and whistles that you need in bells and whistles that you need in bells and whistles that you need in order to deploy it into a proper order to deploy it into a proper order to deploy it into a proper enterprise environment. And I think enterprise environment. And I think enterprise environment. And I think that's something that we need to work that's something that we need to work that's something that we need to work on. on. on. Sure, now we got a sandbox. Cool. That's Sure, now we got a sandbox. Cool. That's Sure, now we got a sandbox. Cool. That's nice, but we still need like a lot of nice, but we still need like a lot of nice, but we still need like a lot of orchestration on top of it, which is orchestration on top of it, which is orchestration on top of it, which is something that you can something that you can something that you can take from a vendor. There are also like take from a vendor. There are also like take from a vendor. There are also like a couple of open source project at the a couple of open source project at the a couple of open source project at the moment which are working on this. moment which are working on this. moment which are working on this. There's the Kubernetes um agent sandbox There's the Kubernetes um agent sandbox There's the Kubernetes um agent sandbox uh special interest group that's working uh special interest group that's working uh special interest group that's working on this. There's um open sandbox which on this. There's um open sandbox which on this. There's um open sandbox which also working on this to integrate these also working on this to integrate these also working on this to integrate these kind of things. So, kind of things. So, kind of things. So, um keep that in mind if you really want um keep that in mind if you really want um keep that in mind if you really want to, you know, contain an agent in a to, you know, contain an agent in a to, you know, contain an agent in a production environment. It's not there production environment. It's not there production environment. It's not there yet. It's beta. It's very early in the yet. It's beta. It's very early in the yet. It's beta. It's very early in the in the phase. in the phase. in the phase. We're going to see how it goes. All right. If you take one thing from All right. If you take one thing from this, um the blast radius of an agent is this, um the blast radius of an agent is this, um the blast radius of an agent is an architecture decision.

an architecture decision. an architecture decision. We didn't um We didn't um We didn't um give the agent the credential that it give the agent the credential that it give the agent the credential that it needs um in order to, you know, trigger needs um in order to, you know, trigger needs um in order to, you know, trigger CI or CI or CI or push to GitHub or open a PR. That's just push to GitHub or open a PR. That's just push to GitHub or open a PR. That's just simply not needed. You can just push simply not needed. You can just push simply not needed. You can just push that functionality in functionality into that functionality in functionality into that functionality in functionality into a deterministic layer. a deterministic layer. a deterministic layer. Um that kind of really limits the blast Um that kind of really limits the blast Um that kind of really limits the blast radius of an agent. radius of an agent. radius of an agent. So, that choice, what's that what's So, that choice, what's that what's So, that choice, what's that what's deterministic and what's agentic, that deterministic and what's agentic, that deterministic and what's agentic, that really is, you know, your security model really is, you know, your security model really is, you know, your security model in this case. All right. All right. Um Um Um that was it for me. I just have like one that was it for me. I just have like one that was it for me. I just have like one last word, which is just an invitation last word, which is just an invitation last word, which is just an invitation to the community just to talk about the to the community just to talk about the to the community just to talk about the things that I just mentioned. Um I think things that I just mentioned. Um I think things that I just mentioned. Um I think tomorrow there's a a dedicated uh tomorrow there's a a dedicated uh tomorrow there's a a dedicated uh sandbox panel where a lot of talks are sandbox panel where a lot of talks are sandbox panel where a lot of talks are just around the sandbox thingy. just around the sandbox thingy. just around the sandbox thingy. And surely I want to know from you guys And surely I want to know from you guys And surely I want to know from you guys how you run agents in production just to how you run agents in production just to how you run agents in production just to learn and to have a discussion about learn and to have a discussion about learn and to have a discussion about that in order to, you know, bring the that in order to, you know, bring the that in order to, you know, bring the community on and just just figure it out community on and just just figure it out community on and just just figure it out um all together. um all together. um all together. Thank you. Thank you. Thank you. >> [applause]