Agents, codebases, and teams — Aditya Khandelwal, Amazon AGI Lab

>> Today, I'm going to be talking about >> Today, I'm going to be talking about agents code bases and teams. agents code bases and teams. agents code bases and teams. Essentially, Essentially, Essentially, how do you get your team to actually how do you get your team to actually how do you get your team to actually ship together with agents? And like, I ship together with agents? And like, I ship together with agents? And like, I think for the longest time, like the one think for the longest time, like the one think for the longest time, like the one thing that's bugged me is like there's thing that's bugged me is like there's thing that's bugged me is like there's so much content about, you know, how do so much content about, you know, how do so much content about, you know, how do you set up your own code base to like you set up your own code base to like you set up your own code base to like work well with agents, you know, like work well with agents, you know, like work well with agents, you know, like what skills do you add? You know, this what skills do you add? You know, this what skills do you add? You know, this skill's better, that setup's better. But skill's better, that setup's better. But skill's better, that setup's better. But it all seems to break the moment you it all seems to break the moment you it all seems to break the moment you like actually try to use it with your like actually try to use it with your like actually try to use it with your team in your actual production setup. team in your actual production setup. team in your actual production setup. And like for individual repos, it kind And like for individual repos, it kind And like for individual repos, it kind of makes sense. But the moment you of makes sense. But the moment you of makes sense. But the moment you actually try to use it with your own actually try to use it with your own actually try to use it with your own like team setup, it tends to break. like team setup, it tends to break. like team setup, it tends to break. And like I think over the past few And like I think over the past few And like I think over the past few months, I've figured out how to make it months, I've figured out how to make it months, I've figured out how to make it work work work with a team of folks. So like I was with a team of folks. So like I was with a team of folks. So like I was leading a team of like 10 people over leading a team of like 10 people over leading a team of like 10 people over the last few months, and I think we the last few months, and I think we the last few months, and I think we found like a good solution, and I want found like a good solution, and I want found like a good solution, and I want to share that with you guys. to share that with you guys. to share that with you guys. But before we get into that, I just want But before we get into that, I just want But before we get into that, I just want to recap like, you know, what's been the to recap like, you know, what's been the to recap like, you know, what's been the journey that we've been on. journey that we've been on. journey that we've been on. Right? So we started, you know, code Right? So we started, you know, code Right? So we started, you know, code coding agents took off, and a few people coding agents took off, and a few people coding agents took off, and a few people got really, really good leverage, right? got really, really good leverage, right? got really, really good leverage, right? And I think all of us were kind of And I think all of us were kind of And I think all of us were kind of asking like, is this AGI? You know, like asking like, is this AGI? You know, like asking like, is this AGI? You know, like did we achieve it?

did we achieve it? did we achieve it? Then, you know, companies took that and Then, you know, companies took that and Then, you know, companies took that and said, "Well, if one person can do so said, "Well, if one person can do so said, "Well, if one person can do so well, let's just get everyone and let's well, let's just get everyone and let's well, let's just get everyone and let's mandate it." And like token max, right? mandate it." And like token max, right? mandate it." And like token max, right? And that was clearly a And that was clearly a And that was clearly a a galaxy brain moment, right? a galaxy brain moment, right? a galaxy brain moment, right? And then the inevitable happened, you And then the inevitable happened, you And then the inevitable happened, you know, AI slop shipped, and there's a know, AI slop shipped, and there's a know, AI slop shipped, and there's a bunch of sev twos. I'm not going to name bunch of sev twos. I'm not going to name bunch of sev twos. I'm not going to name which companies, but you know, which companies, but you know, which companies, but you know, essentially you saw people kind of essentially you saw people kind of essentially you saw people kind of retracting. They said, retracting. They said, retracting. They said, "I don't think "I don't think "I don't think I don't think this is the best option I don't think this is the best option I don't think this is the best option here." And eventually model prices you here." And eventually model prices you here." And eventually model prices you know, climbed. We saw, you know, people know, climbed. We saw, you know, people know, climbed. We saw, you know, people figured out that tokens have to be paid figured out that tokens have to be paid figured out that tokens have to be paid for. Like you just can't token max your for. Like you just can't token max your for. Like you just can't token max your way through life. And budgets got bolted way through life. And budgets got bolted way through life. And budgets got bolted on. And you know, essentially money is on. And you know, essentially money is on. And you know, essentially money is being lit on fire, and the money has to being lit on fire, and the money has to being lit on fire, and the money has to come from somewhere. come from somewhere. come from somewhere. So, So, So, given this journey, I want to actually given this journey, I want to actually given this journey, I want to actually like this is like this is like this is the enterprise journey, right? And what the enterprise journey, right? And what the enterprise journey, right? And what does that do for a single developer? does that do for a single developer? does that do for a single developer? And I think this is an important framing And I think this is an important framing And I think this is an important framing cuz it really talks about cuz it really talks about cuz it really talks about people as a part of a team, right? And I people as a part of a team, right? And I people as a part of a team, right? And I want to look at it from like two axes. want to look at it from like two axes. want to look at it from like two axes. So, there is the fear axis where people So, there is the fear axis where people So, there is the fear axis where people lie on the spectrum, right? They're lie on the spectrum, right? They're lie on the spectrum, right? They're coming It's like Is it coming from my coming It's like Is it coming from my coming It's like Is it coming from my job? Like am I going to be out of a job?

job? Like am I going to be out of a job? job? Like am I going to be out of a job? Or is it like a really handy tool? And Or is it like a really handy tool? And Or is it like a really handy tool? And they're really, you know, they're not they're really, you know, they're not they're really, you know, they're not that fearful. Versus like the confidence that fearful. Versus like the confidence that fearful. Versus like the confidence they have in how much they're executing they have in how much they're executing they have in how much they're executing it. So, they can either use it a lot or it. So, they can either use it a lot or it. So, they can either use it a lot or they can use it, you know, not that much they can use it, you know, not that much they can use it, you know, not that much because they don't really know how to because they don't really know how to because they don't really know how to use it that well. use it that well. use it that well. Now, when we started, like people said, Now, when we started, like people said, Now, when we started, like people said, "Oh, you know, what is this? Is this the "Oh, you know, what is this? Is this the "Oh, you know, what is this? Is this the end? Like am I needed?" And fear was end? Like am I needed?" And fear was end? Like am I needed?" And fear was pretty high. Utilization was pretty low pretty high. Utilization was pretty low pretty high. Utilization was pretty low cuz people didn't really know how to use cuz people didn't really know how to use cuz people didn't really know how to use it. it. it. And then when a few people got outsized And then when a few people got outsized And then when a few people got outsized leverage, you saw, you know, early leverage, you saw, you know, early leverage, you saw, you know, early adopters adopters adopters basically people saw them and people basically people saw them and people basically people saw them and people said, "Okay, well, it looks like I'm said, "Okay, well, it looks like I'm said, "Okay, well, it looks like I'm still kind of needed if I figure out how still kind of needed if I figure out how still kind of needed if I figure out how to use this thing. So, let me actually to use this thing. So, let me actually to use this thing. So, let me actually try using it, right?" try using it, right?" try using it, right?" And then we saw, you know, mandates and And then we saw, you know, mandates and And then we saw, you know, mandates and token maxing, and people kind of got a token maxing, and people kind of got a token maxing, and people kind of got a little skeptical. Like confidence stayed little skeptical. Like confidence stayed little skeptical. Like confidence stayed the same, but people tried to use it a the same, but people tried to use it a the same, but people tried to use it a lot more, right? lot more, right? lot more, right? And then we realized, uh you know, And then we realized, uh you know, And then we realized, uh you know, there's a bunch of slop shipping, there's a bunch of slop shipping, there's a bunch of slop shipping, there's self-tours, and there's self-tours, and there's self-tours, and I, you know, it's like I'm not really I, you know, it's like I'm not really I, you know, it's like I'm not really that scared cuz, you know, it just slip that scared cuz, you know, it just slip that scared cuz, you know, it just slip it just ships slop. Like I'm going to it just ships slop. Like I'm going to it just ships slop. Like I'm going to still be needed. And they don't even still be needed. And they don't even still be needed. And they don't even know how to use it that well cuz like know how to use it that well cuz like know how to use it that well cuz like now the confidence is greater, right?

now the confidence is greater, right? now the confidence is greater, right? And so you got to figure out how to get And so you got to figure out how to get And so you got to figure out how to get people from all the way wherever they people from all the way wherever they people from all the way wherever they are on the spectrum to where they're not are on the spectrum to where they're not are on the spectrum to where they're not fearful and they're actually using it a fearful and they're actually using it a fearful and they're actually using it a whole lot more. whole lot more. whole lot more. And this is the framing that I want to And this is the framing that I want to And this is the framing that I want to like I want everyone to keep in mind as like I want everyone to keep in mind as like I want everyone to keep in mind as they're actually trying to get a team to they're actually trying to get a team to they're actually trying to get a team to adopt good like AI usage and good AI adopt good like AI usage and good AI adopt good like AI usage and good AI patterns, right? patterns, right? patterns, right? And so, the The is, you know, what does And so, the The is, you know, what does And so, the The is, you know, what does it take? it take? it take? Step one, create a Cloud MD. Step one, create a Cloud MD. Step one, create a Cloud MD. Step two, add some skills. Step two, add some skills. Step two, add some skills. Is that it? Is that it? Is that it? Do we like solve it? Do we like solve it? Do we like solve it? You know, I think we all know like you You know, I think we all know like you You know, I think we all know like you guys are here because like clearly guys are here because like clearly guys are here because like clearly life's not that simple and like stuff's life's not that simple and like stuff's life's not that simple and like stuff's messy, right? messy, right? messy, right? And I think a few people might ask, you And I think a few people might ask, you And I think a few people might ask, you know, why doesn't this work? Like isn't know, why doesn't this work? Like isn't know, why doesn't this work? Like isn't that what like everyone does? that what like everyone does? that what like everyone does? And I want to like just talk about a few And I want to like just talk about a few And I want to like just talk about a few things you might see that actually things you might see that actually things you might see that actually indicate that yeah, like this isn't indicate that yeah, like this isn't indicate that yeah, like this isn't working. working. working. So, the first thing is if you're So, the first thing is if you're So, the first thing is if you're babysitting your agents, babysitting your agents, babysitting your agents, it's not the right setup, right? And you it's not the right setup, right? And you it's not the right setup, right? And you got to realize that. If you're seeing got to realize that. If you're seeing got to realize that. If you're seeing people in your team babysitting their people in your team babysitting their people in your team babysitting their agents, something's wrong.

agents, something's wrong. agents, something's wrong. Is, you know, one of the things that I Is, you know, one of the things that I Is, you know, one of the things that I heard a lot was is, you know, insert heard a lot was is, you know, insert heard a lot was is, you know, insert whatever latest model there is being whatever latest model there is being whatever latest model there is being really dumb today. really dumb today. really dumb today. The model didn't change, right? The The model didn't change, right? The The model didn't change, right? The hardness may have changed underneath. hardness may have changed underneath. hardness may have changed underneath. But if it's really like that's that's But if it's really like that's that's But if it's really like that's that's acceptable to like small changes in the acceptable to like small changes in the acceptable to like small changes in the hardness, clearly your own code base hardness, clearly your own code base hardness, clearly your own code base isn't set up well. It's silently burning context and money. It's silently burning context and money. Like you don't realize it, you know, you Like you don't realize it, you know, you Like you don't realize it, you know, you go you blow through like 500k context, go you blow through like 500k context, go you blow through like 500k context, you might go to like 750k million and you might go to like 750k million and you might go to like 750k million and hit auto compact even though you're not hit auto compact even though you're not hit auto compact even though you're not doing like a really complicated task. doing like a really complicated task. doing like a really complicated task. Clearly something's wrong. Clearly something's wrong. Clearly something's wrong. You know, if you have long ass sessions You know, if you have long ass sessions You know, if you have long ass sessions needing constant intervention, there's needing constant intervention, there's needing constant intervention, there's still something that's wrong. still something that's wrong. still something that's wrong. If you're getting a constant slot If you're getting a constant slot If you're getting a constant slot factory, obviously know like things are factory, obviously know like things are factory, obviously know like things are things are not good. things are not good. things are not good. And if you're asking if you find And if you're asking if you find And if you're asking if you find yourself asking, you know, how are these yourself asking, you know, how are these yourself asking, you know, how are these other companies shipping so fast? Like other companies shipping so fast? Like other companies shipping so fast? Like how are model companies releasing models how are model companies releasing models how are model companies releasing models at like a month and a half two-month at like a month and a half two-month at like a month and a half two-month cadence? cadence? cadence? Clearly they they have something which Clearly they they have something which Clearly they they have something which we don't, right?

we don't, right? we don't, right? And so, I guess everyone's kind of And so, I guess everyone's kind of And so, I guess everyone's kind of thinking so, you know, how do we solve thinking so, you know, how do we solve thinking so, you know, how do we solve this correctly? this correctly? this correctly? And so, I think the first thing to And so, I think the first thing to And so, I think the first thing to realize is we need to frame it realize is we need to frame it realize is we need to frame it correctly, right? It isn't really an correctly, right? It isn't really an correctly, right? It isn't really an IC's job. IC's job. IC's job. It's a job for leadership. It's a job It's a job for leadership. It's a job It's a job for leadership. It's a job for the company, right? Making engineers for the company, right? Making engineers for the company, right? Making engineers work well with their agents is truly the work well with their agents is truly the work well with their agents is truly the most impactful thing you could do as an most impactful thing you could do as an most impactful thing you could do as an organization. organization. organization. Because that's going to enable Because that's going to enable Because that's going to enable your engineers to ship faster and with your engineers to ship faster and with your engineers to ship faster and with confidence and avoid, you know, a lot of confidence and avoid, you know, a lot of confidence and avoid, you know, a lot of incidents. incidents. incidents. If we live in this figure it out for If we live in this figure it out for If we live in this figure it out for yourself paradigm, like people are going yourself paradigm, like people are going yourself paradigm, like people are going to get outsized productivity, some to get outsized productivity, some to get outsized productivity, some people aren't, and like the people who people aren't, and like the people who people aren't, and like the people who are generating like 10 PRs a day are are generating like 10 PRs a day are are generating like 10 PRs a day are going to like look like, you know, gods going to like look like, you know, gods going to like look like, you know, gods compared to people who are shipping like compared to people who are shipping like compared to people who are shipping like one to two. And the one to two PR people one to two. And the one to two PR people one to two. And the one to two PR people are actually going to get left with the are actually going to get left with the are actually going to get left with the review burden. And that's actually a review burden. And that's actually a review burden. And that's actually a really, really bad thing. Cuz now not really, really bad thing. Cuz now not really, really bad thing. Cuz now not only can they not ship, they're going to only can they not ship, they're going to only can they not ship, they're going to actually see bad code and then curse the actually see bad code and then curse the actually see bad code and then curse the agents and hence not be able to get on agents and hence not be able to get on agents and hence not be able to get on to the let's ship 10 PRs, right?

to the let's ship 10 PRs, right? to the let's ship 10 PRs, right? And so it's really important to do this. And so it's really important to do this. And so it's really important to do this. If it's a problem facing the team, If it's a problem facing the team, If it's a problem facing the team, there's a few things you can do, right? there's a few things you can do, right? there's a few things you can do, right? The most impactful things that you can The most impactful things that you can The most impactful things that you can do to set up your code base to like make do to set up your code base to like make do to set up your code base to like make it work well require team buy-in. You it work well require team buy-in. You it work well require team buy-in. You can't just like if you want to change can't just like if you want to change can't just like if you want to change the way your code base is organized, you the way your code base is organized, you the way your code base is organized, you can't do that as an IC, right? And can't do that as an IC, right? And can't do that as an IC, right? And if you're treated as a leadership if you're treated as a leadership if you're treated as a leadership problem, then you can do things like problem, then you can do things like problem, then you can do things like this. this. this. So, the other thing this needs is So, the other thing this needs is So, the other thing this needs is harness engineering, right? Per code harness engineering, right? Per code harness engineering, right? Per code base. And I think there's a lot of base. And I think there's a lot of base. And I think there's a lot of content on this, so I just want to talk content on this, so I just want to talk content on this, so I just want to talk about a few principles, but I don't want about a few principles, but I don't want about a few principles, but I don't want to like make this talk about that to like make this talk about that to like make this talk about that because there's a lot of smart people. because there's a lot of smart people. because there's a lot of smart people. You're at AI engineer, right? Like this You're at AI engineer, right? Like this You're at AI engineer, right? Like this this conference is all about people this conference is all about people this conference is all about people telling you how to you best set up, you telling you how to you best set up, you telling you how to you best set up, you know, your code base to make things know, your code base to make things know, your code base to make things function well. So, I don't want to talk function well. So, I don't want to talk function well. So, I don't want to talk too much about this, but there's a few too much about this, but there's a few too much about this, but there's a few key principles here. key principles here. key principles here. Smart prompt injection is one of them. Smart prompt injection is one of them. Smart prompt injection is one of them. You want to treat your entire code base You want to treat your entire code base You want to treat your entire code base as one way to that so that you're able as one way to that so that you're able as one way to that so that you're able to like smartly prompt inject the model to like smartly prompt inject the model to like smartly prompt inject the model with just the right context at just the with just the right context at just the with just the right context at just the right time. Without you needing to do right time. Without you needing to do right time. Without you needing to do it.

it. it. And that's the framing. You want to be And that's the framing. You want to be And that's the framing. You want to be able to say, "Okay, I've set it off on able to say, "Okay, I've set it off on able to say, "Okay, I've set it off on this task. It has like a map of how to this task. It has like a map of how to this task. It has like a map of how to find the things it needs at the time it find the things it needs at the time it find the things it needs at the time it needs it." If it's looking at some code needs it." If it's looking at some code needs it." If it's looking at some code and that code has, let's say some and that code has, let's say some and that code has, let's say some documentation, documentation, documentation, it the documentation needs to live in it the documentation needs to live in it the documentation needs to live in the comments. So, if it ever grabbed the comments. So, if it ever grabbed the comments. So, if it ever grabbed into that code, it reads the comment, into that code, it reads the comment, into that code, it reads the comment, goes to that file, finds all the goes to that file, finds all the goes to that file, finds all the information about it. information about it. information about it. That's just one example. The second is That's just one example. The second is That's just one example. The second is close the loop, right? You got to make a close the loop, right? You got to make a close the loop, right? You got to make a self-healing system cuz slop is self-healing system cuz slop is self-healing system cuz slop is inevitable. There is going to be some inevitable. There is going to be some inevitable. There is going to be some slop that's going to seep in. But, you slop that's going to seep in. But, you slop that's going to seep in. But, you need to have a pipeline and a way to need to have a pipeline and a way to need to have a pipeline and a way to close the loop to remove the slop, to close the loop to remove the slop, to close the loop to remove the slop, to detect it, and to be able to like detect it, and to be able to like detect it, and to be able to like self-heal the system. self-heal the system. self-heal the system. And then you need to iterate And then you need to iterate And then you need to iterate continuously. And I can't emphasize this continuously. And I can't emphasize this continuously. And I can't emphasize this enough. Like, you can't assume that you enough. Like, you can't assume that you enough. Like, you can't assume that you do this for a month and you're done. do this for a month and you're done. do this for a month and you're done. Like, things are going to change Like, things are going to change Like, things are going to change constantly underneath. So, you need to constantly underneath. So, you need to constantly underneath. So, you need to keep this as one of the things that you keep this as one of the things that you keep this as one of the things that you have to do as an organization. have to do as an organization. have to do as an organization. And the third most important thing is And the third most important thing is And the third most important thing is treat it like a human problem, guys. treat it like a human problem, guys. treat it like a human problem, guys. Like, this isn't It's not, you know, oh, Like, this isn't It's not, you know, oh, Like, this isn't It's not, you know, oh, it's this tool. Like, people will figure it's this tool. Like, people will figure it's this tool. Like, people will figure it out. Let's just mandate our way it out. Let's just mandate our way it out. Let's just mandate our way through life. Like, that's just not through life. Like, that's just not through life. Like, that's just not going to work. So, treat it like a human going to work. So, treat it like a human going to work. So, treat it like a human problem. Fear is real. Human emotions problem. Fear is real. Human emotions problem. Fear is real. Human emotions are real. We should recognize it.

are real. We should recognize it. are real. We should recognize it. So, you know, enough gyan or as like So, you know, enough gyan or as like So, you know, enough gyan or as like it's more like the Hindi way to say it's more like the Hindi way to say it's more like the Hindi way to say like, you know, I'm enough prof like I'm like, you know, I'm enough prof like I'm like, you know, I'm enough prof like I'm kind of giving you sermons. But, like, kind of giving you sermons. But, like, kind of giving you sermons. But, like, how do you really do this, right? Like, how do you really do this, right? Like, how do you really do this, right? Like, these are all principles. these are all principles. these are all principles. What's the real playbook? So, here's What's the real playbook? So, here's What's the real playbook? So, here's what we did and here's like I'm not what we did and here's like I'm not what we did and here's like I'm not going to over like emphasize on this is going to over like emphasize on this is going to over like emphasize on this is the exact way to do it, but this is like the exact way to do it, but this is like the exact way to do it, but this is like roughly how we did it and you can take roughly how we did it and you can take roughly how we did it and you can take from it what you choose. from it what you choose. from it what you choose. The first thing is do the basics, right? The first thing is do the basics, right? The first thing is do the basics, right? You got to do them right. Progressive You got to do them right. Progressive You got to do them right. Progressive disclosure, I can't emphasize this disclosure, I can't emphasize this disclosure, I can't emphasize this enough, is really, really powerful, enough, is really, really powerful, enough, is really, really powerful, right? Find your best ICs and find what right? Find your best ICs and find what right? Find your best ICs and find what how they are making the code base work how they are making the code base work how they are making the code base work for them. Take those practices and pass for them. Take those practices and pass for them. Take those practices and pass them or guide. them or guide. them or guide. People can't live in their own People can't live in their own People can't live in their own practices. And this is really hard for practices. And this is really hard for practices. And this is really hard for engineers to do it. It's basically engineers to do it. It's basically engineers to do it. It's basically accepting that my setup is imperfect. accepting that my setup is imperfect. accepting that my setup is imperfect. And like engineers don't like to hear And like engineers don't like to hear And like engineers don't like to hear that. But you got to figure out a way to that. But you got to figure out a way to that. But you got to figure out a way to find those best practices and ship them find those best practices and ship them find those best practices and ship them across. Make sure that that's kind of a across. Make sure that that's kind of a across. Make sure that that's kind of a shared setup. shared setup. shared setup. The second thing we did was there's one The second thing we did was there's one The second thing we did was there's one high value skill that we invested in. In high value skill that we invested in. In high value skill that we invested in. In our case it was this thing called ship our case it was this thing called ship our case it was this thing called ship it.

it. it. What it did was the moment you're done What it did was the moment you're done What it did was the moment you're done with your code, it takes care of with your code, it takes care of with your code, it takes care of everything from code done to PR ready everything from code done to PR ready everything from code done to PR ready for review. for review. for review. Which means you got to open a PR, figure Which means you got to open a PR, figure Which means you got to open a PR, figure out your opinions, handle the handle all out your opinions, handle the handle all out your opinions, handle the handle all the comments, you know, handle all the the comments, you know, handle all the the comments, you know, handle all the PR descriptions, the merge comments, PR descriptions, the merge comments, PR descriptions, the merge comments, everything, right? It handles CI everything, right? It handles CI everything, right? It handles CI failures. It basically runs through failures. It basically runs through failures. It basically runs through these loops. these loops. these loops. And what this meant was often the skill And what this meant was often the skill And what this meant was often the skill was running for over an hour. was running for over an hour. was running for over an hour. And that scared people, but once you And that scared people, but once you And that scared people, but once you actually figure out once they saw the actually figure out once they saw the actually figure out once they saw the value, they get invested, right? Because value, they get invested, right? Because value, they get invested, right? Because it's one skill which tells them, "Okay, it's one skill which tells them, "Okay, it's one skill which tells them, "Okay, this AI thing can actually work for me. this AI thing can actually work for me. this AI thing can actually work for me. I don't need to constantly baby sit it. I don't need to constantly baby sit it. I don't need to constantly baby sit it. I can trust it." I can trust it." I can trust it." The third thing and really important is The third thing and really important is The third thing and really important is to close the loop, right? So we wired to close the loop, right? So we wired to close the loop, right? So we wired issues and boards into the repo. Like we issues and boards into the repo. Like we issues and boards into the repo. Like we added CICD. We added gigantic reviews. added CICD. We added gigantic reviews. added CICD. We added gigantic reviews. We have like a code gardener that We have like a code gardener that We have like a code gardener that actually goes back and looks through a actually goes back and looks through a actually goes back and looks through a whole bunch of like every night it'll whole bunch of like every night it'll whole bunch of like every night it'll run and look at the code and check if run and look at the code and check if run and look at the code and check if something like not organized correctly. something like not organized correctly. something like not organized correctly. What does correct organization mean will What does correct organization mean will What does correct organization mean will depend on your code base. Get people invested and I can't Get people invested and I can't emphasize this enough. You have to win emphasize this enough. You have to win emphasize this enough. You have to win over the skeptics. It's really easy to over the skeptics. It's really easy to over the skeptics. It's really easy to say like the skeptic is just someone say like the skeptic is just someone say like the skeptic is just someone who's scared. It's really hard to get who's scared. It's really hard to get who's scared. It's really hard to get them to buy in, but if you can get them them to buy in, but if you can get them them to buy in, but if you can get them to buy in, you know you're doing to buy in, you know you're doing to buy in, you know you're doing something right.

something right. something right. You have to get them to be able to edit You have to get them to be able to edit You have to get them to be able to edit and play with the shared setup cuz and play with the shared setup cuz and play with the shared setup cuz that's the true way you know that that's the true way you know that that's the true way you know that they're actually invested, right? And they're actually invested, right? And they're actually invested, right? And this is where you got to it ensure this is where you got to it ensure this is where you got to it ensure you're iterating constantly. If people you're iterating constantly. If people you're iterating constantly. If people are like and this is the hardest thing are like and this is the hardest thing are like and this is the hardest thing for engineers again cuz you're basically for engineers again cuz you're basically for engineers again cuz you're basically saying I'm never going to like get to saying I'm never going to like get to saying I'm never going to like get to perfection in my setup. perfection in my setup. perfection in my setup. But you got to be okay with that. You But you got to be okay with that. You But you got to be okay with that. You have to do it and you You to treat it have to do it and you You to treat it have to do it and you You to treat it like, you know, X% of your IC time is like, you know, X% of your IC time is like, you know, X% of your IC time is probably going to be spent on it trading probably going to be spent on it trading probably going to be spent on it trading on this thing, which is not going to on this thing, which is not going to on this thing, which is not going to lead to like meaningful PRs like up lead to like meaningful PRs like up lead to like meaningful PRs like up front, but it's useful and it's worth front, but it's useful and it's worth front, but it's useful and it's worth it. it. it. And, you know, I don't want to say this And, you know, I don't want to say this And, you know, I don't want to say this is perfect, right? Like we faced a ton is perfect, right? Like we faced a ton is perfect, right? Like we faced a ton of issues while doing this. of issues while doing this. of issues while doing this. And I'm just going to walk you through And I'm just going to walk you through And I'm just going to walk you through some of them. But it's an iteration some of them. But it's an iteration some of them. But it's an iteration loop, so you got to treat it like a loop, so you got to treat it like a loop, so you got to treat it like a piece of feedback. piece of feedback. piece of feedback. So, what are the problems we hit, right? So, what are the problems we hit, right? So, what are the problems we hit, right? There are too many issues. Like when we There are too many issues. Like when we There are too many issues. Like when we started like we blew up to like 4 or 500 started like we blew up to like 4 or 500 started like we blew up to like 4 or 500 issues issues issues uh without like, you know, just I think uh without like, you know, just I think uh without like, you know, just I think within like a couple of weeks, which is within like a couple of weeks, which is within like a couple of weeks, which is a crazy number for like a repo. a crazy number for like a repo. a crazy number for like a repo. And then, you know, there are so many And then, you know, there are so many And then, you know, there are so many like different agents all trying to like different agents all trying to like different agents all trying to create issues cuz they've not been wired create issues cuz they've not been wired create issues cuz they've not been wired correctly.

correctly. correctly. There's a lack of agreement. People, you There's a lack of agreement. People, you There's a lack of agreement. People, you know, as soon as people saw like, "Oh, know, as soon as people saw like, "Oh, know, as soon as people saw like, "Oh, this isn't working perfectly or the way this isn't working perfectly or the way this isn't working perfectly or the way I expected it." There is super easy for I expected it." There is super easy for I expected it." There is super easy for them to say, "You know what? I'm just them to say, "You know what? I'm just them to say, "You know what? I'm just going to go back to babysitting my going to go back to babysitting my going to go back to babysitting my agent." You don't want that. You want to agent." You don't want that. You want to agent." You don't want that. You want to actually take their feedback and put it actually take their feedback and put it actually take their feedback and put it back into the skill and improve the back into the skill and improve the back into the skill and improve the skill. skill. skill. Agents are taking too long. This is Agents are taking too long. This is Agents are taking too long. This is actually one of those expectation actually one of those expectation actually one of those expectation setting things. setting things. setting things. It's good if agents take too long. That It's good if agents take too long. That It's good if agents take too long. That means you can actually go off and do means you can actually go off and do means you can actually go off and do other things and you have confidence other things and you have confidence other things and you have confidence that they're doing the right thing. that they're doing the right thing. that they're doing the right thing. At the end of the day like the moment we At the end of the day like the moment we At the end of the day like the moment we hit this reasoning paradigm, the longer hit this reasoning paradigm, the longer hit this reasoning paradigm, the longer the agent like thought, the better its the agent like thought, the better its the agent like thought, the better its output. You can treat like it's It's output. You can treat like it's It's output. You can treat like it's It's almost like a similar mindset for your almost like a similar mindset for your almost like a similar mindset for your entire code base and for your skills. entire code base and for your skills. entire code base and for your skills. There's going to be merge hell and like There's going to be merge hell and like There's going to be merge hell and like we just have to deal with it. We have to we just have to deal with it. We have to we just have to deal with it. We have to figure out a way to deal with this. figure out a way to deal with this. figure out a way to deal with this. There is going to be slop when you're There is going to be slop when you're There is going to be slop when you're going to write experiments. Treat it going to write experiments. Treat it going to write experiments. Treat it like it's own thing, right? What we said like it's own thing, right? What we said like it's own thing, right? What we said was okay, people are generating this was okay, people are generating this was okay, people are generating this code, but it's not relevant. It's not code, but it's not relevant. It's not code, but it's not relevant. It's not going to be shipped. It's just an It's a going to be shipped. It's just an It's a going to be shipped. It's just an It's a prototype. So, treat it like one. Get it prototype. So, treat it like one. Get it prototype. So, treat it like one. Get it to opt out of all the rigorous other to opt out of all the rigorous other to opt out of all the rigorous other standards you've got across your code standards you've got across your code standards you've got across your code base.

base. base. And realize people vary on the spectrum, And realize people vary on the spectrum, And realize people vary on the spectrum, right? And depending on the day, right? And depending on the day, right? And depending on the day, depending on like what they're going depending on like what they're going depending on like what they're going through, they're going to vary on the through, they're going to vary on the through, they're going to vary on the spectrum. You have to be able to talk to spectrum. You have to be able to talk to spectrum. You have to be able to talk to them and figure out, "Hey, okay, why why them and figure out, "Hey, okay, why why them and figure out, "Hey, okay, why why are you facing this? Like is it if the are you facing this? Like is it if the are you facing this? Like is it if the model changed, the hardness changed, model changed, the hardness changed, model changed, the hardness changed, again you need to go revisit something. again you need to go revisit something. again you need to go revisit something. Figure that out. Figure that out. Figure that out. And I think the biggest like the easiest And I think the biggest like the easiest And I think the biggest like the easiest way to say this is instead of saying way to say this is instead of saying way to say this is instead of saying like the model is so dumb, like we have like the model is so dumb, like we have like the model is so dumb, like we have to ask how can I make it smarter or how to ask how can I make it smarter or how to ask how can I make it smarter or how can I edit like and not This is where I can I edit like and not This is where I can I edit like and not This is where I have like crossed out the my. It's not a have like crossed out the my. It's not a have like crossed out the my. It's not a personal setup. It's the shared setup personal setup. It's the shared setup personal setup. It's the shared setup that you have to invest in. that you have to invest in. that you have to invest in. And it's a mindset, right? You have to And it's a mindset, right? You have to And it's a mindset, right? You have to go full send. go full send. go full send. And like I want to end you with I want And like I want to end you with I want And like I want to end you with I want to end with this. Like, I learned skiing to end with this. Like, I learned skiing to end with this. Like, I learned skiing like a couple years back and the first like a couple years back and the first like a couple years back and the first like the hardest thing for me was you like the hardest thing for me was you like the hardest thing for me was you actually have to commit to it. actually have to commit to it. actually have to commit to it. If you're pizza braking, you're going to If you're pizza braking, you're going to If you're pizza braking, you're going to crash. Like, no matter what. You have to crash. Like, no matter what. You have to crash. Like, no matter what. You have to commit to the speed in order to actually commit to the speed in order to actually commit to the speed in order to actually get and feel like, okay, get and feel like, okay, get and feel like, okay, that's how I can turn and that's how I that's how I can turn and that's how I that's how I can turn and that's how I can truly ski. And so, I'm going to can truly ski. And so, I'm going to can truly ski. And so, I'm going to leave you with this.

leave you with this. leave you with this. Like, just Like, just Like, just be okay with be okay with failing. You be okay with be okay with failing. You be okay with be okay with failing. You have to, you know, go full send and be have to, you know, go full send and be have to, you know, go full send and be okay with falling. It's fine. okay with falling. It's fine. okay with falling. It's fine. The point is to be able to recover from The point is to be able to recover from The point is to be able to recover from that and that will allow you to truly that and that will allow you to truly that and that will allow you to truly feel the AGI. feel the AGI. feel the AGI. Yeah, well, that's that's me and I'm Yeah, well, that's that's me and I'm Yeah, well, that's that's me and I'm happy to take any questions. Uh so, the I'm going to repeat the Uh so, the I'm going to repeat the question for the recording. Uh question for the recording. Uh question for the recording. Uh strategies that you found best for strategies that you found best for strategies that you found best for progressive disclosure. So, I think a progressive disclosure. So, I think a progressive disclosure. So, I think a couple things, right? Uh couple things, right? Uh couple things, right? Uh the first thing is the first thing is the first thing is even in your skill.md files, don't even in your skill.md files, don't even in your skill.md files, don't overload it. Like, we've kind of set a overload it. Like, we've kind of set a overload it. Like, we've kind of set a hard limit for like 100 lines in your hard limit for like 100 lines in your hard limit for like 100 lines in your skill.md cuz your skill is really a skill.md cuz your skill is really a skill.md cuz your skill is really a folder. folder. folder. So, that's step one. Make sure and I So, that's step one. Make sure and I So, that's step one. Make sure and I think I spoke about this during the think I spoke about this during the think I spoke about this during the talk, but talk, but talk, but when you have like when you have like when you have like some code that requires like a runbook, some code that requires like a runbook, some code that requires like a runbook, make sure the runbook is reflected in make sure the runbook is reflected in make sure the runbook is reflected in the comments so that if a code if the comments so that if a code if the comments so that if a code if somehow the code like the agent figures somehow the code like the agent figures somehow the code like the agent figures its way into like grapping into the code its way into like grapping into the code its way into like grapping into the code base and find that file, it knows I need base and find that file, it knows I need base and find that file, it knows I need to go look at this for like all the to go look at this for like all the to go look at this for like all the description of how this is relevant.

description of how this is relevant. description of how this is relevant. Right? You have to organize and I think Right? You have to organize and I think Right? You have to organize and I think this is why I talk about harness this is why I talk about harness this is why I talk about harness engineering because your entire code engineering because your entire code engineering because your entire code base can be set up to encourage base can be set up to encourage base can be set up to encourage progressive disclosure. Don't overload progressive disclosure. Don't overload progressive disclosure. Don't overload your cloud MD or your agent's MD file your cloud MD or your agent's MD file your cloud MD or your agent's MD file into like one big thing. You want to into like one big thing. You want to into like one big thing. You want to make sure that like it's a thin index make sure that like it's a thin index make sure that like it's a thin index that can point through the right files that can point through the right files that can point through the right files and that's what the agent gets in its and that's what the agent gets in its and that's what the agent gets in its like first prompt cuz that's what gets like first prompt cuz that's what gets like first prompt cuz that's what gets loaded when it starts to work. loaded when it starts to work. loaded when it starts to work. So these are like some really powerful So these are like some really powerful So these are like some really powerful strategies that and the way you know strategies that and the way you know strategies that and the way you know this is working is when you give it a this is working is when you give it a this is working is when you give it a prompt. When you give it the first prompt. When you give it the first prompt. When you give it the first prompt, see what it's doing. Is it prompt, see what it's doing. Is it prompt, see what it's doing. Is it grabbing or does it know where to go? grabbing or does it know where to go? grabbing or does it know where to go? But how much context is it burning when But how much context is it burning when But how much context is it burning when it like immediately? So is it like I it like immediately? So is it like I it like immediately? So is it like I think like 20, 25K tokens get taken think like 20, 25K tokens get taken think like 20, 25K tokens get taken anyway, but like how much more is anyway, but like how much more is anyway, but like how much more is getting added? If you're coming to like getting added? If you're coming to like getting added? If you're coming to like 40K, 50K, like something's wrong. 40K, 50K, like something's wrong. 40K, 50K, like something's wrong. That's not really progressive That's not really progressive That's not really progressive disclosure. disclosure. disclosure. So you have to figure out these So you have to figure out these So you have to figure out these boundaries and then like based on this boundaries and then like based on this boundaries and then like based on this it's an iteration cycle.

All right. Well, if there there aren't All right. Well, if there there aren't any other questions, feel free to find any other questions, feel free to find any other questions, feel free to find me. Happy to talk about harness me. Happy to talk about harness me. Happy to talk about harness engineering in general or like anything engineering in general or like anything engineering in general or like anything else, but yeah, thank you for listening.