Prototyping as Leadership: How a CTO Ships with AI Agents — Hursh Agrawal, The Browser Company

Hi everyone, Hi everyone, thanks for coming. Uh, I'm Hersh thanks for coming. Uh, I'm Hersh thanks for coming. Uh, I'm Hersh Agarwal. I'm the CTO and co-founder of Agarwal. I'm the CTO and co-founder of Agarwal. I'm the CTO and co-founder of the browser company, makers of the Arc the browser company, makers of the Arc the browser company, makers of the Arc and Dia browsers. Uh, and I'm going to and Dia browsers. Uh, and I'm going to and Dia browsers. Uh, and I'm going to talk about prototyping as leadership as talk about prototyping as leadership as talk about prototyping as leadership as you get to a manager schedule, how you you get to a manager schedule, how you you get to a manager schedule, how you can keep building. So this is my actual can keep building. So this is my actual can keep building. So this is my actual calendar from last week. Uh I imagine calendar from last week. Uh I imagine calendar from last week. Uh I imagine this is kind familiar to some of you in this is kind familiar to some of you in this is kind familiar to some of you in leadership. Um that was my actual week. leadership. Um that was my actual week. leadership. Um that was my actual week. I uh have a whole org that reports up I uh have a whole org that reports up I uh have a whole org that reports up into me. So I have 15 plus recurring into me. So I have 15 plus recurring into me. So I have 15 plus recurring meetings a week uh standups, reviews, meetings a week uh standups, reviews, meetings a week uh standups, reviews, recruiting meetings, etc. Uh and seven recruiting meetings, etc. Uh and seven recruiting meetings, etc. Uh and seven direct reports. And I've noticed direct reports. And I've noticed direct reports. And I've noticed something over the last several months, something over the last several months, something over the last several months, which is I've started to consistently which is I've started to consistently which is I've started to consistently ship, you know, two to 10 PRs a week. Uh ship, you know, two to 10 PRs a week. Uh ship, you know, two to 10 PRs a week. Uh and this is new. This was not possible and this is new. This was not possible and this is new. This was not possible several months ago. It's really because several months ago. It's really because several months ago. It's really because of these new AI agents. Uh, and notably, of these new AI agents. Uh, and notably, of these new AI agents. Uh, and notably, I also have a toddler at home, so I like I also have a toddler at home, so I like I also have a toddler at home, so I like cannot work 996. You know, I'm working cannot work 996. You know, I'm working cannot work 996. You know, I'm working 40 50 hours a week. So, I really have to 40 50 hours a week. So, I really have to 40 50 hours a week. So, I really have to fit all this in into a regular week. fit all this in into a regular week. fit all this in into a regular week. And pre-I agents, uh, as you grew as a And pre-I agents, uh, as you grew as a And pre-I agents, uh, as you grew as a leader, you had more of the org leader, you had more of the org leader, you had more of the org reporting to you. Uh, you would sort of reporting to you. Uh, you would sort of reporting to you. Uh, you would sort of the way you would influence the org was the way you would influence the org was the way you would influence the org was through communicating to people. Uh so through communicating to people. Uh so through communicating to people. Uh so you'd uh write road maps, docs, you'd uh write road maps, docs, you'd uh write road maps, docs, meetings, and you'd sort of incept uh meetings, and you'd sort of incept uh meetings, and you'd sort of incept uh your context and what you wanted to your context and what you wanted to your context and what you wanted to build with your employees uh and your build with your employees uh and your build with your employees uh and your engineers. And now, interestingly, in engineers. And now, interestingly, in engineers. And now, interestingly, in the last few months, as coding agents the last few months, as coding agents the last few months, as coding agents have become more autonomous and able to have become more autonomous and able to have become more autonomous and able to handle longer tasks, the manager handle longer tasks, the manager handle longer tasks, the manager schedule, as Paul Graham put it, is schedule, as Paul Graham put it, is schedule, as Paul Graham put it, is suddenly usable as building time. You

suddenly usable as building time. You suddenly usable as building time. You can actually ship stuff. can actually ship stuff. can actually ship stuff. And so I think building is part of the And so I think building is part of the And so I think building is part of the job now. you can do it first of all as a job now. you can do it first of all as a job now. you can do it first of all as a leader but it's I think it's also leader but it's I think it's also leader but it's I think it's also becoming necessary as part of the job becoming necessary as part of the job becoming necessary as part of the job for two reasons. First, uh, the for two reasons. First, uh, the for two reasons. First, uh, the technology world is changing where technology world is changing where technology world is changing where suddenly the core technology that's part suddenly the core technology that's part suddenly the core technology that's part of our products are these frontier of our products are these frontier of our products are these frontier models that change every 3 months, which models that change every 3 months, which models that change every 3 months, which is a new dynamic that's come out. And as is a new dynamic that's come out. And as is a new dynamic that's come out. And as each new frontier model comes out, its each new frontier model comes out, its each new frontier model comes out, its capabilities change. The contours of capabilities change. The contours of capabilities change. The contours of what it's useful for change. It's, you what it's useful for change. It's, you what it's useful for change. It's, you know, how it actually reacts to know, how it actually reacts to know, how it actually reacts to prompting changes. And there's so much prompting changes. And there's so much prompting changes. And there's so much noise on Twitter, even internally for noise on Twitter, even internally for noise on Twitter, even internally for us, and so many opinions with each new us, and so many opinions with each new us, and so many opinions with each new model release on what's good and what's model release on what's good and what's model release on what's good and what's bad. And I found it is impossible to bad. And I found it is impossible to bad. And I found it is impossible to tell what a new model is good for unless tell what a new model is good for unless tell what a new model is good for unless you have your hands in it and you're you have your hands in it and you're you have your hands in it and you're using it all day long. And so having an using it all day long. And so having an using it all day long. And so having an ability to slot in building time into ability to slot in building time into ability to slot in building time into your schedule means you can figure out your schedule means you can figure out your schedule means you can figure out the contours of what is this new model the contours of what is this new model the contours of what is this new model family capable of and both how do I family capable of and both how do I family capable of and both how do I direct my engineers in terms of uh direct my engineers in terms of uh direct my engineers in terms of uh setting expectations on how they should setting expectations on how they should setting expectations on how they should be building but also getting a sense of be building but also getting a sense of be building but also getting a sense of how does this fit into our product? How how does this fit into our product? How how does this fit into our product? How does this fit into our business? You does this fit into our business? You does this fit into our business? You know how is this how is our strategy know how is this how is our strategy know how is this how is our strategy going to change because of this? What's going to change because of this? What's going to change because of this? What's going to happen in three to six months going to happen in three to six months going to happen in three to six months when the open source models catch up?

when the open source models catch up? when the open source models catch up? All of that intuition comes from All of that intuition comes from All of that intuition comes from actually using the models and building. actually using the models and building. actually using the models and building. I've also found even if you've built the I've also found even if you've built the I've also found even if you've built the intuition, it's actually tough to intuition, it's actually tough to intuition, it's actually tough to communicate that to other people who communicate that to other people who communicate that to other people who haven't played with the models. Uh, and haven't played with the models. Uh, and haven't played with the models. Uh, and so you're like, "Oh, this new thing is so you're like, "Oh, this new thing is so you're like, "Oh, this new thing is possible. It's going to be amazing." And possible. It's going to be amazing." And possible. It's going to be amazing." And your engineers are like, "Okay, yeah, your engineers are like, "Okay, yeah, your engineers are like, "Okay, yeah, sure." Um, so it's really helpful to sure." Um, so it's really helpful to sure." Um, so it's really helpful to have some time to actually build stuff have some time to actually build stuff have some time to actually build stuff so you can show them, you know, you can so you can show them, you know, you can so you can show them, you know, you can be like, "Hey, I built this prototype be like, "Hey, I built this prototype be like, "Hey, I built this prototype with this new model family. It works in with this new model family. It works in with this new model family. It works in the product. Here's an actual prototype the product. Here's an actual prototype the product. Here's an actual prototype you can play with." And this is just so you can play with." And this is just so you can play with." And this is just so much faster and more efficient than much faster and more efficient than much faster and more efficient than trying to convince people every 3 months trying to convince people every 3 months trying to convince people every 3 months when a new model family comes out. when a new model family comes out. when a new model family comes out. I also think leaders are really well I also think leaders are really well I also think leaders are really well suited for it. You know, you all have uh suited for it. You know, you all have uh suited for it. You know, you all have uh context, more context than anybody else context, more context than anybody else context, more context than anybody else uh in the organization about the uh in the organization about the uh in the organization about the business, the strategy, the trade-offs, business, the strategy, the trade-offs, business, the strategy, the trade-offs, the decisions to be made, what needs to the decisions to be made, what needs to the decisions to be made, what needs to be done, what's uh what's coming up, be done, what's uh what's coming up, be done, what's uh what's coming up, what are imperatives. And so the what are imperatives. And so the what are imperatives. And so the steering of a leader uh as you are steering of a leader uh as you are steering of a leader uh as you are prompting these models is to per token prompting these models is to per token prompting these models is to per token uh more impactful than an IC's. And so uh more impactful than an IC's. And so uh more impactful than an IC's. And so you can really fit this into a little you can really fit this into a little you can really fit this into a little bit of time. The delegation skill also bit of time. The delegation skill also bit of time. The delegation skill also transfers like you delegate to people.

transfers like you delegate to people. transfers like you delegate to people. That's sort of one of the core skills of That's sort of one of the core skills of That's sort of one of the core skills of being a leader and that transfers to being a leader and that transfers to being a leader and that transfers to agents pretty well. You know it's agents pretty well. You know it's agents pretty well. You know it's setting goals, giving context, checking setting goals, giving context, checking setting goals, giving context, checking in, even coaching the agent. You know in, even coaching the agent. You know in, even coaching the agent. You know what if you tried this? Um, and then what if you tried this? Um, and then what if you tried this? Um, and then I've also found I I think this is maybe I've also found I I think this is maybe I've also found I I think this is maybe specific to now. Uh, maybe this will specific to now. Uh, maybe this will specific to now. Uh, maybe this will change in several months as these models change in several months as these models change in several months as these models get better, but the models are really get better, but the models are really get better, but the models are really good at execution, but still not good at execution, but still not good at execution, but still not unbelievable at judgment. You know, unbelievable at judgment. You know, unbelievable at judgment. You know, often the model will come back and be often the model will come back and be often the model will come back and be like, hey, that algorithm idea is not like, hey, that algorithm idea is not like, hey, that algorithm idea is not possible or like I can't do this. And so possible or like I can't do this. And so possible or like I can't do this. And so you have as a leader have to be like, you have as a leader have to be like, you have as a leader have to be like, oh, have you tried this thing? And the oh, have you tried this thing? And the oh, have you tried this thing? And the model's like, oh, okay, cool, cool, model's like, oh, okay, cool, cool, model's like, oh, okay, cool, cool, cool. I'll try that. And so it's a cool. I'll try that. And so it's a cool. I'll try that. And so it's a really nice dynamic you can have with really nice dynamic you can have with really nice dynamic you can have with these models if you have time slotted these models if you have time slotted these models if you have time slotted out for building. uh that really out for building. uh that really out for building. uh that really leverages all the context you have. leverages all the context you have. leverages all the context you have. So the great Julie Zo had a great tweet So the great Julie Zo had a great tweet So the great Julie Zo had a great tweet about this uh a month ago. She pulled about this uh a month ago. She pulled about this uh a month ago. She pulled some Bay Area technical leaders on what some Bay Area technical leaders on what some Bay Area technical leaders on what can you actually build? What should you can you actually build? What should you can you actually build? What should you be building in this new world? And four be building in this new world? And four be building in this new world? And four categories emerged. Uh you could build categories emerged. Uh you could build categories emerged. Uh you could build internal tools. So internal efficiency internal tools. So internal efficiency internal tools. So internal efficiency uh quality of life improvements just uh quality of life improvements just uh quality of life improvements just like gardening around the codebase and like gardening around the codebase and like gardening around the codebase and the product. This is really helpful. Um the product. This is really helpful. Um the product. This is really helpful. Um I really like this. I learned from this I really like this. I learned from this I really like this. I learned from this the celebration story. You can build the celebration story. You can build the celebration story. You can build artifacts to celebrate people on your artifacts to celebrate people on your artifacts to celebrate people on your team. And then arguably, I think the team. And then arguably, I think the team. And then arguably, I think the most important is the vision piece is most important is the vision piece is most important is the vision piece is really playing with the new model really playing with the new model really playing with the new model families, understanding what's possible families, understanding what's possible families, understanding what's possible viscerally yourself, and then trying to viscerally yourself, and then trying to viscerally yourself, and then trying to fit that into the business and building fit that into the business and building fit that into the business and building products that can show that off to your products that can show that off to your products that can show that off to your engineers so you can really push uh the engineers so you can really push uh the engineers so you can really push uh the boundaries of what's possible with the boundaries of what's possible with the boundaries of what's possible with the product and business. I she's right. I product and business. I she's right. I product and business. I she's right. I would not take any critical path work.

would not take any critical path work. would not take any critical path work. the last thing you want to do is to have the last thing you want to do is to have the last thing you want to do is to have that dependent on you, especially you're that dependent on you, especially you're that dependent on you, especially you're going to be dragged into fires and going to be dragged into fires and going to be dragged into fires and recruiting calls and meetings, etc. Uh recruiting calls and meetings, etc. Uh recruiting calls and meetings, etc. Uh so really trying to do one of these four so really trying to do one of these four so really trying to do one of these four categories is is optimal. categories is is optimal. categories is is optimal. So this is this is my how I work. Uh and So this is this is my how I work. Uh and So this is this is my how I work. Uh and what two to three hours a day of coding what two to three hours a day of coding what two to three hours a day of coding can actually look like. Uh so I usually can actually look like. Uh so I usually can actually look like. Uh so I usually have a morning coding block about an have a morning coding block about an have a morning coding block about an hour and this is when I look over last hour and this is when I look over last hour and this is when I look over last night's code, what the uh agent did last night's code, what the uh agent did last night's code, what the uh agent did last night, review it. I'll talk a little bit night, review it. I'll talk a little bit night, review it. I'll talk a little bit more about this. Uh, and then a few more about this. Uh, and then a few more about this. Uh, and then a few maybe steering blocks throughout the day maybe steering blocks throughout the day maybe steering blocks throughout the day interspersed between one-on- ones and interspersed between one-on- ones and interspersed between one-on- ones and reviews and meetings and everything else reviews and meetings and everything else reviews and meetings and everything else you need to do. And then the most you need to do. And then the most you need to do. And then the most important block is that 5:00 p.m. block important block is that 5:00 p.m. block important block is that 5:00 p.m. block at the end of the day where you really at the end of the day where you really at the end of the day where you really set up whatever overnight run it is you set up whatever overnight run it is you set up whatever overnight run it is you want to run. And I'll talk about that want to run. And I'll talk about that want to run. And I'll talk about that whether that's coding or training models whether that's coding or training models whether that's coding or training models or whatever you want the agent to do. or whatever you want the agent to do. or whatever you want the agent to do. I'll go over sort of three examples of I'll go over sort of three examples of I'll go over sort of three examples of tasks you can do. But really what I tasks you can do. But really what I tasks you can do. But really what I found work for me is this one overnight found work for me is this one overnight found work for me is this one overnight loop. uh you first in that 5pm block loop. uh you first in that 5pm block loop. uh you first in that 5pm block gather context I'll talk about that uh gather context I'll talk about that uh gather context I'll talk about that uh you set up the run you know you get ask you set up the run you know you get ask you set up the run you know you get ask answer any clarifying questions uh and answer any clarifying questions uh and answer any clarifying questions uh and then claude code or whatever your coding then claude code or whatever your coding then claude code or whatever your coding agent is overnight does the thing you agent is overnight does the thing you agent is overnight does the thing you know does the work for four six eight know does the work for four six eight know does the work for four six eight hours and in the morning you get this hours and in the morning you get this hours and in the morning you get this beautiful report uh and you figure out beautiful report uh and you figure out beautiful report uh and you figure out what to do with it and then you ship what to do with it and then you ship what to do with it and then you ship whatever it is so I'll talk about whatever it is so I'll talk about whatever it is so I'll talk about building features which is sort of the building features which is sort of the building features which is sort of the most obvious one uh I found this worked most obvious one uh I found this worked most obvious one uh I found this worked really well with evals and hill climbing really well with evals and hill climbing really well with evals and hill climbing so optimizing AI features uh and then a so optimizing AI features uh and then a so optimizing AI features uh and then a new thing actually in the maybe more new thing actually in the maybe more new thing actually in the maybe more recent uh generations of models is you recent uh generations of models is you recent uh generations of models is you can have these models train other ML can have these models train other ML can have these models train other ML models overnight too and this works

models overnight too and this works models overnight too and this works really really well. really really well. really really well. So building features um the big mindset So building features um the big mindset So building features um the big mindset shift for me that really helped was shift for me that really helped was shift for me that really helped was starting to think about building starting to think about building starting to think about building features not in terms of how do I build features not in terms of how do I build features not in terms of how do I build this feature and break it up and then this feature and break it up and then this feature and break it up and then give it prompting on how to do the give it prompting on how to do the give it prompting on how to do the individual task but rather what is all individual task but rather what is all individual task but rather what is all the context this frontier model needs to the context this frontier model needs to the context this frontier model needs to be able to make decisions like I would be able to make decisions like I would be able to make decisions like I would make. So trying to give it as much make. So trying to give it as much make. So trying to give it as much context as possible because if it's context as possible because if it's context as possible because if it's working for 6 hours, 8 hours overnight, working for 6 hours, 8 hours overnight, working for 6 hours, 8 hours overnight, you're not going to be there to steer you're not going to be there to steer you're not going to be there to steer it. And so you want it to have as much it. And so you want it to have as much it. And so you want it to have as much context as you do about the business context as you do about the business context as you do about the business goals of whatever you're trying to get goals of whatever you're trying to get goals of whatever you're trying to get it to do. So, uh, a little tip I found it to do. So, uh, a little tip I found it to do. So, uh, a little tip I found that has worked well for me, um, if you that has worked well for me, um, if you that has worked well for me, um, if you have a co-work agent, whatever your, uh, have a co-work agent, whatever your, uh, have a co-work agent, whatever your, uh, cloud co-work or codeex or I recommend cloud co-work or codeex or I recommend cloud co-work or codeex or I recommend DIA, I feel like that's the best one. DIA, I feel like that's the best one. DIA, I feel like that's the best one. Not biased at all. Um, whatever it is Not biased at all. Um, whatever it is Not biased at all. Um, whatever it is that's connected to your Slack and your that's connected to your Slack and your that's connected to your Slack and your Jira, Confluence, notion, the repo, etc. Jira, Confluence, notion, the repo, etc. Jira, Confluence, notion, the repo, etc. Uh, just I before a meeting or Uh, just I before a meeting or Uh, just I before a meeting or something, I'll be like at like 3:00 4 something, I'll be like at like 3:00 4 something, I'll be like at like 3:00 4 p.m. I'll be like, "Hey, I really want p.m. I'll be like, "Hey, I really want p.m. I'll be like, "Hey, I really want to build this. go and do like 20 minutes to build this. go and do like 20 minutes to build this. go and do like 20 minutes of research and go dig through all of of research and go dig through all of of research and go dig through all of Slack and Notion and everything else and Slack and Notion and everything else and Slack and Notion and everything else and come up with a cloud code prompt that I come up with a cloud code prompt that I come up with a cloud code prompt that I can post paste into a cloud code and can post paste into a cloud code and can post paste into a cloud code and just give me as much context as possible just give me as much context as possible just give me as much context as possible the trade-offs what we tried before what the trade-offs what we tried before what the trade-offs what we tried before what works what doesn't why are we're doing works what doesn't why are we're doing works what doesn't why are we're doing it what is like the business context of it what is like the business context of it what is like the business context of this feature I want to build or this this feature I want to build or this this feature I want to build or this model I want to train and that takes model I want to train and that takes model I want to train and that takes like 30 seconds to whisper flow into DIA like 30 seconds to whisper flow into DIA like 30 seconds to whisper flow into DIA or whatever your co-work agent is and or whatever your co-work agent is and or whatever your co-work agent is and the nice thing is this comes back with a the nice thing is this comes back with a the nice thing is this comes back with a giant prompt that then you can paste

giant prompt that then you can paste giant prompt that then you can paste into cloud code or cursor or codeex or into cloud code or cursor or codeex or into cloud code or cursor or codeex or whatever your coding agent is. So this whatever your coding agent is. So this whatever your coding agent is. So this is an example of a 5mm prompt I'll have is an example of a 5mm prompt I'll have is an example of a 5mm prompt I'll have that I give it before I go to bed. I'll that I give it before I go to bed. I'll that I give it before I go to bed. I'll say implement this whole feature and say implement this whole feature and say implement this whole feature and then just paste all of that context I then just paste all of that context I then just paste all of that context I got from my co-work agent. Uh and then got from my co-work agent. Uh and then got from my co-work agent. Uh and then it's important to think about it's important to think about it's important to think about verification. So as this uh agent is verification. So as this uh agent is verification. So as this uh agent is running overnight, you want it to verify running overnight, you want it to verify running overnight, you want it to verify how it's doing as it's building and how it's doing as it's building and how it's doing as it's building and testing the feature. So first I'll be testing the feature. So first I'll be testing the feature. So first I'll be like hey write the tests first so they like hey write the tests first so they like hey write the tests first so they capture what you do. This is really capture what you do. This is really capture what you do. This is really helpful because I I found of decoding helpful because I I found of decoding helpful because I I found of decoding agents if you if they write tests agents if you if they write tests agents if you if they write tests afterwards, they're a little sloppish. afterwards, they're a little sloppish. afterwards, they're a little sloppish. Um, and then I'll tell it to test the Um, and then I'll tell it to test the Um, and then I'll tell it to test the endto-end flow with computer use. You endto-end flow with computer use. You endto-end flow with computer use. You know, go around, click around, just make know, go around, click around, just make know, go around, click around, just make sure the flow works and matches the sure the flow works and matches the sure the flow works and matches the business context I gave you. And so the business context I gave you. And so the business context I gave you. And so the modern models are really good at modern models are really good at modern models are really good at reasoning about, okay, does this flow reasoning about, okay, does this flow reasoning about, okay, does this flow actually solve the problem I was trying actually solve the problem I was trying actually solve the problem I was trying to solve? to solve? to solve? And then uh I usually prompted to do as And then uh I usually prompted to do as And then uh I usually prompted to do as much work as possible so that when I much work as possible so that when I much work as possible so that when I wake up in the morning, the stack is wake up in the morning, the stack is wake up in the morning, the stack is ready. So split this into reviewer ready. So split this into reviewer ready. So split this into reviewer friendly PRs with clear descriptions, friendly PRs with clear descriptions, friendly PRs with clear descriptions, manage CI and get CI green and just make manage CI and get CI green and just make manage CI and get CI green and just make sure you're you're monitoring uh just to sure you're you're monitoring uh just to sure you're you're monitoring uh just to make sure the the whole stack is make sure the the whole stack is make sure the the whole stack is passable. AI code review is really passable. AI code review is really passable. AI code review is really important. We have a bunch of internal important. We have a bunch of internal important. We have a bunch of internal AI code reviewers uh you know pre the AI code reviewers uh you know pre the AI code reviewers uh you know pre the previous talk was talking about codto previous talk was talking about codto previous talk was talking about codto and other AI code reviewer. If you don't and other AI code reviewer. If you don't and other AI code reviewer. If you don't have any, I would look on Twitter for a have any, I would look on Twitter for a have any, I would look on Twitter for a bunch of AI code review skills and then bunch of AI code review skills and then bunch of AI code review skills and then just prompt it, hey, once you're done, just prompt it, hey, once you're done, just prompt it, hey, once you're done, run this AI code review skill in a clean run this AI code review skill in a clean run this AI code review skill in a clean sub agent, and then fix those things.

sub agent, and then fix those things. sub agent, and then fix those things. And then watch the PRs, fix every bot And then watch the PRs, fix every bot And then watch the PRs, fix every bot comment, every, you know, CI check that comment, every, you know, CI check that comment, every, you know, CI check that shows up, any sort of uh anything you shows up, any sort of uh anything you shows up, any sort of uh anything you need to do, resolve the threads, run need to do, resolve the threads, run need to do, resolve the threads, run this autonomously. Uh, don't ask me this autonomously. Uh, don't ask me this autonomously. Uh, don't ask me questions, and then I'll like throw in a questions, and then I'll like throw in a questions, and then I'll like throw in a little encouraging something, you know, little encouraging something, you know, little encouraging something, you know, be like, you'll do great. I believe in be like, you'll do great. I believe in be like, you'll do great. I believe in you. It's going to be great. Um, I don't you. It's going to be great. Um, I don't you. It's going to be great. Um, I don't know if that helps or not, but it's it's know if that helps or not, but it's it's know if that helps or not, but it's it's a I tend to do that. Uh, and then I tell a I tend to do that. Uh, and then I tell a I tend to do that. Uh, and then I tell them I'm going to bed. Just leave me a them I'm going to bed. Just leave me a them I'm going to bed. Just leave me a report in the morning on what you do. I report in the morning on what you do. I report in the morning on what you do. I need the full stack ready and a report need the full stack ready and a report need the full stack ready and a report on what trade-offs you made, how you on what trade-offs you made, how you on what trade-offs you made, how you did. Uh, and it does great. Uh, it did. Uh, and it does great. Uh, it did. Uh, and it does great. Uh, it actually modern models, new Opus 4.8 or actually modern models, new Opus 4.8 or actually modern models, new Opus 4.8 or the new GBT. They can handle what used the new GBT. They can handle what used the new GBT. They can handle what used to be, you know, weeks of work uh, in to be, you know, weeks of work uh, in to be, you know, weeks of work uh, in one overnight run and you come back in one overnight run and you come back in one overnight run and you come back in the morning with this uh, beautiful the morning with this uh, beautiful the morning with this uh, beautiful package ready for you. It actually makes package ready for you. It actually makes package ready for you. It actually makes the mornings kind of nice because you the mornings kind of nice because you the mornings kind of nice because you have this little present ready for you. have this little present ready for you. have this little present ready for you. Another example is optimizing AI Another example is optimizing AI Another example is optimizing AI features. So, first you're at the AI features. So, first you're at the AI features. So, first you're at the AI engineering conference. I assume you all engineering conference. I assume you all engineering conference. I assume you all are familiar with this. When you first are familiar with this. When you first are familiar with this. When you first build an LLM call or agent into a into a build an LLM call or agent into a into a build an LLM call or agent into a into a feature, it's not optimized. You know, feature, it's not optimized. You know, feature, it's not optimized. You know, you need to optimize it hill climb on you need to optimize it hill climb on you need to optimize it hill climb on some eval to get the quality, the some eval to get the quality, the some eval to get the quality, the latency, the cost to where you want it latency, the cost to where you want it latency, the cost to where you want it to be. This is also all doable to be. This is also all doable to be. This is also all doable overnight. Um, especially with the overnight. Um, especially with the overnight. Um, especially with the modern coding agents. So, uh, usually if modern coding agents. So, uh, usually if modern coding agents. So, uh, usually if I'm prototyping a feature, I will, uh, I'm prototyping a feature, I will, uh, I'm prototyping a feature, I will, uh, put a little feedback button on it. So put a little feedback button on it. So put a little feedback button on it. So after the LLM call or the agent runs, after the LLM call or the agent runs, after the LLM call or the agent runs, I'll be like, "Hey, uh just add a I'll be like, "Hey, uh just add a I'll be like, "Hey, uh just add a feedback button and a little text box.

feedback button and a little text box. feedback button and a little text box. And then in one of my 1 hour slots or And then in one of my 1 hour slots or And then in one of my 1 hour slots or 30-minut slots throughout the day, I'll 30-minut slots throughout the day, I'll 30-minut slots throughout the day, I'll just take the time to uh collect a bunch just take the time to uh collect a bunch just take the time to uh collect a bunch of feedback." So I'll uh you know, run of feedback." So I'll uh you know, run of feedback." So I'll uh you know, run it a few times, collect a bit of it a few times, collect a bit of it a few times, collect a bit of feedback, be like, "Hey, this was bad or feedback, be like, "Hey, this was bad or feedback, be like, "Hey, this was bad or this was good or this could have been this was good or this could have been this was good or this could have been better." And then just have it save JSON better." And then just have it save JSON better." And then just have it save JSON dumps on my downloads folder. So it'll dumps on my downloads folder. So it'll dumps on my downloads folder. So it'll save a dump of each of the runs with the save a dump of each of the runs with the save a dump of each of the runs with the system prompt and the inputs and my system prompt and the inputs and my system prompt and the inputs and my feedback. uh and you just collect a feedback. uh and you just collect a feedback. uh and you just collect a handful, you know, like even four, five, handful, you know, like even four, five, handful, you know, like even four, five, 10 are fine. If you can get some other 10 are fine. If you can get some other 10 are fine. If you can get some other uh co-workers to run it, that would be uh co-workers to run it, that would be uh co-workers to run it, that would be great. Collect like 20 or 30. great. Collect like 20 or 30. great. Collect like 20 or 30. And then similarly, uh at 5:00 p.m. when And then similarly, uh at 5:00 p.m. when And then similarly, uh at 5:00 p.m. when you're setting up an overnight run, uh you're setting up an overnight run, uh you're setting up an overnight run, uh just say, "Hey, like here are, you know, just say, "Hey, like here are, you know, just say, "Hey, like here are, you know, 10 20 30 feedback JSONs. This is all the 10 20 30 feedback JSONs. This is all the 10 20 30 feedback JSONs. This is all the information they have in them. Turn this information they have in them. Turn this information they have in them. Turn this into an eval set." Like just do it into an eval set." Like just do it into an eval set." Like just do it locally with SQLite or Markdown or locally with SQLite or Markdown or locally with SQLite or Markdown or whatever you want. And then over the whatever you want. And then over the whatever you want. And then over the next few minutes, talk me through how next few minutes, talk me through how next few minutes, talk me through how would you design evals or scoring would you design evals or scoring would you design evals or scoring functions to optimize for this and let's functions to optimize for this and let's functions to optimize for this and let's do it interactively. And then I want you do it interactively. And then I want you do it interactively. And then I want you to a build a harness that runs this call to a build a harness that runs this call to a build a harness that runs this call against the evals and hill climbs until against the evals and hill climbs until against the evals and hill climbs until the score goes up. Like just do whatever the score goes up. Like just do whatever the score goes up. Like just do whatever you need to do to get these scores to go you need to do to get these scores to go you need to do to get these scores to go up. And obviously with this kind of up. And obviously with this kind of up. And obviously with this kind of thing, if you have such little data, thing, if you have such little data, thing, if you have such little data, overfitting is a problem. But even that overfitting is a problem. But even that overfitting is a problem. But even that I found these modern models, you can I found these modern models, you can I found these modern models, you can just tell it to not overfit. And it just tell it to not overfit. And it just tell it to not overfit. And it actually does a pretty good job. Uh and actually does a pretty good job. Uh and actually does a pretty good job. Uh and to just say don't overfitit, just keep to just say don't overfitit, just keep to just say don't overfitit, just keep it general. uh run overnight it general. uh run overnight it general. uh run overnight autonomously until we align on a scoring autonomously until we align on a scoring autonomously until we align on a scoring rubric and then give me a full report uh rubric and then give me a full report uh rubric and then give me a full report uh in the morning. And then a little tip, in the morning. And then a little tip, in the morning. And then a little tip, it's also nice to just say, "Hey, save it's also nice to just say, "Hey, save it's also nice to just say, "Hey, save this flow as a generic skill so I can this flow as a generic skill so I can this flow as a generic skill so I can reuse it." Uh and that allows you to

reuse it." Uh and that allows you to reuse it." Uh and that allows you to improve this flow over time and improve improve this flow over time and improve improve this flow over time and improve that skill. And this works actually in that skill. And this works actually in that skill. And this works actually in the morning. You wake up to ideally the morning. You wake up to ideally the morning. You wake up to ideally something hill climbed. Uh and actually something hill climbed. Uh and actually something hill climbed. Uh and actually when using that feature when we we ship when using that feature when we we ship when using that feature when we we ship these features to employees to to these features to employees to to these features to employees to to further dog fooding to production it further dog fooding to production it further dog fooding to production it actually does improve the quality. You actually does improve the quality. You actually does improve the quality. You know these are not overfit. Um and we we know these are not overfit. Um and we we know these are not overfit. Um and we we use this flow quite a bit internally. Uh use this flow quite a bit internally. Uh use this flow quite a bit internally. Uh we have an internal code reviewer we have an internal code reviewer we have an internal code reviewer similar to uh some of the external ones. similar to uh some of the external ones. similar to uh some of the external ones. And same thing there we add that hill And same thing there we add that hill And same thing there we add that hill climb to pretty good quality just with climb to pretty good quality just with climb to pretty good quality just with these overnight runs. these overnight runs. these overnight runs. And then the last example uh and this is And then the last example uh and this is And then the last example uh and this is again pretty new and really interesting. again pretty new and really interesting. again pretty new and really interesting. uh you can train custom models with this uh you can train custom models with this uh you can train custom models with this mechanism overnight you know in one mechanism overnight you know in one mechanism overnight you know in one night. So uh this is an example of a night. So uh this is an example of a night. So uh this is an example of a modern BERT PII classifier we trained. modern BERT PII classifier we trained. modern BERT PII classifier we trained. We were trying Opus and Haiku for it and We were trying Opus and Haiku for it and We were trying Opus and Haiku for it and it was expensive and latency was not it was expensive and latency was not it was expensive and latency was not great and we just couldn't get the great and we just couldn't get the great and we just couldn't get the precision recall to be amazing. And so precision recall to be amazing. And so precision recall to be amazing. And so uh we collected a bunch of training data uh we collected a bunch of training data uh we collected a bunch of training data and then overnight one night I was like and then overnight one night I was like and then overnight one night I was like here is a bunch of business context and here is a bunch of business context and here is a bunch of business context and a bunch of training data we collected. a bunch of training data we collected. a bunch of training data we collected. uh a clean up the training data cloud uh a clean up the training data cloud uh a clean up the training data cloud just bolster it with synthetic data just bolster it with synthetic data just bolster it with synthetic data whatever you need to here's a bunch of whatever you need to here's a bunch of whatever you need to here's a bunch of open AI and enthropic keys uh use an open AI and enthropic keys uh use an open AI and enthropic keys uh use an ensemble of frontier models to push the ensemble of frontier models to push the ensemble of frontier models to push the quality quality quality I don't know anything about what ML I don't know anything about what ML I don't know anything about what ML model will be good here so like you model will be good here so like you model will be good here so like you decide you know just give me the best decide you know just give me the best decide you know just give me the best model class give me a few options and in model class give me a few options and in model class give me a few options and in fact train two like don't even train one fact train two like don't even train one fact train two like don't even train one train two separate ones uh and then I train two separate ones uh and then I train two separate ones uh and then I gave it AWS access I was like give me a gave it AWS access I was like give me a gave it AWS access I was like give me a provision sandbox do not give it prod provision sandbox do not give it prod provision sandbox do not give it prod that's not a good idea that's how you that's not a good idea that's how you that's not a good idea that's how you take prod down um pick the right GPU and take prod down um pick the right GPU and take prod down um pick the right GPU and EC2 cluster like train it, test against

EC2 cluster like train it, test against EC2 cluster like train it, test against eval examples, deprovision it, whatever eval examples, deprovision it, whatever eval examples, deprovision it, whatever you need to do. Just give me back the you need to do. Just give me back the you need to do. Just give me back the train models. Uh, and then give me a train models. Uh, and then give me a train models. Uh, and then give me a report in the morning. And in fact, even report in the morning. And in fact, even report in the morning. And in fact, even put in how I would host this with my put in how I would host this with my put in how I would host this with my codebase on inference once this model is codebase on inference once this model is codebase on inference once this model is ready. And again, ask me any clarifying ready. And again, ask me any clarifying ready. And again, ask me any clarifying questions. You can do it. I believe in questions. You can do it. I believe in questions. You can do it. I believe in you. You're going to do great. Uh, and you. You're going to do great. Uh, and you. You're going to do great. Uh, and in the morning, just have this ready. in the morning, just have this ready. in the morning, just have this ready. And of course, it works beautifully. In And of course, it works beautifully. In And of course, it works beautifully. In the morning, you have uh, two ML models the morning, you have uh, two ML models the morning, you have uh, two ML models trained, a full report, great results. trained, a full report, great results. trained, a full report, great results. Uh, and we've, uh, pushed a bunch of Uh, and we've, uh, pushed a bunch of Uh, and we've, uh, pushed a bunch of these to production. these to production. these to production. So those are three examples. Um I would So those are three examples. Um I would So those are three examples. Um I would try try your own. Uh but I think the try try your own. Uh but I think the try try your own. Uh but I think the main takeaway for me in doing these is main takeaway for me in doing these is main takeaway for me in doing these is really to push on task scope. So this really to push on task scope. So this really to push on task scope. So this was a tweet by Simon Lass, one of the was a tweet by Simon Lass, one of the was a tweet by Simon Lass, one of the notion founders. Uh and he he called out notion founders. Uh and he he called out notion founders. Uh and he he called out these modern models are just capable of these modern models are just capable of these modern models are just capable of so much more than we think. uh and I so much more than we think. uh and I so much more than we think. uh and I think it's our jobs as leaders to really think it's our jobs as leaders to really think it's our jobs as leaders to really understand the contours of how much we understand the contours of how much we understand the contours of how much we can push these models because it affects can push these models because it affects can push these models because it affects both how we lead and how we influence both how we lead and how we influence both how we lead and how we influence our teams but also how they fit into our our teams but also how they fit into our our teams but also how they fit into our products and what these our new products products and what these our new products products and what these our new products are capable of in terms of utility. So are capable of in terms of utility. So are capable of in terms of utility. So I'd really push you overnight try to I'd really push you overnight try to I'd really push you overnight try to think of how do I give it more and more think of how do I give it more and more think of how do I give it more and more uh scope so that we can do you know uh scope so that we can do you know uh scope so that we can do you know weeks of work months of work overnight weeks of work months of work overnight weeks of work months of work overnight and see what these models are really and see what these models are really and see what these models are really capable of.

capable of. capable of. I will say a caveat a lot of this works I will say a caveat a lot of this works I will say a caveat a lot of this works because of existing organizational because of existing organizational because of existing organizational scaffolding. So you'll have to build scaffolding. So you'll have to build scaffolding. So you'll have to build this up in your organization. You know, this up in your organization. You know, this up in your organization. You know, AI code reviewers, this really helps. AI code reviewers, this really helps. AI code reviewers, this really helps. You have an internal one we trained, but You have an internal one we trained, but You have an internal one we trained, but also just external ones, whatever you also just external ones, whatever you also just external ones, whatever you need. Um, agents.md hygiene, you know, need. Um, agents.md hygiene, you know, need. Um, agents.md hygiene, you know, cloud.md, agents.mmd, proper CI that you cloud.md, agents.mmd, proper CI that you cloud.md, agents.mmd, proper CI that you can trust. And then we have a bunch of can trust. And then we have a bunch of can trust. And then we have a bunch of other things like uh really other things like uh really other things like uh really sophisticated feature flags. Uh, we have sophisticated feature flags. Uh, we have sophisticated feature flags. Uh, we have a prototype branch that you can push to a prototype branch that you can push to a prototype branch that you can push to that goes to employees, but it doesn't that goes to employees, but it doesn't that goes to employees, but it doesn't go to production. Just layers like that. go to production. Just layers like that. go to production. Just layers like that. So you're not taking prod down basically So you're not taking prod down basically So you're not taking prod down basically as you're prototyping and showing these as you're prototyping and showing these as you're prototyping and showing these things off. Not amazing for the CTO to things off. Not amazing for the CTO to things off. Not amazing for the CTO to take prod down. take prod down. take prod down. I will say again code hygiene really I will say again code hygiene really I will say again code hygiene really matter really matters. I've been humbled matter really matters. I've been humbled matter really matters. I've been humbled a lot. You know my code has annoyed a lot. You know my code has annoyed a lot. You know my code has annoyed engineers. It's caused sevs. Uh coding engineers. It's caused sevs. Uh coding engineers. It's caused sevs. Uh coding agents are not perfect yet. So just uh agents are not perfect yet. So just uh agents are not perfect yet. So just uh your mileage may vary. you will get your mileage may vary. you will get your mileage may vary. you will get humbled, but I I still think it's worth humbled, but I I still think it's worth humbled, but I I still think it's worth it uh because you learn so much and you it uh because you learn so much and you it uh because you learn so much and you can model what taking on more and more can model what taking on more and more can model what taking on more and more scope even with a busy calendar looks scope even with a busy calendar looks scope even with a busy calendar looks like. Some tactical tips on hygiene. Uh like. Some tactical tips on hygiene. Uh like. Some tactical tips on hygiene. Uh I would test everything before the PR I would test everything before the PR I would test everything before the PR goes up. That morning slot, that's a lot goes up. That morning slot, that's a lot goes up. That morning slot, that's a lot of what I'm doing is just testing what of what I'm doing is just testing what of what I'm doing is just testing what the overnight run did. Uh even though the overnight run did. Uh even though the overnight run did. Uh even though there's computer use, etc., it's really there's computer use, etc., it's really there's computer use, etc., it's really important you test it yourself. Um small important you test it yourself. Um small important you test it yourself. Um small readable PRs are really helpful. I think readable PRs are really helpful. I think readable PRs are really helpful. I think they you're modeling to the rest of the they you're modeling to the rest of the they you're modeling to the rest of the team what good looks like. So, if you're team what good looks like. So, if you're team what good looks like. So, if you're putting up three 5,000 line PRs, uh, putting up three 5,000 line PRs, uh, putting up three 5,000 line PRs, uh, other engineers on the team are going to other engineers on the team are going to other engineers on the team are going to start doing that, and that's not start doing that, and that's not start doing that, and that's not amazing. So, uh, it's really important.

amazing. So, uh, it's really important. amazing. So, uh, it's really important. Your hygiene is really good because it's Your hygiene is really good because it's Your hygiene is really good because it's modeling for the rest of the team what modeling for the rest of the team what modeling for the rest of the team what good looks like. Um, and then, ooh, this good looks like. Um, and then, ooh, this good looks like. Um, and then, ooh, this is so tempting. It's so tempting to put is so tempting. It's so tempting to put is so tempting. It's so tempting to put other reviewers on code you haven't read other reviewers on code you haven't read other reviewers on code you haven't read yet. Don't do it. It's uh, mostly yet. Don't do it. It's uh, mostly yet. Don't do it. It's uh, mostly because you're going to look like an ass because you're going to look like an ass because you're going to look like an ass because there's going to be something in because there's going to be something in because there's going to be something in that code that is going to be so that code that is going to be so that code that is going to be so obviously bad and then some senior obviously bad and then some senior obviously bad and then some senior engineer is going to call you and be engineer is going to call you and be engineer is going to call you and be like, "Yo, why didn't you read this?" So like, "Yo, why didn't you read this?" So like, "Yo, why didn't you read this?" So uh I speak from experience. Read the uh I speak from experience. Read the uh I speak from experience. Read the code, review it properly before you add code, review it properly before you add code, review it properly before you add anybody else to it. anybody else to it. anybody else to it. So hopefully this convinces you. So hopefully this convinces you. So hopefully this convinces you. Building is part of the job now. Uh you Building is part of the job now. Uh you Building is part of the job now. Uh you can do it uh even with just one or two can do it uh even with just one or two can do it uh even with just one or two hours a day and you should uh you'll hours a day and you should uh you'll hours a day and you should uh you'll start to feel the models every 3 months start to feel the models every 3 months start to feel the models every 3 months as the new model families come out and as the new model families come out and as the new model families come out and you'll learn the skill on how to push you'll learn the skill on how to push you'll learn the skill on how to push scope as you are uh instructing them. scope as you are uh instructing them. scope as you are uh instructing them. And you'll be able to show the the the And you'll be able to show the the the And you'll be able to show the the the team and the org what's possible as team and the org what's possible as team and the org what's possible as well. you know, just because you as you well. you know, just because you as you well. you know, just because you as you have this capability, you'll have this capability, you'll have this capability, you'll automatically think of ideas on what to automatically think of ideas on what to automatically think of ideas on what to slot in for these overnight runs and slot in for these overnight runs and slot in for these overnight runs and then you can model for the team. Hey, then you can model for the team. Hey, then you can model for the team. Hey, this is what the future looks like. this is what the future looks like. this is what the future looks like. Thank you. Thank you. Thank you. [applause]