Three OpenAI Engineers Shipped A Million Lines. Your Ten-Hour Agent Run Starts Here.

Three engineers at OpenAI shipped an Three engineers at OpenAI shipped an internal product in about a tenth of the internal product in about a tenth of the internal product in about a tenth of the time it would have taken by hand. time it would have taken by hand. time it would have taken by hand. Roughly 1,500 poll requests. It was a Roughly 1,500 poll requests. It was a Roughly 1,500 poll requests. It was a codebase over a million lines by the codebase over a million lines by the codebase over a million lines by the time they were done. Not a line of it time they were done. Not a line of it time they were done. Not a line of it was typed by humans. A few days ago, I was typed by humans. A few days ago, I was typed by humans. A few days ago, I made a videos arguing that you should made a videos arguing that you should made a videos arguing that you should effectively keep your agents desk neat effectively keep your agents desk neat effectively keep your agents desk neat and tidy if you wanted it to do useful and tidy if you wanted it to do useful and tidy if you wanted it to do useful work. I'm going to continue that in this work. I'm going to continue that in this work. I'm going to continue that in this video, but I'm going to talk about the video, but I'm going to talk about the video, but I'm going to talk about the biggest, boldest things we do with biggest, boldest things we do with biggest, boldest things we do with agents today. the long running work we agents today. the long running work we agents today. the long running work we do maybe over multiple agent sessions do maybe over multiple agent sessions do maybe over multiple agent sessions over 6 hours 8 hours 10 hours more and over 6 hours 8 hours 10 hours more and over 6 hours 8 hours 10 hours more and I'm going to tell you the secret of how I'm going to tell you the secret of how I'm going to tell you the secret of how good engineers and good builders manage good engineers and good builders manage good engineers and good builders manage those sessions and how you can too and those sessions and how you can too and those sessions and how you can too and hint it's not just by keeping your hint it's not just by keeping your hint it's not just by keeping your metaphorical desk clean the set of files metaphorical desk clean the set of files metaphorical desk clean the set of files that you want to get all of this to work that you want to get all of this to work that you want to get all of this to work for you is over on the substack I'll for you is over on the substack I'll for you is over on the substack I'll tell you all about it at the end of the tell you all about it at the end of the tell you all about it at the end of the video but before I want to give you why video but before I want to give you why video but before I want to give you why it works and I want to give you the it works and I want to give you the it works and I want to give you the principle so that you understand principle so that you understand principle so that you understand regardless of how you use agents or regardless of how you use agents or regardless of how you use agents or whether you work with codeex or claude, whether you work with codeex or claude, whether you work with codeex or claude, how to actually do the work of shaping how to actually do the work of shaping how to actually do the work of shaping agent runs and accomplish ambitious agent runs and accomplish ambitious agent runs and accomplish ambitious projects. Three engineers at OpenAI projects. Three engineers at OpenAI projects. Three engineers at OpenAI shipped an internal product in about a shipped an internal product in about a shipped an internal product in about a tenth of the time it would have taken by tenth of the time it would have taken by tenth of the time it would have taken by hand. Now, that sounds great, but that's hand. Now, that sounds great, but that's hand. Now, that sounds great, but that's not the real story. Yes, it was a big not the real story. Yes, it was a big not the real story. Yes, it was a big project, roughly 1,500 poll requests. It project, roughly 1,500 poll requests. It project, roughly 1,500 poll requests. It was a codebase over a million lines by was a codebase over a million lines by was a codebase over a million lines by the time they were done. And yes, of the time they were done. And yes, of the time they were done. And yes, of course, not a line of it was typed by course, not a line of it was typed by course, not a line of it was typed by humans. Now individual codecs run that humans. Now individual codecs run that humans. Now individual codecs run that that team pursued ran for more than 6 that team pursued ran for more than 6 that team pursued ran for more than 6 hours if if not longer at a stretch and

hours if if not longer at a stretch and hours if if not longer at a stretch and as it was running the team had a context as it was running the team had a context as it was running the team had a context problem that anyone who's running agent problem that anyone who's running agent problem that anyone who's running agent is going to recognize. A giant is going to recognize. A giant is going to recognize. A giant instruction file that you give an agent instruction file that you give an agent instruction file that you give an agent for a run like that will crowd out the for a run like that will crowd out the for a run like that will crowd out the task and a huge manual for the project task and a huge manual for the project task and a huge manual for the project as a whole will turn into, in their as a whole will turn into, in their as a whole will turn into, in their words, a graveyard of stale rules. In words, a graveyard of stale rules. In words, a graveyard of stale rules. In other words, how do you give the agent other words, how do you give the agent other words, how do you give the agent direction so you can accomplish a giant direction so you can accomplish a giant direction so you can accomplish a giant giant project without causing that giant project without causing that giant project without causing that project in its own rules to overwhelm project in its own rules to overwhelm project in its own rules to overwhelm the agent? By the end of this video, the agent? By the end of this video, the agent? By the end of this video, you're going to know how to start one of you're going to know how to start one of you're going to know how to start one of these long projects before you know these long projects before you know these long projects before you know every step, how to change the direction every step, how to change the direction every step, how to change the direction without restarting, and how to make that without restarting, and how to make that without restarting, and how to make that correction reach the work the agent correction reach the work the agent correction reach the work the agent hasn't done yet, how to steer it as it hasn't done yet, how to steer it as it hasn't done yet, how to steer it as it goes. More capable agents increase the goes. More capable agents increase the goes. More capable agents increase the reach of your judgment. They also reach of your judgment. They also reach of your judgment. They also increase the cost of leaving an outdated increase the cost of leaving an outdated increase the cost of leaving an outdated judgment in charge. So this method works judgment in charge. So this method works judgment in charge. So this method works across claude across codecs and across across claude across codecs and across across claude across codecs and across ordinary long conversations you have ordinary long conversations you have ordinary long conversations you have with AI because the current state can with AI because the current state can with AI because the current state can live beyond any single session. I think live beyond any single session. I think live beyond any single session. I think this matters more than the headline this matters more than the headline this matters more than the headline context window number because it changes context window number because it changes context window number because it changes what you can safely put into motion.

what you can safely put into motion. what you can safely put into motion. Open AAI replaced that giant manual I Open AAI replaced that giant manual I Open AAI replaced that giant manual I told you about with a very short map. told you about with a very short map. told you about with a very short map. The map pointed codecs toward active The map pointed codecs toward active The map pointed codecs toward active execution plans, toward decision logs, execution plans, toward decision logs, execution plans, toward decision logs, toward design documents, toward an toward design documents, toward an toward design documents, toward an architecture map and quality grades for architecture map and quality grades for architecture map and quality grades for every part of the codebase it was every part of the codebase it was every part of the codebase it was touching. As the product evolved, those touching. As the product evolved, those touching. As the product evolved, those materials changed along the way. So, materials changed along the way. So, materials changed along the way. So, they formed an accurate map of the they formed an accurate map of the they formed an accurate map of the current state of the execution as the current state of the execution as the current state of the execution as the giant codebase rebuild progressed. So, giant codebase rebuild progressed. So, giant codebase rebuild progressed. So, the agent didn't need every old the agent didn't need every old the agent didn't need every old instruction competing for attention. It instruction competing for attention. It instruction competing for attention. It needed a reliable way to find the best needed a reliable way to find the best needed a reliable way to find the best current information for the next piece current information for the next piece current information for the next piece of work. Anthropic has arrived at a of work. Anthropic has arrived at a of work. Anthropic has arrived at a similar practice with Claude. In its similar practice with Claude. In its similar practice with Claude. In its work on longunning scientific computing, work on longunning scientific computing, work on longunning scientific computing, Claude code uses a progress file as Claude code uses a progress file as Claude code uses a progress file as portable memory between sessions. That portable memory between sessions. That portable memory between sessions. That file records the current state. It file records the current state. It file records the current state. It records completed work. It records known records completed work. It records known records completed work. It records known limitations and failed approaches with limitations and failed approaches with limitations and failed approaches with the reason they failed. So a new cloud the reason they failed. So a new cloud the reason they failed. So a new cloud session can read it, pick up the next session can read it, pick up the next session can read it, pick up the next task, and avoid walking into the same task, and avoid walking into the same task, and avoid walking into the same dead end on a giant project. Enthropic dead end on a giant project. Enthropic dead end on a giant project. Enthropic uses the same basic pattern in its long uses the same basic pattern in its long uses the same basic pattern in its long running agent coding harness, a progress running agent coding harness, a progress running agent coding harness, a progress file, structured handoffs, and version file, structured handoffs, and version file, structured handoffs, and version history that let fresh sessions continue history that let fresh sessions continue history that let fresh sessions continue that job. These are different companies.

that job. These are different companies. that job. These are different companies. They're different systems, but they're They're different systems, but they're They're different systems, but they're responding to the same change. AI is responding to the same change. AI is responding to the same change. AI is moving from producing an answer to moving from producing an answer to moving from producing an answer to carrying work that develops over hours carrying work that develops over hours carrying work that develops over hours and days and many many separate runs. and days and many many separate runs. and days and many many separate runs. Once that happens, context can't remain Once that happens, context can't remain Once that happens, context can't remain sort of like a frozen packet from the sort of like a frozen packet from the sort of like a frozen packet from the beginning of the job that you send out beginning of the job that you send out beginning of the job that you send out in a rocket. It doesn't work that way. in a rocket. It doesn't work that way. in a rocket. It doesn't work that way. It has to change as the work teaches you It has to change as the work teaches you It has to change as the work teaches you what the job actually is. The stakes are what the job actually is. The stakes are what the job actually is. The stakes are very high, right? A longunning agent can very high, right? A longunning agent can very high, right? A longunning agent can take a useful direction much, much take a useful direction much, much take a useful direction much, much farther than a chatbot would. It can farther than a chatbot would. It can farther than a chatbot would. It can also take a slightly wrong direction also take a slightly wrong direction also take a slightly wrong direction through 6 hours of research and code and through 6 hours of research and code and through 6 hours of research and code and analysis and drafts before you realize analysis and drafts before you realize analysis and drafts before you realize how far off base you are. So I call this how far off base you are. So I call this how far off base you are. So I call this practice progressive context shaping. practice progressive context shaping. practice progressive context shaping. You begin with a clear brief, sure, but You begin with a clear brief, sure, but You begin with a clear brief, sure, but then you change the small set of then you change the small set of then you change the small set of instructions and decisions that should instructions and decisions that should instructions and decisions that should govern what happens next. As the work govern what happens next. As the work govern what happens next. As the work continues to produce evidence along the continues to produce evidence along the continues to produce evidence along the way, that small current state receives way, that small current state receives way, that small current state receives priority over everything that happened priority over everything that happened priority over everything that happened along the way. Good prompt engineering along the way. Good prompt engineering along the way. Good prompt engineering absolutely still matters. A strong absolutely still matters. A strong absolutely still matters. A strong opening prompt should define the result opening prompt should define the result opening prompt should define the result that you want. It should identify the that you want. It should identify the that you want. It should identify the materials the agent can use. It should materials the agent can use. It should materials the agent can use. It should set the boundaries on what it may do and set the boundaries on what it may do and set the boundaries on what it may do and may not do and give it a very sensible may not do and give it a very sensible may not do and give it a very sensible first move to try. If you know what first move to try. If you know what first move to try. If you know what quality looks like, you should say so in quality looks like, you should say so in quality looks like, you should say so in the instruction set. If publication or the instruction set. If publication or the instruction set. If publication or spending or deletion or contact with spending or deletion or contact with spending or deletion or contact with another person requires approval, that's another person requires approval, that's another person requires approval, that's another thing you should write down and

another thing you should write down and another thing you should write down and say. The opening prompt gives the agent say. The opening prompt gives the agent say. The opening prompt gives the agent enough direction to begin useful work enough direction to begin useful work enough direction to begin useful work and puts those basic guardrails in and puts those basic guardrails in and puts those basic guardrails in place. But the opening prompt is written place. But the opening prompt is written place. But the opening prompt is written before the work begins. It can only before the work begins. It can only before the work begins. It can only express what you know at that moment. A express what you know at that moment. A express what you know at that moment. A serious project produces information serious project produces information serious project produces information along the way. The research may weaken along the way. The research may weaken along the way. The research may weaken your original thesis. The first your original thesis. The first your original thesis. The first implementation may expose a dependency implementation may expose a dependency implementation may expose a dependency that changes how the architecture works. that changes how the architecture works. that changes how the architecture works. Right? The customer evidence points Right? The customer evidence points Right? The customer evidence points toward a different use case. Potentially toward a different use case. Potentially toward a different use case. Potentially the first 50 results that you get could the first 50 results that you get could the first 50 results that you get could reveal that your original request will reveal that your original request will reveal that your original request will produce a lot of volume and very little produce a lot of volume and very little produce a lot of volume and very little value. So maybe this whole thing is not value. So maybe this whole thing is not value. So maybe this whole thing is not what you wanted it to be. So, at this what you wanted it to be. So, at this what you wanted it to be. So, at this point, the job is not to defend your point, the job is not to defend your point, the job is not to defend your original prompt or to sort of push original prompt or to sort of push original prompt or to sort of push through in a bullheaded way. The job is through in a bullheaded way. The job is through in a bullheaded way. The job is to change what the agent treats as the to change what the agent treats as the to change what the agent treats as the current version of the assignment. That current version of the assignment. That current version of the assignment. That is where prompting becomes less like is where prompting becomes less like is where prompting becomes less like placing an order and more like directing placing an order and more like directing placing an order and more like directing work. I found myself doing this across a work. I found myself doing this across a work. I found myself doing this across a wide range of tools. Sometimes I wide range of tools. Sometimes I wide range of tools. Sometimes I interrupt Claude code while it is interrupt Claude code while it is interrupt Claude code while it is steering. I steer codecs after reviewing steering. I steer codecs after reviewing steering. I steer codecs after reviewing a result or let a project develop a result or let a project develop a result or let a project develop through several turns in claude or chat through several turns in claude or chat through several turns in claude or chat GPT. I've kept the changing state in a GPT. I've kept the changing state in a GPT. I've kept the changing state in a current markdown file, in a plan, in a current markdown file, in a plan, in a current markdown file, in a plan, in a JSON, in an issue tracker, and in a JSON, in an issue tracker, and in a JSON, in an issue tracker, and in a carefully updated conversation. So, I've carefully updated conversation. So, I've carefully updated conversation. So, I've tried a bunch of different approaches, tried a bunch of different approaches, tried a bunch of different approaches, right? The specific container here right? The specific container here right? The specific container here matters much, much less than whether the

matters much, much less than whether the matters much, much less than whether the next action gets an updated state on the next action gets an updated state on the next action gets an updated state on the run. One of the clearest public examples run. One of the clearest public examples run. One of the clearest public examples we have of this comes from ARISE. Its we have of this comes from ARISE. Its we have of this comes from ARISE. Its agent, Alex, was asked to summarize agent, Alex, was asked to summarize agent, Alex, was asked to summarize multiple traces. During one memorable multiple traces. During one memorable multiple traces. During one memorable run, the agent made 27 model calls. run, the agent made 27 model calls. run, the agent made 27 model calls. Almost all of them went into Almost all of them went into Almost all of them went into reorganizing its own to-do list. It kept reorganizing its own to-do list. It kept reorganizing its own to-do list. It kept moving information around without moving information around without moving information around without completing the assignment. Sometimes completing the assignment. Sometimes completing the assignment. Sometimes that does sound like AI, right? The that does sound like AI, right? The that does sound like AI, right? The original request had been buried under original request had been buried under original request had been buried under tool output and under intermediate tool output and under intermediate tool output and under intermediate results and the agents own activity and results and the agents own activity and results and the agents own activity and and Arise didn't solve that by adding a and Arise didn't solve that by adding a and Arise didn't solve that by adding a more forceful sentence at the top of the more forceful sentence at the top of the more forceful sentence at the top of the prompt. The team decided to solve that prompt. The team decided to solve that prompt. The team decided to solve that by moving the current plan entirely by moving the current plan entirely by moving the current plan entirely outside the conversation window and outside the conversation window and outside the conversation window and storing it on disk. On every model call, storing it on disk. On every model call, storing it on disk. On every model call, the system rebuilt a short plan message the system rebuilt a short plan message the system rebuilt a short plan message from the latest state and placed it in from the latest state and placed it in from the latest state and placed it in front of the noisy history. The agent front of the noisy history. The agent front of the noisy history. The agent could see what was done, what was in could see what was done, what was in could see what was done, what was in progress, what came next. Doesn't that progress, what came next. Doesn't that progress, what came next. Doesn't that sound nice if you're the agent? Aris's sound nice if you're the agent? Aris's sound nice if you're the agent? Aris's example separates transcript, the example separates transcript, the example separates transcript, the transcript of the work so far from transcript of the work so far from transcript of the work so far from current context. That's really current context. That's really current context. That's really important. It's like putting the road important. It's like putting the road important. It's like putting the road safety signs up front. The transcript safety signs up front. The transcript safety signs up front. The transcript still held real information, but the still held real information, but the still held real information, but the maintained plan received much more maintained plan received much more maintained plan received much more authority than the pile of events that authority than the pile of events that authority than the pile of events that happened along the way. Arise gave the happened along the way. Arise gave the happened along the way. Arise gave the agent an updated answer to a very agent an updated answer to a very agent an updated answer to a very practical question.

practical question. practical question. Given everything that has happened, what Given everything that has happened, what Given everything that has happened, what should I be doing next? What should I be should I be doing next? What should I be should I be doing next? What should I be doing now? That is close to how I use doing now? That is close to how I use doing now? That is close to how I use current.mmarkdown in a personal current.mmarkdown in a personal current.mmarkdown in a personal workflow. It it's an ordinary file, workflow. It it's an ordinary file, workflow. It it's an ordinary file, right? It's not a claud or codeex right? It's not a claud or codeex right? It's not a claud or codeex feature. Anybody can use it and you and feature. Anybody can use it and you and feature. Anybody can use it and you and the agent just need to agree to keep it the agent just need to agree to keep it the agent just need to agree to keep it current. In claude code, current. In claude code, current. In claude code, claude.mmarkdown can tell claude to read claude.mmarkdown can tell claude to read claude.mmarkdown can tell claude to read it at the start of the job and update it it at the start of the job and update it it at the start of the job and update it after every consequential decision. In after every consequential decision. In after every consequential decision. In codeex, agents.mmarkdown or the task codeex, agents.mmarkdown or the task codeex, agents.mmarkdown or the task itself can give the same instruction. In itself can give the same instruction. In itself can give the same instruction. In a normal chat, you can attach or paste a normal chat, you can attach or paste a normal chat, you can attach or paste the current state when you begin a new the current state when you begin a new the current state when you begin a new conversation. In an ordinary claude or conversation. In an ordinary claude or conversation. In an ordinary claude or chat GPT conversation, the same practice chat GPT conversation, the same practice chat GPT conversation, the same practice takes on a couple of different forms. takes on a couple of different forms. takes on a couple of different forms. While the thread is still useful, you While the thread is still useful, you While the thread is still useful, you can steer it directly. And when a can steer it directly. And when a can steer it directly. And when a decision s needs to survive, when it decision s needs to survive, when it decision s needs to survive, when it needs to transcend the current chat needs to transcend the current chat needs to transcend the current chat you're in, uh, you just ask for a you're in, uh, you just ask for a you're in, uh, you just ask for a replacement project brief that reflects replacement project brief that reflects replacement project brief that reflects the new state and you read it and you the new state and you read it and you the new state and you read it and you correct it and you carry that brief into correct it and you carry that brief into correct it and you carry that brief into your next conversation with the old your next conversation with the old your next conversation with the old thread becomes history and the brief thread becomes history and the brief thread becomes history and the brief says what's true today, what's true now.

says what's true today, what's true now. says what's true today, what's true now. You don't need to preserve every You don't need to preserve every You don't need to preserve every correction this way. You want to save correction this way. You want to save correction this way. You want to save the change only when it affects what the change only when it affects what the change only when it affects what you're going for, your your target for you're going for, your your target for you're going for, your your target for this run, your goal, the definition of this run, your goal, the definition of this run, your goal, the definition of done. If you would be annoyed when a done. If you would be annoyed when a done. If you would be annoyed when a fresh agent misses the decision that you fresh agent misses the decision that you fresh agent misses the decision that you would want to make, then you should put would want to make, then you should put would want to make, then you should put it in the project state or you should it in the project state or you should it in the project state or you should have the agent put it in the project have the agent put it in the project have the agent put it in the project state. On the other hand, if it only state. On the other hand, if it only state. On the other hand, if it only helps explain how you got there, it helps explain how you got there, it helps explain how you got there, it probably doesn't belong in that file. It probably doesn't belong in that file. It probably doesn't belong in that file. It probably is in just the history and you probably is in just the history and you probably is in just the history and you cannot mess with it a lot. Other coding cannot mess with it a lot. Other coding cannot mess with it a lot. Other coding agents expose project rules or agents expose project rules or agents expose project rules or instruction files that can point to the instruction files that can point to the instruction files that can point to the same kind of state while a multi- aent same kind of state while a multi- aent same kind of state while a multi- aent system might use a ticket or a JSON system might use a ticket or a JSON system might use a ticket or a JSON record or a database. OpenAI Symphony record or a database. OpenAI Symphony record or a database. OpenAI Symphony system uses a project board to define system uses a project board to define system uses a project board to define and track real work. Whether the and track real work. Whether the and track real work. Whether the mechanism is markdown or an issue in mechanism is markdown or an issue in mechanism is markdown or an issue in linear, the next agent action has to linear, the next agent action has to linear, the next agent action has to read the updated decision. I learned the read the updated decision. I learned the read the updated decision. I learned the value of this in a long codeex project value of this in a long codeex project value of this in a long codeex project of my own. I was building a benchmark of my own. I was building a benchmark of my own. I was building a benchmark across a really large set of my own across a really large set of my own across a really large set of my own work. So the run discovered 339 work. So the run discovered 339 work. So the run discovered 339 different sources and it generated over different sources and it generated over different sources and it generated over a thousand questions and it produced 250 a thousand questions and it produced 250 a thousand questions and it produced 250 verified answers and that was real verified answers and that was real verified answers and that was real progress. And then the delegated agent progress. And then the delegated agent progress. And then the delegated agent began repeatedly continuing itself began repeatedly continuing itself began repeatedly continuing itself during package synchronization. In other during package synchronization. In other during package synchronization. In other words, it was still active but it words, it was still active but it words, it was still active but it stopped adding useful value. The stopped adding useful value. The stopped adding useful value. The original objective had encouraged original objective had encouraged original objective had encouraged continuous stress testing and early in continuous stress testing and early in continuous stress testing and early in the run that instruction helped the the run that instruction helped the the run that instruction helped the agent to kind of keep going. But at some

agent to kind of keep going. But at some agent to kind of keep going. But at some point it became the wrong instruction. point it became the wrong instruction. point it became the wrong instruction. The useful next move was no longer to The useful next move was no longer to The useful next move was no longer to generate more questions or keep the generate more questions or keep the generate more questions or keep the continuation loop alive. It was actually continuation loop alive. It was actually continuation loop alive. It was actually to deepen the evidence behind the most to deepen the evidence behind the most to deepen the evidence behind the most valuable answers we discovered to valuable answers we discovered to valuable answers we discovered to questions along the way and then to questions along the way and then to questions along the way and then to stop. So I stopped the run and I changed stop. So I stopped the run and I changed stop. So I stopped the run and I changed the state. The saved record said the the state. The saved record said the the state. The saved record said the project was checkpointed after a retry project was checkpointed after a retry project was checkpointed after a retry loop and prohibited resuming the loop and prohibited resuming the loop and prohibited resuming the unbounded run. And the next pass would unbounded run. And the next pass would unbounded run. And the next pass would select the 50 highest value answers to select the 50 highest value answers to select the 50 highest value answers to those questions, replace generic those questions, replace generic those questions, replace generic evidence with evidence specific to each evidence with evidence specific to each evidence with evidence specific to each question, validate the package and stop. question, validate the package and stop. question, validate the package and stop. All completed work remained available All completed work remained available All completed work remained available under a new direction. That is under a new direction. That is under a new direction. That is progressive context shaping in a real progressive context shaping in a real progressive context shaping in a real project. Because what I found was I was project. Because what I found was I was project. Because what I found was I was getting stuck because the old getting stuck because the old getting stuck because the old instruction was essentially pushing the instruction was essentially pushing the instruction was essentially pushing the agent too far in a direction that was agent too far in a direction that was agent too far in a direction that was unhelpful for ultimately getting my goal unhelpful for ultimately getting my goal unhelpful for ultimately getting my goal accomplished. The next run needed to do accomplished. The next run needed to do accomplished. The next run needed to do different work. So I changed the agent different work. So I changed the agent different work. So I changed the agent state it would inherit and turned an state it would inherit and turned an state it would inherit and turned an open-ended generation job into a very open-ended generation job into a very open-ended generation job into a very bounded job that was supposed to finish bounded job that was supposed to finish bounded job that was supposed to finish up. So then I got 1,000 questions, 250 up. So then I got 1,000 questions, 250 up. So then I got 1,000 questions, 250 answers and a source map that remained answers and a source map that remained answers and a source map that remained useful. So at the end of it, I actually useful. So at the end of it, I actually useful. So at the end of it, I actually got the questions and answers that got the questions and answers that got the questions and answers that mattered the most to me and I got real mattered the most to me and I got real mattered the most to me and I got real value. Anthropic scientific computing value. Anthropic scientific computing value. Anthropic scientific computing example shows the same move at a smaller example shows the same move at a smaller example shows the same move at a smaller scale. In one solver project, the scale. In one solver project, the scale. In one solver project, the progress notes record that an attempted

progress notes record that an attempted progress notes record that an attempted method was too stiff and name the method method was too stiff and name the method method was too stiff and name the method that replaced it. A new claude session that replaced it. A new claude session that replaced it. A new claude session doesn't need to relive the entire failed doesn't need to relive the entire failed doesn't need to relive the entire failed attempt. It can see the failure and move attempt. It can see the failure and move attempt. It can see the failure and move on. It just needs the consequence. on. It just needs the consequence. on. It just needs the consequence. Right? This approach failed for this Right? This approach failed for this Right? This approach failed for this reason. Use this replacement. Continue reason. Use this replacement. Continue reason. Use this replacement. Continue from the current state. This is what from the current state. This is what from the current state. This is what helpful forgetting can look like for helpful forgetting can look like for helpful forgetting can look like for agents. The system doesn't erase the agents. The system doesn't erase the agents. The system doesn't erase the evidence of the old attempt. That the evidence of the old attempt. That the evidence of the old attempt. That the history can remain in a change log history can remain in a change log history can remain in a change log somewhere, but the failed method doesn't somewhere, but the failed method doesn't somewhere, but the failed method doesn't have the same position as the active have the same position as the active have the same position as the active plan. the agent carries the lesson plan. the agent carries the lesson plan. the agent carries the lesson forward without carrying every single forward without carrying every single forward without carrying every single token that produced the lesson and kind token that produced the lesson and kind token that produced the lesson and kind of getting confused along the way. Now, of getting confused along the way. Now, of getting confused along the way. Now, there are four kinds of context worth there are four kinds of context worth there are four kinds of context worth separating here and I'm going to go separating here and I'm going to go separating here and I'm going to go through all four. The first is stable through all four. The first is stable through all four. The first is stable instruction. How to work, where to find instruction. How to work, where to find instruction. How to work, where to find things, what actions require approval, things, what actions require approval, things, what actions require approval, and what standards should remain in and what standards should remain in and what standards should remain in force. Claude might read that from force. Claude might read that from force. Claude might read that from claw.markdown. Codeex might read it from claw.markdown. Codeex might read it from claw.markdown. Codeex might read it from an agents.mmarkdown file. It's a an agents.mmarkdown file. It's a an agents.mmarkdown file. It's a markdown file, right? Another tool might markdown file, right? Another tool might markdown file, right? Another tool might call it project rules. You get the idea.

call it project rules. You get the idea. call it project rules. You get the idea. The second is current project state. The second is current project state. The second is current project state. What the goal means now, which decisions What the goal means now, which decisions What the goal means now, which decisions are active, what remains unresolved, are active, what remains unresolved, are active, what remains unresolved, what should happen next, and when the what should happen next, and when the what should happen next, and when the agent really needs to stop. This is agent really needs to stop. This is agent really needs to stop. This is often changing material and it may live often changing material and it may live often changing material and it may live in again a markdown file, a ticket, a in again a markdown file, a ticket, a in again a markdown file, a ticket, a structured data, but it's different. structured data, but it's different. structured data, but it's different. It's different from the actions that It's different from the actions that It's different from the actions that require approval. It's different from require approval. It's different from require approval. It's different from the guardrail stuff. It's really about the guardrail stuff. It's really about the guardrail stuff. It's really about the condition of the goal. The third is the condition of the goal. The third is the condition of the goal. The third is the map. What material exists and where the map. What material exists and where the map. What material exists and where it lives? How does the agent get it lives? How does the agent get it lives? How does the agent get resources? Research files, design resources? Research files, design resources? Research files, design documents, transcripts, prior drafts. documents, transcripts, prior drafts. documents, transcripts, prior drafts. The agent doesn't need all of it in The agent doesn't need all of it in The agent doesn't need all of it in front of it right now. It needs a front of it right now. It needs a front of it right now. It needs a reliable way to find the bit that reliable way to find the bit that reliable way to find the bit that matters for the next decision. Most matters for the next decision. Most matters for the next decision. Most people will skip this layer. And people will skip this layer. And people will skip this layer. And skipping it is why they end up just kind skipping it is why they end up just kind skipping it is why they end up just kind of pasting everything in. Have you ever of pasting everything in. Have you ever of pasting everything in. Have you ever done that? The fourth is history. What done that? The fourth is history. What done that? The fourth is history. What happened before, what changed, why a happened before, what changed, why a happened before, what changed, why a decision was made, and how to recover an decision was made, and how to recover an decision was made, and how to recover an earlier version. So, get or a change log earlier version. So, get or a change log earlier version. So, get or a change log or a decision log. Um, transcripts, or a decision log. Um, transcripts, or a decision log. Um, transcripts, they're all useful history, and history they're all useful history, and history they're all useful history, and history matters, but we want history to not matters, but we want history to not matters, but we want history to not masquerade as current instructions. So, masquerade as current instructions. So, masquerade as current instructions. So, we keep it as a separate set of data we keep it as a separate set of data we keep it as a separate set of data that the agent can access instead.

that the agent can access instead. that the agent can access instead. OpenAI used exactly this separation in OpenAI used exactly this separation in OpenAI used exactly this separation in that millionline project I talked about that millionline project I talked about that millionline project I talked about at the beginning of this video. A short at the beginning of this video. A short at the beginning of this video. A short agents.mmarkdown map the repository agents.mmarkdown map the repository agents.mmarkdown map the repository active execution plans carried progress active execution plans carried progress active execution plans carried progress and decision logs and design and and decision logs and design and and decision logs and design and architecture documents held deeper architecture documents held deeper architecture documents held deeper knowledge. The team even ran recurring knowledge. The team even ran recurring knowledge. The team even ran recurring work to find stale documentation and work to find stale documentation and work to find stale documentation and update it along the way. And so instead update it along the way. And so instead update it along the way. And so instead of squeezing a million line project into of squeezing a million line project into of squeezing a million line project into a single prompt, they gave the agent a a single prompt, they gave the agent a a single prompt, they gave the agent a way to locate current knowledge as the way to locate current knowledge as the way to locate current knowledge as the product evolved. It's also what product evolved. It's also what product evolved. It's also what Enthropic did for fresh cloud sessions. Enthropic did for fresh cloud sessions. Enthropic did for fresh cloud sessions. An initializer set up the working An initializer set up the working An initializer set up the working environment in the progress file. Later environment in the progress file. Later environment in the progress file. Later sessions read the progress and inspected sessions read the progress and inspected sessions read the progress and inspected the version history and selected the version history and selected the version history and selected unfinished work and made incremental unfinished work and made incremental unfinished work and made incremental change and left an updated state for change and left an updated state for change and left an updated state for whoever comes next. That's just good whoever comes next. That's just good whoever comes next. That's just good agent hygiene. 4 months later, Enthropic agent hygiene. 4 months later, Enthropic agent hygiene. 4 months later, Enthropic rebuilt that harness. They dropped the rebuilt that harness. They dropped the rebuilt that harness. They dropped the sprint scaffolding entirely because the sprint scaffolding entirely because the sprint scaffolding entirely because the newer model no longer needed the work newer model no longer needed the work newer model no longer needed the work broken into chunks and they kept the broken into chunks and they kept the broken into chunks and they kept the structured handoff between sessions structured handoff between sessions structured handoff between sessions because that still needed to happen. So because that still needed to happen. So because that still needed to happen. So the scaffolding ended up being the scaffolding ended up being the scaffolding ended up being disposable as agents evolve, but the disposable as agents evolve, but the disposable as agents evolve, but the current state was not. The handoff is current state was not. The handoff is current state was not. The handoff is part of the work because a long project part of the work because a long project part of the work because a long project no matter how what sort of big agents no matter how what sort of big agents no matter how what sort of big agents evolved to be can't depend on one evolved to be can't depend on one evolved to be can't depend on one conversation staying alive forever. Now conversation staying alive forever. Now conversation staying alive forever. Now anthropics research across roughly anthropics research across roughly anthropics research across roughly 400,000 claude code sessions makes the 400,000 claude code sessions makes the 400,000 claude code sessions makes the human role in all of this much clearer.

human role in all of this much clearer. human role in all of this much clearer. In a typical session, the prompter, the In a typical session, the prompter, the In a typical session, the prompter, the person in charge, made about 70% of the person in charge, made about 70% of the person in charge, made about 70% of the planning decisions, while Claude made planning decisions, while Claude made planning decisions, while Claude made about 80% of the execution decisions. about 80% of the execution decisions. about 80% of the execution decisions. That result is specific to claude code, That result is specific to claude code, That result is specific to claude code, but I think the division is a useful but I think the division is a useful but I think the division is a useful rule of thumb. The person doesn't have rule of thumb. The person doesn't have rule of thumb. The person doesn't have to narrate every file change or command to narrate every file change or command to narrate every file change or command these days. And anyone who uses agents these days. And anyone who uses agents these days. And anyone who uses agents will tell you they they generally don't. will tell you they they generally don't. will tell you they they generally don't. Otherwise, why would you do that? The Otherwise, why would you do that? The Otherwise, why would you do that? The person decides what ought to be built, person decides what ought to be built, person decides what ought to be built, what the new evidence means, what counts what the new evidence means, what counts what the new evidence means, what counts as done. They steer the agent, and the as done. They steer the agent, and the as done. They steer the agent, and the agent handles much of the path between agent handles much of the path between agent handles much of the path between those decisions. Progressive context those decisions. Progressive context those decisions. Progressive context shaping is how those planning decisions shaping is how those planning decisions shaping is how those planning decisions survive between check-ins. If you survive between check-ins. If you survive between check-ins. If you correct only the draft in front of you, correct only the draft in front of you, correct only the draft in front of you, the correction may just stop there. If the correction may just stop there. If the correction may just stop there. If you change the current project state, you change the current project state, you change the current project state, the correction can alter the remaining the correction can alter the remaining the correction can alter the remaining research and the next implementation and research and the next implementation and research and the next implementation and the work given to another agent and the the work given to another agent and the the work given to another agent and the task resumed tomorrow. Keeping current task resumed tomorrow. Keeping current task resumed tomorrow. Keeping current agent state becomes more important as agent state becomes more important as agent state becomes more important as people run several agents at once. people run several agents at once. people run several agents at once. OpenAI learned this when it discovered OpenAI learned this when it discovered OpenAI learned this when it discovered its engineers could comfortably manage its engineers could comfortably manage its engineers could comfortably manage only three to five codec sessions at only three to five codec sessions at only three to five codec sessions at once before context switching became once before context switching became once before context switching became painful. People forgot which session was painful. People forgot which session was painful. People forgot which session was doing what and spent their time moving doing what and spent their time moving doing what and spent their time moving between terminals. Symphony moved the between terminals. Symphony moved the between terminals. Symphony moved the work into a project board where agents work into a project board where agents work into a project board where agents pulled tickets. And I've talked about pulled tickets. And I've talked about pulled tickets. And I've talked about that before. On some teams, landed pull that before. On some teams, landed pull that before. On some teams, landed pull requests rose 500% during the first 3 requests rose 500% during the first 3 requests rose 500% during the first 3 weeks after Symphony. The operating weeks after Symphony. The operating weeks after Symphony. The operating change is something that we have to kind change is something that we have to kind change is something that we have to kind of wrap our heads around because of wrap our heads around because of wrap our heads around because essentially what it did was instead of essentially what it did was instead of essentially what it did was instead of holding every session in their head,

holding every session in their head, holding every session in their head, people use the ticket to carry the people use the ticket to carry the people use the ticket to carry the deliverable, the status, the latest deliverable, the status, the latest deliverable, the status, the latest direction. Does this sound familiar? direction. Does this sound familiar? direction. Does this sound familiar? They're basically carrying current state They're basically carrying current state They're basically carrying current state for the agent much more easily in a for the agent much more easily in a for the agent much more easily in a ticket board and they can manage many ticket board and they can manage many ticket board and they can manage many more agents at once and they could more agents at once and they could more agents at once and they could change the work where the next agent change the work where the next agent change the work where the next agent would find it by just changing the would find it by just changing the would find it by just changing the ticket. You can use the same idea ticket. You can use the same idea ticket. You can use the same idea without committing to Symphony. You can without committing to Symphony. You can without committing to Symphony. You can start with one project whose path is start with one project whose path is start with one project whose path is uncertain but whose outcome matters. You uncertain but whose outcome matters. You uncertain but whose outcome matters. You can ask the agent for an early result can ask the agent for an early result can ask the agent for an early result that can teach you something like a that can teach you something like a that can teach you something like a research map or maybe a first MVP research map or maybe a first MVP research map or maybe a first MVP implementation or something, a diagnosis implementation or something, a diagnosis implementation or something, a diagnosis of the problem, something like that. of the problem, something like that. of the problem, something like that. Don't ask for the entire project in one Don't ask for the entire project in one Don't ask for the entire project in one blind run. If the early work is likely blind run. If the early work is likely blind run. If the early work is likely to change your judgment, ask for that to change your judgment, ask for that to change your judgment, ask for that checkpoint first. At the start, give the checkpoint first. At the start, give the checkpoint first. At the start, give the agent four things. the outcome, the agent four things. the outcome, the agent four things. the outcome, the relevant source material, the action relevant source material, the action relevant source material, the action boundaries, and the first useful boundaries, and the first useful boundaries, and the first useful checkpoint along the way. That checkpoint along the way. That checkpoint along the way. That checkpoint should contain work that you checkpoint should contain work that you checkpoint should contain work that you can have an opinion about, right? Think can have an opinion about, right? Think can have an opinion about, right? Think about the problem and report back is a about the problem and report back is a about the problem and report back is a very weak thing to ask, right? Instead, very weak thing to ask, right? Instead, very weak thing to ask, right? Instead, say something like, "Compare the say something like, "Compare the say something like, "Compare the sources, identify where they disagree, sources, identify where they disagree, sources, identify where they disagree, and propose the first version of the and propose the first version of the and propose the first version of the plan that you think would actually plan that you think would actually plan that you think would actually satisfy my requirements." That gives you satisfy my requirements." That gives you satisfy my requirements." That gives you something that you can wrestle with, something that you can wrestle with, something that you can wrestle with, right? get your teeth into and then right? get your teeth into and then right? get your teeth into and then create a very small current state record create a very small current state record create a very small current state record along the way. It doesn't need to be along the way. It doesn't need to be along the way. It doesn't need to be elaborate. It should say what the elaborate. It should say what the elaborate. It should say what the project is trying to accomplish, which project is trying to accomplish, which project is trying to accomplish, which decisions currently govern the work, decisions currently govern the work, decisions currently govern the work, which questions remain open, and what

which questions remain open, and what which questions remain open, and what the agent ought to do next and what the agent ought to do next and what the agent ought to do next and what condition should make it stop or return condition should make it stop or return condition should make it stop or return to you. You of course also want to add to you. You of course also want to add to you. You of course also want to add what can the agent do and what can the what can the agent do and what can the what can the agent do and what can the agent not do without your approval. Now, agent not do without your approval. Now, agent not do without your approval. Now, I call mine current. markdown. It's an I call mine current. markdown. It's an I call mine current. markdown. It's an ordinary file. It's short enough that I ordinary file. It's short enough that I ordinary file. It's short enough that I will actually read it and it sits where will actually read it and it sits where will actually read it and it sits where the agent will look and can read and the agent will look and can read and the agent will look and can read and write to it. And when I do a run like write to it. And when I do a run like write to it. And when I do a run like this and I reach that checkpoint, I can this and I reach that checkpoint, I can this and I reach that checkpoint, I can review the work and I can locate where I review the work and I can locate where I review the work and I can locate where I need to change that current markdown need to change that current markdown need to change that current markdown file to adjust the next run of the agent file to adjust the next run of the agent file to adjust the next run of the agent so that I keep the agent on track. The so that I keep the agent on track. The so that I keep the agent on track. The evidence may fail to support the claim evidence may fail to support the claim evidence may fail to support the claim I'm making. Users might care about a I'm making. Users might care about a I'm making. Users might care about a different problem. You can imagine all different problem. You can imagine all different problem. You can imagine all sorts of progressive discovery along the sorts of progressive discovery along the sorts of progressive discovery along the way. Ultimately, this puts our human way. Ultimately, this puts our human way. Ultimately, this puts our human decisions above the loop as decisions above the loop as decisions above the loop as appropriately in charge of agents when appropriately in charge of agents when appropriately in charge of agents when they are doing these big runs. We need they are doing these big runs. We need they are doing these big runs. We need to be able to make corrections along the to be able to make corrections along the to be able to make corrections along the way. So, here's an example of a useful way. So, here's an example of a useful way. So, here's an example of a useful instruction, right? Update the current instruction, right? Update the current instruction, right? Update the current project state so that this evidence project state so that this evidence project state so that this evidence replaces our earlier assumption about replaces our earlier assumption about replaces our earlier assumption about the buyer. apply the new decision to the the buyer. apply the new decision to the the buyer. apply the new decision to the research still open in the research still open in the research still open in the implementation plan and then keep going implementation plan and then keep going implementation plan and then keep going and then preserve this other old and then preserve this other old and then preserve this other old assumption in the decision history assumption in the decision history assumption in the decision history rather than using it as current rather than using it as current rather than using it as current guidance. So, I'm actually giving you guidance. So, I'm actually giving you guidance. So, I'm actually giving you specific language here, but you get the specific language here, but you get the specific language here, but you get the idea. You don't actually have to go in idea. You don't actually have to go in idea. You don't actually have to go in and manually type into this file. You and manually type into this file. You and manually type into this file. You can talk to your AI and basically say, can talk to your AI and basically say, can talk to your AI and basically say, "We're working on current.Markdown "We're working on current.Markdown "We're working on current.Markdown together. Uh, I am reading what you're together. Uh, I am reading what you're together. Uh, I am reading what you're doing here in this checkpoint. I think

doing here in this checkpoint. I think doing here in this checkpoint. I think it's incorrect in these ways. Update it's incorrect in these ways. Update it's incorrect in these ways. Update current. markdown in these ways so you current. markdown in these ways so you current. markdown in these ways so you don't make this mistake again and then don't make this mistake again and then don't make this mistake again and then keep going on our larger project. That's keep going on our larger project. That's keep going on our larger project. That's the idea. This is the capability I think the idea. This is the capability I think the idea. This is the capability I think people miss when they look only at people miss when they look only at people miss when they look only at context window size for agents. A larger context window size for agents. A larger context window size for agents. A larger context window may retain more events context window may retain more events context window may retain more events technically, but compaction may continue technically, but compaction may continue technically, but compaction may continue to work after the window fills and to work after the window fills and to work after the window fills and memory may end up retrieving something memory may end up retrieving something memory may end up retrieving something weird from earlier in the session. So weird from earlier in the session. So weird from earlier in the session. So those mechanisms can't decide which new those mechanisms can't decide which new those mechanisms can't decide which new fact should change your project. They're fact should change your project. They're fact should change your project. They're not a substitute for your judgment. You not a substitute for your judgment. You not a substitute for your judgment. You You still supply that judgment. There's You still supply that judgment. There's You still supply that judgment. There's actually an experiment you can run to actually an experiment you can run to actually an experiment you can run to see this for yourself. You can run the see this for yourself. You can run the see this for yourself. You can run the same agent task multiple times. You run same agent task multiple times. You run same agent task multiple times. You run it once just the way you normally would. it once just the way you normally would. it once just the way you normally would. You run it once with a very focused You run it once with a very focused You run it once with a very focused packet. It has a current.mmarkdown file. packet. It has a current.mmarkdown file. packet. It has a current.mmarkdown file. It has the evidence rules I talked It has the evidence rules I talked It has the evidence rules I talked about, etc. And you run it once with about, etc. And you run it once with about, etc. And you run it once with everything you have like max out the everything you have like max out the everything you have like max out the context window. Now, a lot of people context window. Now, a lot of people context window. Now, a lot of people might think the third one wins, or else might think the third one wins, or else might think the third one wins, or else they might think that the third one at they might think that the third one at they might think that the third one at least allows the agent to have a more least allows the agent to have a more least allows the agent to have a more complete answer. No. When when when you complete answer. No. When when when you complete answer. No. When when when you see what comes out of those runs in see what comes out of those runs in see what comes out of those runs in almost every case, a focused package of almost every case, a focused package of almost every case, a focused package of context that allows you to update state context that allows you to update state context that allows you to update state as you go, it wins because it allows you as you go, it wins because it allows you as you go, it wins because it allows you to put your judgment into place properly to put your judgment into place properly to put your judgment into place properly over the course of the run. Dumping a over the course of the run. Dumping a over the course of the run. Dumping a bunch of context is not a substitute for bunch of context is not a substitute for bunch of context is not a substitute for your judgment. Prompting is still there.

your judgment. Prompting is still there. your judgment. Prompting is still there. Reviewing work is still there, but the Reviewing work is still there, but the Reviewing work is still there, but the whole project no longer depends on one whole project no longer depends on one whole project no longer depends on one perfect prompt at the beginning or one perfect prompt at the beginning or one perfect prompt at the beginning or one conversation and everyone remembers that conversation and everyone remembers that conversation and everyone remembers that conversation equally. Instead, you give conversation equally. Instead, you give conversation equally. Instead, you give the agent enough direction to begin. You the agent enough direction to begin. You the agent enough direction to begin. You learn from what it produces and you learn from what it produces and you learn from what it produces and you change the state that governs what comes change the state that governs what comes change the state that governs what comes next. So maybe take a project that next. So maybe take a project that next. So maybe take a project that you've put on ice because you can't see you've put on ice because you can't see you've put on ice because you can't see every step and you feel like you need to every step and you feel like you need to every step and you feel like you need to sort of define it in advance to give it sort of define it in advance to give it sort of define it in advance to give it to AI and be more bold, right? Look for to AI and be more bold, right? Look for to AI and be more bold, right? Look for where you're going. Look for the source where you're going. Look for the source where you're going. Look for the source material. Look for the boundaries or material. Look for the boundaries or material. Look for the boundaries or guardrails for AI and then give the guardrails for AI and then give the guardrails for AI and then give the agent a little place to keep the current agent a little place to keep the current agent a little place to keep the current state. When the work changes your state. When the work changes your state. When the work changes your judgment, rewrite that state before the judgment, rewrite that state before the judgment, rewrite that state before the agent keeps going. Now, if you want a agent keeps going. Now, if you want a agent keeps going. Now, if you want a starter kit on all of this, I've got starter kit on all of this, I've got starter kit on all of this, I've got you. Uh, I made four ordinary files in a you. Uh, I made four ordinary files in a you. Uh, I made four ordinary files in a set of prompts. I made a a set of prompts. I made a a set of prompts. I made a a readme.markdown that explains this readme.markdown that explains this readme.markdown that explains this method to your agent. I made a method to your agent. I made a method to your agent. I made a current.mmarkdown that holds the state. current.mmarkdown that holds the state. current.mmarkdown that holds the state. I made a contextmap.mmarkdown that says I made a contextmap.mmarkdown that says I made a contextmap.mmarkdown that says what exists and where it lives that your what exists and where it lives that your what exists and where it lives that your agent can help fill out. Uh, and I made agent can help fill out. Uh, and I made agent can help fill out. Uh, and I made a decisions.mmarkdown that holds what a decisions.mmarkdown that holds what a decisions.mmarkdown that holds what you've already decided on and why over you've already decided on and why over you've already decided on and why over the course of your agent run. None of it the course of your agent run. None of it the course of your agent run. None of it is a feature for any particular tool. It is a feature for any particular tool. It is a feature for any particular tool. It works in Claude. It works in codeex. It works in Claude. It works in codeex. It works in Claude. It works in codeex. It works in ordinary chat. The link is on works in ordinary chat. The link is on works in ordinary chat. The link is on my Substack. This is how you get the my Substack. This is how you get the my Substack. This is how you get the benefit of a six-hour agent without benefit of a six-hour agent without benefit of a six-hour agent without asking a six-hour old prompt to keep asking a six-hour old prompt to keep asking a six-hour old prompt to keep running your project.