Anthropic's CCA Exam as a Field-Guide for Agentic Engineering — Frank Coyle, UC Berkeley

>> Okay, I'm getting rolling and uh welcome >> Okay, I'm getting rolling and uh welcome aboard. We just had a little technical aboard. We just had a little technical aboard. We just had a little technical issues, issues, issues, but uh we resolved them. So, my name is but uh we resolved them. So, my name is but uh we resolved them. So, my name is Frank Coyle. Frank Coyle. Frank Coyle. Uh I am a computer science guy. I've Uh I am a computer science guy. I've Uh I am a computer science guy. I've been teaching computer science for over been teaching computer science for over been teaching computer science for over 30 years, 30 years, 30 years, and I'm now teaching at Berkeley. And and I'm now teaching at Berkeley. And and I'm now teaching at Berkeley. And one of the problems that uh all my one of the problems that uh all my one of the problems that uh all my students, students, students, past and present, are having is AI, past and present, are having is AI, past and present, are having is AI, because computer science is no longer because computer science is no longer because computer science is no longer the magic pathway to a job. So, I've the magic pathway to a job. So, I've the magic pathway to a job. So, I've been trying to figure out ways to uh been trying to figure out ways to uh been trying to figure out ways to uh help them come up with schemes to help help them come up with schemes to help help them come up with schemes to help them get ready for this world of agentic them get ready for this world of agentic them get ready for this world of agentic AI. And one of the things that sort of AI. And one of the things that sort of AI. And one of the things that sort of uh uh uh dropped into my uh plate was the dropped into my uh plate was the dropped into my uh plate was the something called the Claude Certified something called the Claude Certified something called the Claude Certified Architect exam, which I will be talking Architect exam, which I will be talking Architect exam, which I will be talking about today, and it has um a number of about today, and it has um a number of about today, and it has um a number of aspects to it. And I think if you're aspects to it. And I think if you're aspects to it. And I think if you're interested in a career in agentic AI, interested in a career in agentic AI, interested in a career in agentic AI, then certainly take a look at least what then certainly take a look at least what then certainly take a look at least what the exam is about, because I feel that the exam is about, because I feel that the exam is about, because I feel that um Anthropic knows how people are using um Anthropic knows how people are using um Anthropic knows how people are using their system and what the issues are their system and what the issues are their system and what the issues are going to be.

going to be. going to be. So, before we jump into that, I want to So, before we jump into that, I want to So, before we jump into that, I want to give a little bit of my give a little bit of my give a little bit of my uh uh uh my philosophy. May have to do this manually, getting May have to do this manually, getting stuck. stuck. stuck. So, So, So, this is a quote from uh this is a quote from uh this is a quote from uh a woman named Sister Corita Kent. a woman named Sister Corita Kent. a woman named Sister Corita Kent. Nothing is a mistake. There's no win and Nothing is a mistake. There's no win and Nothing is a mistake. There's no win and no fail. There's only make. no fail. There's only make. no fail. There's only make. Bottom line here is experiment, Bottom line here is experiment, Bottom line here is experiment, experiment, experiment. Not only should experiment, experiment. Not only should experiment, experiment. Not only should you read, but you should do. You should you read, but you should do. You should you read, but you should do. You should make stuff. Now, what happens when you make stuff. Now, what happens when you make stuff. Now, what happens when you make stuff? A lot of times things don't make stuff? A lot of times things don't make stuff? A lot of times things don't work. work. work. Thomas Edison said, "I have not failed. Thomas Edison said, "I have not failed. Thomas Edison said, "I have not failed. I've only found 10,000 ways I've only found 10,000 ways I've only found 10,000 ways that don't work." that don't work." that don't work." And And And what I want to emphasize here is that what I want to emphasize here is that what I want to emphasize here is that what this shows us are something that in what this shows us are something that in what this shows us are something that in the design patterns movement, which came the design patterns movement, which came the design patterns movement, which came around in the early 1990s with around in the early 1990s with around in the early 1990s with object-oriented programming, we had object-oriented programming, we had object-oriented programming, we had patterns for objects. We now have patterns for objects. We now have patterns for objects. We now have patterns for agents, but there's also patterns for agents, but there's also patterns for agents, but there's also anti-patterns. And I think anti-patterns anti-patterns. And I think anti-patterns anti-patterns. And I think anti-patterns are a key are a key are a key to understanding what you should not do to understanding what you should not do to understanding what you should not do because understanding what you should because understanding what you should because understanding what you should not do is the key to leading you to what not do is the key to leading you to what not do is the key to leading you to what you should do.

So, a little bit about the Claude So, a little bit about the Claude Certified Exam, released in March, so Certified Exam, released in March, so Certified Exam, released in March, so it's brand new. it's brand new. it's brand new. It is uh It is uh It is uh it is it is it is based on scenarios. It is timed. It is based on scenarios. It is timed. It is based on scenarios. It is timed. It is proctored. proctored. proctored. It is available to companies in the It is available to companies in the It is available to companies in the Claude ecosystem, the Anthropic Claude ecosystem, the Anthropic Claude ecosystem, the Anthropic ecosystem, but individuals can pay $99 ecosystem, but individuals can pay $99 ecosystem, but individuals can pay $99 and take the exam once every once every and take the exam once every once every and take the exam once every once every 6 months. 6 months. 6 months. And it's not just And it's not just And it's not just multiple-choice questions. It is multiple-choice questions. It is multiple-choice questions. It is multiple-choice, but they're multiple-choice, but they're multiple-choice, but they're they are based on they are based on they are based on uh realistic constraints and realistic uh realistic constraints and realistic uh realistic constraints and realistic scenarios. scenarios. scenarios. The five domains. The five domains. The five domains. There are five domains that are covered There are five domains that are covered There are five domains that are covered and they give you the percentages of and they give you the percentages of and they give you the percentages of each. So, agentic architecture, 27%. each. So, agentic architecture, 27%. each. So, agentic architecture, 27%. Claude code, how to configure the Claude Claude code, how to configure the Claude Claude code, how to configure the Claude code system and workflow, 20%. How to code system and workflow, 20%. How to code system and workflow, 20%. How to doing prompt engineering, structuring doing prompt engineering, structuring doing prompt engineering, structuring your output, using JSON all over the your output, using JSON all over the your output, using JSON all over the place.

place. place. Tool design. Model context protocol Tool design. Model context protocol Tool design. Model context protocol integration. These are topics that you integration. These are topics that you integration. These are topics that you should understand and know whether should understand and know whether should understand and know whether you're going to take the exam or not. you're going to take the exam or not. you're going to take the exam or not. This is going to help you get ready for This is going to help you get ready for This is going to help you get ready for whatever whatever whatever the agentic world is going to throw at the agentic world is going to throw at the agentic world is going to throw at you. And then there's going to be you. And then there's going to be you. And then there's going to be contact management and reliability. So contact management and reliability. So contact management and reliability. So these are the these are the these are the areas of of the kind of questions you're areas of of the kind of questions you're areas of of the kind of questions you're going to run into. going to run into. going to run into. Then there are and they they provide you Then there are and they they provide you Then there are and they they provide you with six production scenarios and your with six production scenarios and your with six production scenarios and your the exam will randomly choose four and the exam will randomly choose four and the exam will randomly choose four and all the questions will be centered all the questions will be centered all the questions will be centered around the four that they choose. around the four that they choose. around the four that they choose. And what I'm going to do is walk you And what I'm going to do is walk you And what I'm going to do is walk you through through through um um um the production scenarios and give you the production scenarios and give you the production scenarios and give you some anti-patterns to be aware of some anti-patterns to be aware of some anti-patterns to be aware of because there's a number of ways you can because there's a number of ways you can because there's a number of ways you can solve the problem but one of the big solve the problem but one of the big solve the problem but one of the big things is what not to do and that often things is what not to do and that often things is what not to do and that often can be the key to getting these can be the key to getting these can be the key to getting these questions right. So, number one customer questions right. So, number one customer questions right. So, number one customer support resolution agent. So we have support resolution agent. So we have support resolution agent. So we have agentic loops, control, something called agentic loops, control, something called agentic loops, control, something called stop reason which is stop reason which is stop reason which is uh what Cloud Code has. Every time uh what Cloud Code has. Every time uh what Cloud Code has. Every time something happens, there's a stop reason something happens, there's a stop reason something happens, there's a stop reason and you need to take a look at that and you need to take a look at that and you need to take a look at that because that can give you a lot of because that can give you a lot of because that can give you a lot of information about what's going on.

information about what's going on. information about what's going on. Uh scenario two, code generation. Uh scenario two, code generation. Uh scenario two, code generation. Three, multi-agent research system which Three, multi-agent research system which Three, multi-agent research system which we'll look at. How do you How do you we'll look at. How do you How do you we'll look at. How do you How do you distribute your agents? Hub and spoke. distribute your agents? Hub and spoke. distribute your agents? Hub and spoke. Who's the orchestrator? How much Who's the orchestrator? How much Who's the orchestrator? How much information should they know? All these information should they know? All these information should they know? All these are important factors. Um are important factors. Um are important factors. Um scenario four, developer scenario four, developer scenario four, developer productivity with code. So how do you do productivity with code. So how do you do productivity with code. So how do you do subtask isolation? Keep your tasks in subtask isolation? Keep your tasks in subtask isolation? Keep your tasks in their little universes. And this their little universes. And this their little universes. And this hearkens back to what we learn in hearkens back to what we learn in hearkens back to what we learn in computer science from doing computer science from doing computer science from doing multi-threaded programming. multi-threaded programming. multi-threaded programming. When you have multiple threads operating When you have multiple threads operating When you have multiple threads operating and sharing memory, then you get into and sharing memory, then you get into and sharing memory, then you get into issues with synchronization. You You to issues with synchronization. You You to issues with synchronization. You You to put locks put locks put locks Keep the little threads independent. Keep the little threads independent. Keep the little threads independent. Keep your agents independent. Keep your agents independent. Keep your agents independent. Um Um Um and then some cloud code for continuous and then some cloud code for continuous and then some cloud code for continuous integration. integration. integration. And then we'll look at some patterns for And then we'll look at some patterns for And then we'll look at some patterns for structured data extraction. Okay, that's structured data extraction. Okay, that's structured data extraction. Okay, that's kind of where we're going to go.

kind of where we're going to go. kind of where we're going to go. Now, here's something that I I I like to Now, here's something that I I I like to Now, here's something that I I I like to point out. Everybody's talking about point out. Everybody's talking about point out. Everybody's talking about loops, right? Every The loop is the new loops, right? Every The loop is the new loops, right? Every The loop is the new thing. thing. thing. Um Um Um uh Boris Cherney says he doesn't write uh Boris Cherney says he doesn't write uh Boris Cherney says he doesn't write code, but his job is to write loops. code, but his job is to write loops. code, but his job is to write loops. And Peter Steinberger And Peter Steinberger And Peter Steinberger master of Open Claw says, "I don't I master of Open Claw says, "I don't I master of Open Claw says, "I don't I don't uh I don't code anymore. I just don't uh I don't code anymore. I just don't uh I don't code anymore. I just design loops design loops design loops that prompt your agents." that prompt your agents." that prompt your agents." So, loops are the new big thing, right? So, loops are the new big thing, right? So, loops are the new big thing, right? Well, no, they're not. Okay? Um Well, no, they're not. Okay? Um Well, no, they're not. Okay? Um back in the day back in the day back in the day uh early days of computing, we had uh early days of computing, we had uh early days of computing, we had programming languages were exploding. We programming languages were exploding. We programming languages were exploding. We had Fortran, we had COBOL, and there had Fortran, we had COBOL, and there had Fortran, we had COBOL, and there were big fights. My program My were big fights. My program My were big fights. My program My programming language is better than programming language is better than programming language is better than yours. It can do more. No, it can't. We yours. It can do more. No, it can't. We yours. It can do more. No, it can't. We can do this. can do this. can do this. Böhm and Jacopini, 1966 Böhm and Jacopini, 1966 Böhm and Jacopini, 1966 proved that if you want a language to be proved that if you want a language to be proved that if you want a language to be Turing complete, which means can compute Turing complete, which means can compute Turing complete, which means can compute anything that computers are possibly anything that computers are possibly anything that computers are possibly able to compute, then you need only able to compute, then you need only able to compute, then you need only three things.

three things. three things. The ability to The ability to The ability to to to write statements sequentially, to to write statements sequentially, to to write statements sequentially, okay? okay? okay? To have if-then conditionals, and the To have if-then conditionals, and the To have if-then conditionals, and the third piece is the loop. third piece is the loop. third piece is the loop. If you add the loop, If you add the loop, If you add the loop, you have Turing computability. And now you have Turing computability. And now you have Turing computability. And now we are seeing this being resurrected in we are seeing this being resurrected in we are seeing this being resurrected in the agentic world with the focus on the agentic world with the focus on the agentic world with the focus on loops, cuz up to now we've had sort of loops, cuz up to now we've had sort of loops, cuz up to now we've had sort of sequences. You have prompts, you have sequences. You have prompts, you have sequences. You have prompts, you have maybe if-then, but now we have a loop. maybe if-then, but now we have a loop. maybe if-then, but now we have a loop. And now this is what's giving us the And now this is what's giving us the And now this is what's giving us the power. This is where the agentic stuff power. This is where the agentic stuff power. This is where the agentic stuff is getting very exciting. is getting very exciting. is getting very exciting. Okay. Okay. Okay. I'm start with uh I'm start with uh I'm start with uh with scenario one, customer support with scenario one, customer support with scenario one, customer support resolution. resolution. resolution. So here we have a loop operating and a loop operating and the I'm going to jump to the the I'm going to jump to the the I'm going to jump to the anti-pattern. What you don't want is anti-pattern. What you don't want is anti-pattern. What you don't want is just to let the agent go and do just to let the agent go and do just to let the agent go and do something and get the response back and something and get the response back and something and get the response back and use it, okay? What you want to do is you use it, okay? What you want to do is you use it, okay? What you want to do is you want to loop with something called the want to loop with something called the want to loop with something called the stop reason. So I'm going to show you a stop reason. So I'm going to show you a stop reason. So I'm going to show you a little code here.

little code here. little code here. So here we have while loop. It's a while So here we have while loop. It's a while So here we have while loop. It's a while true, it's a loop. We're looping right true, it's a loop. We're looping right true, it's a loop. We're looping right here, okay? So the first little block is here, okay? So the first little block is here, okay? So the first little block is where we call uh we call the model, where we call uh we call the model, where we call uh we call the model, okay? And we pass it the messages. The okay? And we pass it the messages. The okay? And we pass it the messages. The messages are essentially the sequence of messages are essentially the sequence of messages are essentially the sequence of prompts that exist in the context prompts that exist in the context prompts that exist in the context window, okay? And we are asking the and window, okay? And we are asking the and window, okay? And we are asking the and we have a we have a prompt and we have we have a we have a prompt and we have we have a we have a prompt and we have we have the context and we have a tool. we have the context and we have a tool. we have the context and we have a tool. And we're asking the LLM And we're asking the LLM And we're asking the LLM to do something with this tool and help to do something with this tool and help to do something with this tool and help us out. The problem is the LLM can't do us out. The problem is the LLM can't do us out. The problem is the LLM can't do anything. It is just a probabilistic anything. It is just a probabilistic anything. It is just a probabilistic next word predictor. next word predictor. next word predictor. It can't execute tools. So what it does It can't execute tools. So what it does It can't execute tools. So what it does though is it can figure out though is it can figure out though is it can figure out if you point it to a tool, it can figure if you point it to a tool, it can figure if you point it to a tool, it can figure out how to set things up so that you or out how to set things up so that you or out how to set things up so that you or your code can execute it. So it's your code can execute it. So it's your code can execute it. So it's important to understand that the LLM is important to understand that the LLM is important to understand that the LLM is not executing these tools. It can't do not executing these tools. It can't do not executing these tools. It can't do anything except talk back to you, very anything except talk back to you, very anything except talk back to you, very intelligently sometimes, but all it can intelligently sometimes, but all it can intelligently sometimes, but all it can do is talk back to you. So do is talk back to you. So do is talk back to you. So when it finishes when it finishes when it finishes this this this task and has a result which is basically task and has a result which is basically task and has a result which is basically here is I've I know what you want. I here is I've I know what you want. I here is I've I know what you want. I know what the tool can do. Here's how I know what the tool can do. Here's how I know what the tool can do. Here's how I It sets up the parameters that can then It sets up the parameters that can then It sets up the parameters that can then be or that then used to actually execute be or that then used to actually execute be or that then used to actually execute the tool. So, the second block you see

the tool. So, the second block you see the tool. So, the second block you see why did why did why did the LLM come back to us? That's our stop the LLM come back to us? That's our stop the LLM come back to us? That's our stop reason. reason. reason. Tool use. Oh, okay. We've stopped Tool use. Oh, okay. We've stopped Tool use. Oh, okay. We've stopped because because because the LLM it wants to use the tool. the LLM it wants to use the tool. the LLM it wants to use the tool. So, let's just run the tool. So, that's So, let's just run the tool. So, that's So, let's just run the tool. So, that's what the second block is. Run tool, the what the second block is. Run tool, the what the second block is. Run tool, the response is what the LLM said, and it's response is what the LLM said, and it's response is what the LLM said, and it's basically the parameters that it has basically the parameters that it has basically the parameters that it has extracted from the data that you extracted from the data that you extracted from the data that you provided it. provided it. provided it. Okay? Then it executes that. Okay? Then it executes that. Okay? Then it executes that. Then it goes back. Then it goes back. Then it goes back. That then it continues. Continues means That then it continues. Continues means That then it continues. Continues means the LLM sees it and says, "Oh, the LLM sees it and says, "Oh, the LLM sees it and says, "Oh, successful run. So, okay." successful run. So, okay." successful run. So, okay." Come back down. Come back down. Come back down. We're not running a tool anymore. We're We're not running a tool anymore. We're We're not running a tool anymore. We're end the end of our loop. Bingo. end the end of our loop. Bingo. end the end of our loop. Bingo. Now, Now, Now, then we take the answer, and this is an then we take the answer, and this is an then we take the answer, and this is an opportunity for you to opportunity for you to opportunity for you to have a human in the loop potentially. have a human in the loop potentially. have a human in the loop potentially. You check the confidence. If it looks You check the confidence. If it looks You check the confidence. If it looks good, you keep it. If you don't, then good, you keep it. If you don't, then good, you keep it. If you don't, then you escalate to a human. you escalate to a human. you escalate to a human. So, now there's another reason why you So, now there's another reason why you So, now there's another reason why you need to make sure you check your stop need to make sure you check your stop need to make sure you check your stop reason. One of the stop reasons may be reason. One of the stop reasons may be reason. One of the stop reasons may be you have run out of tokens, and this you have run out of tokens, and this you have run out of tokens, and this response is based on partial when the response is based on partial when the response is based on partial when the LLM had to stop.

LLM had to stop. LLM had to stop. And it's going to give you a response, And it's going to give you a response, And it's going to give you a response, but if you have run out of tokens, then but if you have run out of tokens, then but if you have run out of tokens, then you need to take action. you need to take action. you need to take action. Okay. Okay. Okay. Um Um Um Next scenario. Next scenario. Next scenario. Uh code generation with Claude. So, Uh code generation with Claude. So, Uh code generation with Claude. So, Claude code has this has this concept of Claude code has this has this concept of Claude code has this has this concept of the Claude MD file, a markdown file, the Claude MD file, a markdown file, the Claude MD file, a markdown file, where you put all the things you wanted where you put all the things you wanted where you put all the things you wanted to know. to know. to know. What Anthropic recommends is you have What Anthropic recommends is you have What Anthropic recommends is you have three levels of Claude. three levels of Claude. three levels of Claude. One One One that you have at the top level of your that you have at the top level of your that you have at the top level of your project, project, project, the other that you have in inside your the other that you have in inside your the other that you have in inside your sort of the project folder, and then sort of the project folder, and then sort of the project folder, and then within directories you can also specify. within directories you can also specify. within directories you can also specify. So, the idea is to have a hierarchical So, the idea is to have a hierarchical So, the idea is to have a hierarchical set of rules that that can then control set of rules that that can then control set of rules that that can then control how the system is going to respond. how the system is going to respond. how the system is going to respond. Okay. Okay. Okay. Moving right along, Moving right along, Moving right along, uh we have a multi-agent research uh we have a multi-agent research uh we have a multi-agent research system. So, here we're going to have uh system. So, here we're going to have uh system. So, here we're going to have uh the problem is the problem is the problem is how do I how do I get my agents to to go how do I how do I get my agents to to go how do I how do I get my agents to to go off and do stuff and bring the answers off and do stuff and bring the answers off and do stuff and bring the answers back in a reasonable way? The back in a reasonable way? The back in a reasonable way? The anti-pattern anti-pattern anti-pattern you you you have one agent and you load it up with have one agent and you load it up with have one agent and you load it up with tools, all right? So, I like to think tools, all right? So, I like to think tools, all right? So, I like to think about you about you about you you know, you hire somebody to come to you know, you hire somebody to come to you know, you hire somebody to come to your house, you hire a carpenter to come your house, you hire a carpenter to come your house, you hire a carpenter to come to the house, and the guy shows up with to the house, and the guy shows up with to the house, and the guy shows up with uh uh uh plumbing tools, carpenter tools, plumbing tools, carpenter tools, plumbing tools, carpenter tools, electrical tools. He says, "I can do electrical tools. He says, "I can do electrical tools. He says, "I can do anything." Well, maybe you don't want

anything." Well, maybe you don't want anything." Well, maybe you don't want this guy, maybe you want a a this guy, maybe you want a a this guy, maybe you want a a professional carpenter. So, that's the professional carpenter. So, that's the professional carpenter. So, that's the kind of idea. And this kind of back kind of idea. And this kind of back kind of idea. And this kind of back takes us back to some of the the takes us back to some of the the takes us back to some of the the functional programming functional programming functional programming uh uh uh ideas that functions should be do one ideas that functions should be do one ideas that functions should be do one thing. And if you can get your agents to thing. And if you can get your agents to thing. And if you can get your agents to do one thing, do one thing, do one thing, you with maybe one or two tools you with maybe one or two tools you with maybe one or two tools available to it, then that's going to be available to it, then that's going to be available to it, then that's going to be a win, and that's going to help you with a win, and that's going to help you with a win, and that's going to help you with this exam. So, specialize, this exam. So, specialize, this exam. So, specialize, don't overload. don't overload. don't overload. The other part of this is The other part of this is The other part of this is don't let your agents don't let your agents don't let your agents context spill over into the main context context spill over into the main context context spill over into the main context because context means tokens, tokens because context means tokens, tokens because context means tokens, tokens mean money, mean money, mean money, and the more context you have, the more and the more context you have, the more and the more context you have, the more confused the LLM is going to be in confused the LLM is going to be in confused the LLM is going to be in giving you an answer. So, even though giving you an answer. So, even though giving you an answer. So, even though oh, a million token context window, I oh, a million token context window, I oh, a million token context window, I can put everything in there. No, no, can put everything in there. No, no, can put everything in there. No, no, don't put everything in there. don't put everything in there. don't put everything in there. Limit what's going to go in there Limit what's going to go in there Limit what's going to go in there because then you're going to get because then you're going to get because then you're going to get a much more accurate system.

So, here's a So, here's a Here's an example of a specialized sub Here's an example of a specialized sub Here's an example of a specialized sub agents. agents. agents. You're giving it You're giving it You're giving it So, this would be the critic. So, let's So, this would be the critic. So, let's So, this would be the critic. So, let's say you've run some stuff. Now, you want say you've run some stuff. Now, you want say you've run some stuff. Now, you want to get an agent to look at what's to get an agent to look at what's to get an agent to look at what's happened. What you want to do is just happened. What you want to do is just happened. What you want to do is just give it what it needs to solve that give it what it needs to solve that give it what it needs to solve that critic problem. I'm only giving it here critic problem. I'm only giving it here critic problem. I'm only giving it here the the the we're passing it we're passing it we're passing it the claim and the evidence. So, this is the claim and the evidence. So, this is the claim and the evidence. So, this is your claim is sort of how we're going to your claim is sort of how we're going to your claim is sort of how we're going to solve the problem. Here's Here's the solve the problem. Here's Here's the solve the problem. Here's Here's the evidence, but we're not giving it the evidence, but we're not giving it the evidence, but we're not giving it the the thought processes that went in to the thought processes that went in to the thought processes that went in to creating this claim. Why? creating this claim. Why? creating this claim. Why? When you When you When you When you get a bunch of agents together When you get a bunch of agents together When you get a bunch of agents together collaborating and talking to each other, collaborating and talking to each other, collaborating and talking to each other, there's a tendency to have group think. there's a tendency to have group think. there's a tendency to have group think. And And And all the agents seem to kind of devolve all the agents seem to kind of devolve all the agents seem to kind of devolve into one idea. I mean, it's it's like, into one idea. I mean, it's it's like, into one idea. I mean, it's it's like, you know, you're in a group, you know, you know, you're in a group, you know, you know, you're in a group, you know, you're at a party, and everybody wants you're at a party, and everybody wants you're at a party, and everybody wants pizza except you, but then people talk pizza except you, but then people talk pizza except you, but then people talk you into you into you into you you know, you don't want to be uh you you know, you don't want to be uh you you know, you don't want to be uh you don't want to spoil the party, so you don't want to spoil the party, so you don't want to spoil the party, so you'll go along. And it seems that you'll go along. And it seems that you'll go along. And it seems that agents kind of work in the same way.

agents kind of work in the same way. agents kind of work in the same way. So, you're going to return So, you're going to return So, you're going to return Basically, you're going to give each Basically, you're going to give each Basically, you're going to give each agent only a slice. I didn't think about agent only a slice. I didn't think about agent only a slice. I didn't think about the pizza analogy, but yes. Every agent the pizza analogy, but yes. Every agent the pizza analogy, but yes. Every agent gets its own slice, and and it it should gets its own slice, and and it it should gets its own slice, and and it it should come through. come through. come through. Okay. Fourth scenario, Fourth scenario, developer productivity. So, the developer productivity. So, the developer productivity. So, the anti-pattern. anti-pattern. anti-pattern. Let every subtask dump its full output Let every subtask dump its full output Let every subtask dump its full output into the primary thread, crowding out into the primary thread, crowding out into the primary thread, crowding out the context. Again, this is what we're I the context. Again, this is what we're I the context. Again, this is what we're I was just talking about. This is bad. Let was just talking about. This is bad. Let was just talking about. This is bad. Let the context grow unbounded. Bad, right? the context grow unbounded. Bad, right? the context grow unbounded. Bad, right? For the reasons we just talked about. For the reasons we just talked about. For the reasons we just talked about. You want to isolate your subtask output, You want to isolate your subtask output, You want to isolate your subtask output, and you want to compact and you want to compact and you want to compact long sessions. I'm going to take a long sessions. I'm going to take a long sessions. I'm going to take a second to talk about that. So, here's second to talk about that. So, here's second to talk about that. So, here's here's a here's a here's a an example of a pattern. an example of a pattern. an example of a pattern. Uh Uh Uh you want to have your agent you want to have your agent you want to have your agent uh uh uh look at the logs and create a summary look at the logs and create a summary look at the logs and create a summary of where the problems are in the log.

of where the problems are in the log. of where the problems are in the log. So, here's your task, scan all the logs So, here's your task, scan all the logs So, here's your task, scan all the logs for error. for error. for error. Context fork. So, you're forking the Context fork. So, you're forking the Context fork. So, you're forking the agent into a like a separate thread agent into a like a separate thread agent into a like a separate thread where where where whatever the agent does and thinks and whatever the agent does and thinks and whatever the agent does and thinks and adds tokens to does not come back and adds tokens to does not come back and adds tokens to does not come back and pollute the main pollute the main pollute the main uh uh uh the main context. the main context. the main context. Now, Now, Now, you see here what happens, then you take you see here what happens, then you take you see here what happens, then you take this this this summation, and then you add that summation, and then you add that summation, and then you add that summation without all the other stuff summation without all the other stuff summation without all the other stuff into the overriding context. Now, this into the overriding context. Now, this into the overriding context. Now, this last little block is kind of last little block is kind of last little block is kind of interesting, I think. Because interesting, I think. Because interesting, I think. Because you can check your token count, you can check your token count, you can check your token count, and you can determine how big the token and you can determine how big the token and you can determine how big the token count is. count is. count is. And And And if you can set some limit and you know, if you can set some limit and you know, if you can set some limit and you know, if if you have more than 150,000 tokens, if if you have more than 150,000 tokens, if if you have more than 150,000 tokens, then what you want to do is you can run then what you want to do is you can run then what you want to do is you can run a compact. So, Anthropic and Claude have a compact. So, Anthropic and Claude have a compact. So, Anthropic and Claude have these compaction algorithms these compaction algorithms these compaction algorithms that take this giant context and and that take this giant context and and that take this giant context and and compact it in some way, shape, or form.

compact it in some way, shape, or form. compact it in some way, shape, or form. Not quite sure how the implementation is Not quite sure how the implementation is Not quite sure how the implementation is of that, but there is compaction. Now, a of that, but there is compaction. Now, a of that, but there is compaction. Now, a little side effect a little side channel little side effect a little side channel little side effect a little side channel I've been walking around when you walk I've been walking around when you walk I've been walking around when you walk outside, you see see these guys handing outside, you see see these guys handing outside, you see see these guys handing out these books. out these books. out these books. Okay? Anybody see these guys handing out Okay? Anybody see these guys handing out Okay? Anybody see these guys handing out these but take them. This is this is these but take them. This is this is these but take them. This is this is actually a pretty good little book. In actually a pretty good little book. In actually a pretty good little book. In fact, I was looking at it last night and fact, I was looking at it last night and fact, I was looking at it last night and one of the things it had in it was this one of the things it had in it was this one of the things it had in it was this is by this guy Sam is by this guy Sam is by this guy Sam Sam Bagwell. I have no connection I Sam Bagwell. I have no connection I Sam Bagwell. I have no connection I didn't even know Sam, but it there's a didn't even know Sam, but it there's a didn't even know Sam, but it there's a online page 32. online page 32. online page 32. It says It says It says uh his company provides custom logic for uh his company provides custom logic for uh his company provides custom logic for compression of context. So, he's got an compression of context. So, he's got an compression of context. So, he's got an and you can write your own. He's got a and you can write your own. He's got a and you can write your own. He's got a he's got he you can extend his base he's got he you can extend his base he's got he you can extend his base class and have your own class and have your own class and have your own compression of your data, whatever you compression of your data, whatever you compression of your data, whatever you think is important. So, I think that's think is important. So, I think that's think is important. So, I think that's kind of an interesting spin on this kind of an interesting spin on this kind of an interesting spin on this whole thing. whole thing. whole thing. Okay. Okay. Okay. Cloud code for Cloud code for Cloud code for uh uh continuous integration uh anti-pattern uh anti-pattern Always have interactive modes in a Always have interactive modes in a Always have interactive modes in a pipeline. Well, no no no cuz interactive pipeline. Well, no no no cuz interactive pipeline. Well, no no no cuz interactive modes mean uh modes mean uh modes mean uh Cloud will stop and ask you, "You want Cloud will stop and ask you, "You want Cloud will stop and ask you, "You want to do this? You want to do that? Can I to do this? You want to do that? Can I to do this? You want to do that? Can I have permission for that?" So, there are have permission for that?" So, there are have permission for that?" So, there are ways to set it up so that it'll just run ways to set it up so that it'll just run ways to set it up so that it'll just run straight through, okay?

straight through, okay? straight through, okay? The other The other The other uh uh uh the other tip that I'll give you here the other tip that I'll give you here the other tip that I'll give you here is there's something called is there's something called is there's something called the uh the uh the uh the batch. So, you can take your the batch. So, you can take your the batch. So, you can take your prompts, you can take your work, and you prompts, you can take your work, and you prompts, you can take your work, and you can put them in a batch and for 50% can put them in a batch and for 50% can put them in a batch and for 50% fewer token cost you will get the result fewer token cost you will get the result fewer token cost you will get the result they promise in at at least 24 hours. they promise in at at least 24 hours. they promise in at at least 24 hours. So, if you're going to go take a nap, So, if you're going to go take a nap, So, if you're going to go take a nap, you're going to go on vacation, you're you're going to go on vacation, you're you're going to go on vacation, you're going to go out, take a a day off, run going to go out, take a a day off, run going to go out, take a a day off, run your stuff in batch mode, and you're your stuff in batch mode, and you're your stuff in batch mode, and you're going to have a a going to have a a going to have a a less to pay. Where am I here? Where am I here? All right, I've only got a few few All right, I've only got a few few All right, I've only got a few few minutes left, few seconds left, but I minutes left, few seconds left, but I minutes left, few seconds left, but I want to conclude with this. want to conclude with this. want to conclude with this. Remember, nothing is a mistake. There's Remember, nothing is a mistake. There's Remember, nothing is a mistake. There's no win, there's no fail, there's no no win, there's no fail, there's no no win, there's no fail, there's no exam, exam, exam, only make. You do it and you make it and only make. You do it and you make it and only make. You do it and you make it and you're going to succeed. If you want to you're going to succeed. If you want to you're going to succeed. If you want to reach out to me, reach out to me uh coil reach out to me, reach out to me uh coil reach out to me, reach out to me uh coil at Berkeley, look at my websites. I got at Berkeley, look at my websites. I got at Berkeley, look at my websites. I got a website co-supreme AI. I'm a big jazz a website co-supreme AI. I'm a big jazz a website co-supreme AI. I'm a big jazz fan and I named this website after John fan and I named this website after John fan and I named this website after John Coltrane, Love Supreme, if you know that Coltrane, Love Supreme, if you know that Coltrane, Love Supreme, if you know that song, great. Anyway, that's my story and song, great. Anyway, that's my story and song, great. Anyway, that's my story and I'm sticking to it and I'm about to zero I'm sticking to it and I'm about to zero I'm sticking to it and I'm about to zero time. Okay, time. Okay, time. Okay, >> [applause] >> [applause] >> [applause] >> thank you.