How Anthropic Merges 30,000 Code Changes a Day - Manish Kapur, Sonar & Addy Osmani, Anthropic

[music] >> I'm pretty excited to be here and talk about who verifies the software factory. Uh before we get started, uh can you give me a quick show of hands? Uh raise your hand if your coding agent has pulled uh opened any pull request in the last 1 week. Almost everyone, thanks. And keep your hands raised up if you have merged pull requests and reviewed every line of code that was merged. Oh, I do see a few people. That's excellent. So, what I'm going to talk about is the gap between the uh two uh camps, uh merging requests without reviewing manually, and uh you know, merging requests uh after doing some reviews. So, my name is Manish Kapoor, and I lead the product and solution uh developer relations uh team at Sonar, and I'm excited to be here uh with my guest speaker, Addy Osmani from Anthropic. Uh Addy has been uh had been at Google for about 14 years. Most of you probably know him.

And he's written lots and lots of content and articles. Uh I learn a lot from Addy, so I'm uh really fortunate to have him here, stand next to me, and uh share his wisdom with all of us here today. Uh he recently joined Anthropic, and Addy, so congratulations on your uh role at uh Anthropic. Would you like to tell the team what you're working on at Anthropic, what your role is? >> Yeah, sure. So, I'm responsible for making sure that you all have a really great developer experience building with Cloud Code, and anytime that we discover there are opportunities to make it even smoother, we bring that right back into the product. We ship a couple of times every single day, and so we're very interested in trying to make sure that you have the best possible experience as often as possible. >> Cool. Thank you. All right. Uh so, let's start with some problem and context setting slide. There are two numbers on the slide. CircleCI analyzed 28 million CI workflows this year. And they noticed that the throughput jumped by about 59%.

But yet three in 10 attempts to merge that code into production are failing. And this is a stat from one independent stat from CircleCI. Uh And I think I'm seeing the same trend from most of our customers that I speak to every day. Uh this is a quote from Adi. A software factory is harnessing loops at scale. This resonated quite a bit for me. And if you have not read that article, you can look it up. It's pretty educational article and talking about future how the future will look like in terms of soft software factories. Uh so, Adi, would you like to expand on what you were thinking when you wrote this article about soft loops and harnessing the loops at scale? >> Yeah, absolutely. So, a lot of folks here probably heard this term software factories, and there's two types of factories that are at least worth keeping in mind. Well, you know, software factory is harnessing loops at scale. A lights-on factory is one where you have agents that are helping you with planning, building, checking, and shipping code again and again, but you as a person still apply your judgment and taste inside the picture. And then we have a dark software factory, which is a little bit closer to how, you know, a car factory with robotics might be working where it's constantly doing the work. So, it's scoping, building, checking, and you as the person move down further to thinking about intent, and you're pulled in when they're really the greatest of exceptions to the case.

But, that's sort of what I was thinking. I think a lot of teams are somewhere on this spectrum and are on this journey to figure out like how how do I get further and further into automation in a way that I can trust it. >> Cool. So uh AI coding is now basically a solved problem. We all know agents are there. 85% of the organizations are already using coding agents. It will go to 95% and very soon to 100%. Uh 75% of the uh 76% of the developers that generate AI code, uh they are saying that it it's being code is being produced at a speed that is impossible for them to verify it. So, they are spending more and more time reviewing the code. So, what do you think is the most pressing question that engineering leaders have right now? >> So, I think that teams have gotten better working with agents over the last year. Um models, harnesses have also gotten better. And what I see a lot of leaders being interested in is how do we enable you to ship code faster um with confidence. And that confidence piece is very important. Agents can now generate ways for you to check your code much more cheaply than ever before. And if you think about the things that enabled us to call good code, you know, high quality, uh those were things like good tests, benchmarks, uh specs, anything that gives us confidence. So, agents are able to help us generate that a little bit more easily these days, a little bit more quickly. We can of course refine those things, but I think that those checks are one of the most important things we should be spending our time on

these days. Like if you are hoping to get the best quality output from your agents, you need to give your agents a way to verify that the work is correct. If you've got that in place and it you can make sure that the code is both correct and complete, that can easily two to three by like the quality of your outcomes. >> Yep. Yep. Absolutely right. Spot on. I think code confidence uh we have been running an ad code confidence is scarce. And you really need to build it up now that the coding problem is solved. So, I actually I yesterday night, so this might be a surprise question for you. Uh yesterday night I read a post uh from Anthropic, and I encourage all of you to read it. It talks about the scale at which Anthropic is operating. Uh they are merging about 30,000 code changes every single day. They used to do about 3,000 or so, 2,500 or so code changes just a few months back. They have scaled from 2,500 code changes to 3,000 uh 30,000 code changes per day. That is mind-boggling. So, I didn't What does code review look like at uh Claude Code at Anthropic and with the Claude Code team that you're working with?

>> Yeah, these are great questions. So, we were we were merging about 2,500 changes all the way back in May. Um and even for our recent refresh of Claude.ai, we put out a case study saying that we kind of three-by'd the performance of it. That was accomplished by merging about 3,000 pull requests in a very short space of time with engineers, product designers, a number of different functions contributing to it. So, our workflow for being able to get verification and code review in place is that we first of all have Claude do a first-pass review. What's very important in there is making sure that it has a way to actually try the experience out similar to a real user. So, that can mean end-to-end tasks, you know, opening up the experience, checking it in a browser, running any tools, doing any linting, doing any other checks that are necessary to build up that confidence. Um people usually ask, "Is it okay, well, what happens if, you know, you discover a bug or a gap in functionality?" Well, we have hooks that will then go and run on any edits. They will study the diffs and allow us to get this loop in place where we're correcting the PR as quickly as possible. So, we have uh Claude Code doing the review. We have a um code review product uh that you can also check out. It's pretty good. And the idea there is we're trying to automate code review um as much as possible. This is something that we use ourselves. Um and increasingly what's important is for your teams to be thinking about how can you cut We talk about taste and judgment so much these days, right? It's important for you to be trying to codify

what you consider best practice and taste in your team's shared like Claude.md, agents.md, anywhere where the agent can then consult during code review. And then people are responsible for checking on the agent's evidence. So, how do we know? Some If I'm building out um a piece of UI that's going to be, you know, used by a user in the browser, I want to make sure that it's tested in the browser, maybe generated me a video of that experience where it's been clicking around and gives me confidence. Well, okay, well, this works. If I care about the mobile or desktop experience, maybe it generates me screenshots or videos of that. But basically, you want to get to a place where the agent's giving you as much verification evidence as possible. >> Cool. Yeah, verification is key. I think you need to verify everything that the software factories will produce and that's what we are going to talk about today. Uh so Uh another these stats by the way are from uh the state of code survey report that will be coming out very soon from Sonar. Uh we surveyed about a thousand more than a thousand developers and we found some interesting data which will give us insights as to what's happening in the industry today and where we are headed.

So, 84% say AI features uh uh gets gets features and fixes put to production faster. The pull requests are getting larger and more complex. Uh and you know, yet uh only uh 31% more code is reaching reaching the production without review. So, that's a little little bit of a scary fact particularly for uh production uh applications in the regulated industries in the mission-critical applications where you absolutely make need to make sure uh that you know, uh you have to verify that code and you're confident in the code that is being shipped. So, uh what are the where are the next gains going to come from, Addy? >> I think the next gains are going to come from a combination of us really nailing verification, right? As well as model quality. But models are already like if you you take a look at Opus 55, model quality has already gotten to a place where it's pretty good. Um and I would say that us being able to share best practices about that verification step is going to become increasingly more important because if you tell people, yeah, just like verify the work, you know, their next question is, what does that even mean?

>> Yeah. >> And so, I think that the conversation needs to evolve into specifically what kinds of tests are useful. We lean a lot on end-to-end tests, property tests, any visual verification tests, things that can help us just get that confidence to merge those changes in automatically. >> Cool. You mentioned light factories. Want to go down that and that's the way uh direction we are headed. Uh our survey results show that, you know, today 5% of the companies are using uh software factories. It will get to 34% and surprisingly 39% in the financial services. How many of you are using software factories today? Just curious. A handful of you. And how many are planning to use software factories by next year? Yeah, that's a good question. How do you define a software factory? Would you like to take that question? >> Yeah, yeah, sure. So, we we talked about definition of software factories just a little bit earlier. A software factory is basically loops at scale. And when we think about this, it's all about like what end of the spectrum are you at in terms of how much people are still involved in the process.

The more that the agents are able to generate verification and give you confidence, the more that you can get to that place where you go dark. And I think that yeah, most people are still somewhere in that spectrum and are not quite as this data shows at that dark point just yet. >> Yep. Just to add to what Addy said, I think software factory, think of it like a production line where you start defining the intent and your specifications. Then you start writing the code. So, it's an assembly line where each of these software development life cycle steps happen. And automating them at scale where there's pretty much in a dark factory, there would be no lights on. It's autonomously working, fully autonomously working. A light factory has uh is where people are involved in the process at every step or most steps. Uh whereas a dark factory, the goal would be to turn off the lights and let the factory run independently 24/7. Uh all right. So, you talked about verification. Uh I'll talk a little bit about what we think at Sonar about verification. If you saw our keynote earlier today from our CEO, he talked about verification should be multi-layered. Uh we absolutely 100% believe that each company should adopt a zero trust multi-layered verification approach. And verification should be done in every loop. Whether you're operating inside the agentic loop or whether you're working inside the CI verification loop or whether you're doing code maintenance. In each and every loop, it is important to verify the output of the factories or the agents that code that the agents are producing.

So, we look at it in terms of layers and think of it like, you know, the there's first layer which is a deterministic layer which catches uh bugs, security issues, things like SQL injection, buffer overflow bugs, and all of those are caught. And then there's a second layer where agentic review happens where agents are used to like Adi was talking about, agents review review that code. And in that agentic review, they catch logic and intent issues which the deterministic layer is not well suited to catch. So, we we use each layer is targeted at catching a specific types of issues. And then the third layer could be a software supply chain security layer where you look at all the CVEs, all your malware detection happens, all your malicious package detection happens. All of that happens in the third layer. So, it's a layered approach where each layer is tuned and optimized to catch a certain type of issue. So, we absolutely recommend doing a multi-layered verification. Oops.

So, there's another stat here. 28% you know, companies have approved agents and shared controls and very few shared controls. And 5% are operating software factories autonomously. What are your thoughts there, Adi? Uh in terms of >> So, I've talked to so many companies where a couple of months ago one of their biggest worries was they were allowing their engineering teams to go off and play around with agents and they got to a place where there were hundreds, thousands of agents in some cases and they just didn't know what they were doing. Like, what are these agents actually doing? They're touching all of our systems, accessing all kinds of data, maybe they're deploying to production. And so, a lot of enterprises think about concerns like, you know, agent identity and telemetry and being able to track what's actually happening. >> Yeah. >> And I'd say that the three biggest things for a lot of companies are how do we think about control, how do we think about permissions, and how do we think about the sandbox? Like, how are we making sure that the agent isn't jumping out of the sandbox and doing things that it shouldn't be doing?

>> Yeah. Yeah, we we at sonar we call it guide rails. So, instead of guard rails, we're thinking of calling it as guide rails, where we guide the agents to follow the real tracks, right? A specific path to take and prevent them from going out of bounds or so to say going out of the sandbox like Addie was saying. So, it's very very important to have guide rails for the agent and govern the agents that you're working with. Uh So, let's uh we're coming to a close now as my timer is telling me we just have a few minutes left. Let's go get into a lightning round. Uh one sentence response for each of these questions. And you've seen the questions already on the slide, so >> Can these be long sentences? >> They They can be as long as the time permits. >> Perfect. >> Uh light or dark factory for most teams by end of 2027, yes or no? >> I think we're going to see uh light factories with some dark corners. I don't think we're going to get to everybody in this room using dark factories entirely, but it's going to be something close to that. >> I I would guess Anthropic would do it this year or next year.

>> [laughter] >> All right. Second question. Should agent-written code be labeled in the PR? >> I personally think it should. Historically, we have felt a great deal of ownership around our code, pride around the fact that, "Hey, I wrote this thing. I'm putting my name on it." And at Anthropic, like we we think about this a little bit differently. We've evolved to the point where we don't really mind so much if Claude is like labeled as, "Hey, I'm I'm the one that wrote this thing." because we're a little bit more focused on the outcomes and making sure that the code, the features, like what we're shipping to users is the right thing. And so, I personally think it's fine for the code to be written labeled as being coming from an agent. And that can also help us with things like figuring out, "Okay, well, if the agent wrote it, what are some of the constraints and extra checks that we need to keep in place?" >> I I have the same point of view, Addie. I think in the short run, at least in the foreseeable you know, 6 to 9 months, we should label the PRs till we are confident enough in our factories. We do want to know and track it, and it's also important for the financial customers here in terms of auditability. If you if you know, if you label it, you know who's writing the code because we are at the end of the day still accountable for >> Yeah, absolutely. Accountability is the most important part here. Like we still label like who is the owner for this change. I might have kicked off a cloud cloud run to complete a piece of work, but I'm I'm still on the hook. If something goes wrong, I have to go and fix it.

>> Yep, yep. Key point. Yep. Uh the next question, actually I'll improvise it here because this is a leadership session. What one skill a junior developer or leader should build on this year? >> I think that you should push your teams, if you're a leader, you should push your teams to really think hard about what verification means. Um how are you already approaching things like testing? What is the gap between that and you getting to a place where you're very comfortable with the agents being able to sign off on the code being good enough to merge on their own. And so figuring that out, if you're a junior, it's the other side of that. Like actually getting more experience in the different flavors of tests, whether it's like end-to-end tests, property tests, etc. >> Yep. And the last uh you you actually not following the rules, you're going beyond one sentence, but that's cool. Uh The last one >> 30 seconds left. I'm fine. >> We can go a minute over, I'm sure. Uh the agent opened the PR. It is safe to merge when? >> You have full confidence nothing's going to go wrong or that you have mitigated that as much as possible.

>> Love it. All right. Uh here's my last slide and then we'll wrap it up. Uh so what separates uh teams that are shipping uh fast but not shipping blind? I think that three key takeaways for you. Uh writing code is the fastest part and that is solved. Strong review and verification for our teams is what will will help you maintain that speed. Fast to write is not fast to ship. Uh you have to basically count what is getting merged, looked at what is getting merged, and make sure it doesn't escape the sandbox and the things like that before it gets into the production. Verify everything. And autonomy is earned. Uh you We will get to software factories very soon if not next year. Uh you need to put in independent automated verification inside each loop of your factory. Make sure that you can trust the output that the factory is producing. Uh, with that, I'll wrap it up. Thanks for attending our talk today, uh, a conversation with Adi. Uh, again, Adi, I appreciate you taking the time and joining us here in this conversation.

Uh, visit us at the big red booth, uh, the sonar booth in the expo hall if you have any further questions or would like to, uh, talk about it. Thank you again.