Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs

>> So, it's not just my AI pipeline that's >> So, it's not just my AI pipeline that's on fire, but also my PowerPoint. So, on fire, but also my PowerPoint. So, on fire, but also my PowerPoint. So, it's 2025. Uh we're scaling as a it's 2025. Uh we're scaling as a it's 2025. Uh we're scaling as a business and things are going poorly. business and things are going poorly. business and things are going poorly. We're adding too many customers, we're We're adding too many customers, we're We're adding too many customers, we're not getting the throughput we need, and not getting the throughput we need, and not getting the throughput we need, and we need to improve our underlying we need to improve our underlying we need to improve our underlying technology. technology. technology. And there's three main issues that we're And there's three main issues that we're And there's three main issues that we're facing. The first one is that we're too facing. The first one is that we're too facing. The first one is that we're too slow to meet customer demand. The second slow to meet customer demand. The second slow to meet customer demand. The second one is that this AI pipeline that we've one is that this AI pipeline that we've one is that this AI pipeline that we've built is too complicated to update. And built is too complicated to update. And built is too complicated to update. And the third one is because it's a legacy the third one is because it's a legacy the third one is because it's a legacy code base, or actually more than 10 code base, or actually more than 10 code base, or actually more than 10 repos, nobody actually wants to touch repos, nobody actually wants to touch repos, nobody actually wants to touch the code. It's not a fun experience. the code. It's not a fun experience. the code. It's not a fun experience. So, we made this decision to refactor So, we made this decision to refactor So, we made this decision to refactor over the course of 6 months. And the over the course of 6 months. And the over the course of 6 months. And the real question for this talk today was, real question for this talk today was, real question for this talk today was, "Is this the right move to do?" "Is this the right move to do?" "Is this the right move to do?" So, I'll spend this time answering this So, I'll spend this time answering this So, I'll spend this time answering this question, but let's start off with the question, but let's start off with the question, but let's start off with the use case. So, the company I work at, use case. So, the company I work at, use case. So, the company I work at, WiseDocs, processes complex medical WiseDocs, processes complex medical WiseDocs, processes complex medical claims, which are PDFs that are more claims, which are PDFs that are more claims, which are PDFs that are more than 10,000 pages in size. Some of these than 10,000 pages in size. Some of these than 10,000 pages in size. Some of these files are bigger than video files. So, files are bigger than video files. So, files are bigger than video files. So, it's a pretty complex application, and it's a pretty complex application, and it's a pretty complex application, and because of this, it's actually because of this, it's actually because of this, it's actually non-trivial to scale the different non-trivial to scale the different non-trivial to scale the different parts. parts. parts. So, we're going to talk about the So, we're going to talk about the So, we're going to talk about the pipeline today, which has a number of ML pipeline today, which has a number of ML pipeline today, which has a number of ML models.

models. models. So, I'll divide this talk into a number So, I'll divide this talk into a number So, I'll divide this talk into a number of chapters. We'll start off with the of chapters. We'll start off with the of chapters. We'll start off with the first one, which is the concept of tech first one, which is the concept of tech first one, which is the concept of tech debt. debt. debt. So, I think we all have this feeling So, I think we all have this feeling So, I think we all have this feeling universally, if we've been developers universally, if we've been developers universally, if we've been developers for a while, that we all write bad code. for a while, that we all write bad code. for a while, that we all write bad code. The question is, do we do this The question is, do we do this The question is, do we do this intentionally or not? intentionally or not? intentionally or not? If I look back to some of the earliest If I look back to some of the earliest If I look back to some of the earliest code I used to write, uh it was bad. code I used to write, uh it was bad. code I used to write, uh it was bad. This was more than 15 years ago. I tried This was more than 15 years ago. I tried This was more than 15 years ago. I tried to print an image of this character from to print an image of this character from to print an image of this character from a video game, and I didn't understand a video game, and I didn't understand a video game, and I didn't understand that you can't uh system.out.println that you can't uh system.out.println that you can't uh system.out.println in Java to render something on the in Java to render something on the in Java to render something on the screen. So, hopefully I've come uh screen. So, hopefully I've come uh screen. So, hopefully I've come uh further from that point in time, but further from that point in time, but further from that point in time, but there's these moments where we all know there's these moments where we all know there's these moments where we all know that we've written bad code before. that we've written bad code before. that we've written bad code before. Now, if we think about technical debt as Now, if we think about technical debt as Now, if we think about technical debt as financial debt, it compounds in financial debt, it compounds in financial debt, it compounds in mysterious and sometimes unexpected mysterious and sometimes unexpected mysterious and sometimes unexpected ways. But you should think about it in a ways. But you should think about it in a ways. But you should think about it in a rigorous format as well. For us to rigorous format as well. For us to rigorous format as well. For us to achieve some kind of ROI by taking on achieve some kind of ROI by taking on achieve some kind of ROI by taking on technical debt, such as building a technical debt, such as building a technical debt, such as building a feature or getting new customers, we feature or getting new customers, we feature or getting new customers, we want to make sure that the ROI makes want to make sure that the ROI makes want to make sure that the ROI makes sense. sense. sense. If we introduce additional complexity If we introduce additional complexity If we introduce additional complexity into our code base, we can very quickly into our code base, we can very quickly into our code base, we can very quickly outrun the ROI we've generated.

Now with AI engineering, you've probably Now with AI engineering, you've probably seen a number of different stories that seen a number of different stories that seen a number of different stories that have come out to showcase the progress have come out to showcase the progress have come out to showcase the progress that's been made. These are two case that's been made. These are two case that's been made. These are two case studies from Anthropic, one from Spotify studies from Anthropic, one from Spotify studies from Anthropic, one from Spotify and the other from Stripe, talking about and the other from Stripe, talking about and the other from Stripe, talking about the immense progress that they've made the immense progress that they've made the immense progress that they've made both in shipping velocity and also the both in shipping velocity and also the both in shipping velocity and also the ability to refactor code. ability to refactor code. ability to refactor code. So at this point in time, writing code So at this point in time, writing code So at this point in time, writing code or making changes is something that or making changes is something that or making changes is something that teams are doing faster and faster. teams are doing faster and faster. teams are doing faster and faster. Now I'll pause here. Now I'll pause here. Now I'll pause here. Who here thinks that products have Who here thinks that products have Who here thinks that products have gotten better in the past 20 years? gotten better in the past 20 years? gotten better in the past 20 years? Technical products. Also raise your Technical products. Also raise your Technical products. Also raise your hand. hand. hand. I hope everybody, right? Phones are I hope everybody, right? Phones are I hope everybody, right? Phones are pretty cool. pretty cool. pretty cool. How about 5 years? How about 5 years? How about 5 years? How about the past year? How about the past year? How about the past year? Okay. Okay. Okay. So the challenge is that we're going So the challenge is that we're going So the challenge is that we're going faster and faster through the technology faster and faster through the technology faster and faster through the technology life cycle, but we've lost something. life cycle, but we've lost something. life cycle, but we've lost something. The the product focused on customers in The the product focused on customers in The the product focused on customers in some way has degraded, the some way has degraded, the some way has degraded, the maintainability of the code and the maintainability of the code and the maintainability of the code and the reliability has degraded. You can see reliability has degraded. You can see reliability has degraded. You can see some of the uptimes here from two some of the uptimes here from two some of the uptimes here from two leading companies. I blurred out their leading companies. I blurred out their leading companies. I blurred out their names for for it it doesn't actually names for for it it doesn't actually names for for it it doesn't actually matter who they are, but we are below a matter who they are, but we are below a matter who they are, but we are below a three-nine or even four-nine three-nine or even four-nine three-nine or even four-nine reliability. So even though we're reliability. So even though we're reliability. So even though we're shipping faster and faster, the code shipping faster and faster, the code shipping faster and faster, the code quality and the product quality has not quality and the product quality has not quality and the product quality has not necessarily gone up.

So let's talk about the refactor that we So let's talk about the refactor that we did. So we started this refactor with did. So we started this refactor with did. So we started this refactor with actual code implementation in April and actual code implementation in April and actual code implementation in April and did some pre-work earlier. did some pre-work earlier. did some pre-work earlier. So I'll go through five different tasks So I'll go through five different tasks So I'll go through five different tasks that we did and share some of the the that we did and share some of the the that we did and share some of the the findings that we had before and after, findings that we had before and after, findings that we had before and after, especially with as new models have come especially with as new models have come especially with as new models have come out. out. out. So we spent around 2 months evaluating So we spent around 2 months evaluating So we spent around 2 months evaluating orchestrators for our AI pipeline. We orchestrators for our AI pipeline. We orchestrators for our AI pipeline. We looked at five open-source projects and looked at five open-source projects and looked at five open-source projects and we wanted to benchmark and see how we wanted to benchmark and see how we wanted to benchmark and see how effective they were for our use case. effective they were for our use case. effective they were for our use case. And we started this off before deep And we started this off before deep And we started this off before deep research came out as part of Google and research came out as part of Google and research came out as part of Google and OpenAI, so that web search capability to OpenAI, so that web search capability to OpenAI, so that web search capability to do a comprehensive analysis was still do a comprehensive analysis was still do a comprehensive analysis was still not there. not there. not there. Now, after we actually gathered these Now, after we actually gathered these Now, after we actually gathered these requirements, we built out proof of requirements, we built out proof of requirements, we built out proof of concepts with a team of three to make concepts with a team of three to make concepts with a team of three to make sure that we actually got the right sure that we actually got the right sure that we actually got the right results. results. results. Now, I'm pretty confident we could do Now, I'm pretty confident we could do Now, I'm pretty confident we could do this 90% faster now with the tooling this 90% faster now with the tooling this 90% faster now with the tooling that we have. Before we would manually that we have. Before we would manually that we have. Before we would manually go through, use a little bit of AI, but go through, use a little bit of AI, but go through, use a little bit of AI, but put everything into a Confluence doc, put everything into a Confluence doc, put everything into a Confluence doc, and we'd evaluate across 17 different and we'd evaluate across 17 different and we'd evaluate across 17 different criteria that we came up with. criteria that we came up with. criteria that we came up with. Nowadays, we could build a much more Nowadays, we could build a much more Nowadays, we could build a much more agentic workflow to do that, starting agentic workflow to do that, starting agentic workflow to do that, starting off with deep research, uh making sure off with deep research, uh making sure off with deep research, uh making sure that we match that against the problem that we match that against the problem that we match that against the problem statements that we have, statements that we have, statements that we have, creating sub-agents for each of these creating sub-agents for each of these creating sub-agents for each of these criteria and uh products, and then criteria and uh products, and then criteria and uh products, and then finally building POCs and evaluating.

finally building POCs and evaluating. finally building POCs and evaluating. So, So, So, things have changed in the past year and things have changed in the past year and things have changed in the past year and a half where we could actually go much, a half where we could actually go much, a half where we could actually go much, much faster. But, we still have to much faster. But, we still have to much faster. But, we still have to maintain that same set of quality maintain that same set of quality maintain that same set of quality because it's very easy to undergo AI because it's very easy to undergo AI because it's very easy to undergo AI psychosis, where you look at a deep psychosis, where you look at a deep psychosis, where you look at a deep research report that's 20 pages long and research report that's 20 pages long and research report that's 20 pages long and you say, "Wow, this looks good." And you say, "Wow, this looks good." And you say, "Wow, this looks good." And then those features don't actually exist then those features don't actually exist then those features don't actually exist in the product, and you've set yourself in the product, and you've set yourself in the product, and you've set yourself back. Now, after we've done the initial uh Now, after we've done the initial uh orchestration research and model serving orchestration research and model serving orchestration research and model serving research, we wanted to actually commit research, we wanted to actually commit research, we wanted to actually commit in code. This is just a example of what in code. This is just a example of what in code. This is just a example of what happened when we were experimenting. I happened when we were experimenting. I happened when we were experimenting. I was doing some initial research with was doing some initial research with was doing some initial research with Temporal and committed some activities Temporal and committed some activities Temporal and committed some activities and workflow code to make sure that we and workflow code to make sure that we and workflow code to make sure that we can actually replicate what we have in can actually replicate what we have in can actually replicate what we have in the legacy code base. So then, I did what we wanted to do over So then, I did what we wanted to do over a number of iterations and at the time a number of iterations and at the time a number of iterations and at the time gave it to O3 to actually try to gave it to O3 to actually try to gave it to O3 to actually try to implement this code. implement this code. implement this code. And it did it much faster than I would And it did it much faster than I would And it did it much faster than I would be. This refactor took uh 3 hours of be. This refactor took uh 3 hours of be. This refactor took uh 3 hours of back and forth chatting with Incursr, back and forth chatting with Incursr, back and forth chatting with Incursr, but it made 10 major mistakes. So, at but it made 10 major mistakes. So, at but it made 10 major mistakes. So, at the time when we were going through this the time when we were going through this the time when we were going through this refactor, agentic coding was getting refactor, agentic coding was getting refactor, agentic coding was getting better and better, but it still hadn't better and better, but it still hadn't better and better, but it still hadn't reached the point of where it is now.

reached the point of where it is now. reached the point of where it is now. And it was still a very manual process And it was still a very manual process And it was still a very manual process where you had to intervene and actually where you had to intervene and actually where you had to intervene and actually guide the model and manually edit or guide the model and manually edit or guide the model and manually edit or delete code. Now, I reran these benchmarks on some Now, I reran these benchmarks on some some modern models. So, we have Sonnet some modern models. So, we have Sonnet some modern models. So, we have Sonnet 4.6 and Opus 4.8. And things were much 4.6 and Opus 4.8. And things were much 4.6 and Opus 4.8. And things were much faster. Sonnet 4.6 with one additional faster. Sonnet 4.6 with one additional faster. Sonnet 4.6 with one additional iteration was able to solve the task. iteration was able to solve the task. iteration was able to solve the task. And with Opus, it was basically able to And with Opus, it was basically able to And with Opus, it was basically able to one-shot this problem. So, models are one-shot this problem. So, models are one-shot this problem. So, models are getting significantly better along with getting significantly better along with getting significantly better along with harnesses. harnesses. harnesses. And the interesting part here as well is And the interesting part here as well is And the interesting part here as well is that the way that the models interacted that the way that the models interacted that the way that the models interacted has changed substantially as well. has changed substantially as well. has changed substantially as well. Before with O3, there weren't Before with O3, there weren't Before with O3, there weren't substantial tool calls on certain substantial tool calls on certain substantial tool calls on certain categories. And then as we moved into categories. And then as we moved into categories. And then as we moved into Sonnet 4.6 and Opus, we see now that in Sonnet 4.6 and Opus, we see now that in Sonnet 4.6 and Opus, we see now that in modern harnesses, we get sub-agents, we modern harnesses, we get sub-agents, we modern harnesses, we get sub-agents, we get some of those plan calls, we get get some of those plan calls, we get get some of those plan calls, we get different shell commands, and we get different shell commands, and we get different shell commands, and we get different verifications. different verifications. different verifications. And overall, this process, even though And overall, this process, even though And overall, this process, even though the model execution was a little bit the model execution was a little bit the model execution was a little bit more expensive, it was a lot less more expensive, it was a lot less more expensive, it was a lot less manual. So, we could actually accomplish manual. So, we could actually accomplish manual. So, we could actually accomplish a lot more. a lot more. a lot more. So, if I was rebuilding the same task So, if I was rebuilding the same task So, if I was rebuilding the same task that I had for for this refactor, it that I had for for this refactor, it that I had for for this refactor, it would take around 1/5 of the time to would take around 1/5 of the time to would take around 1/5 of the time to accomplish, which is pretty good accomplish, which is pretty good accomplish, which is pretty good progress.

progress. progress. So, I think all of us realize the So, I think all of us realize the So, I think all of us realize the scenario that models are substantially scenario that models are substantially scenario that models are substantially better now than they were before. Now, this is really important because it Now, this is really important because it shapes the way we think about the shapes the way we think about the shapes the way we think about the software development life cycle. We software development life cycle. We software development life cycle. We think about 2025 and the types of work think about 2025 and the types of work think about 2025 and the types of work that we were doing. We were making some that we were doing. We were making some that we were doing. We were making some small changes. We would give specific small changes. We would give specific small changes. We would give specific code snippets to models. We were just code snippets to models. We were just code snippets to models. We were just starting to get into this agentic starting to get into this agentic starting to get into this agentic framework of the type of work we can do. framework of the type of work we can do. framework of the type of work we can do. And now if we provide a well-constructed And now if we provide a well-constructed And now if we provide a well-constructed spectrum model, it could generally spectrum model, it could generally spectrum model, it could generally execute it at a very very high execute it at a very very high execute it at a very very high capability level. capability level. capability level. And we can see this both in anecdotal And we can see this both in anecdotal And we can see this both in anecdotal experiences as well as some of the experiences as well as some of the experiences as well as some of the thought leadership that has been coming thought leadership that has been coming thought leadership that has been coming out of the big labs. The this image is out of the big labs. The this image is out of the big labs. The this image is one from Anthropic. Let me ask the group a question. Who Let me ask the group a question. Who here has kicked off an agent and here has kicked off an agent and here has kicked off an agent and realized that either the prompt, the realized that either the prompt, the realized that either the prompt, the plan, or the requirements were plan, or the requirements were plan, or the requirements were incomplete or missing? incomplete or missing? incomplete or missing? A lot of people, yeah? It's it's very A lot of people, yeah? It's it's very A lot of people, yeah? It's it's very frustrating, right? You you're like, frustrating, right? You you're like, frustrating, right? You you're like, "Okay, I'm ready to go. It's 11:00 p.m. "Okay, I'm ready to go. It's 11:00 p.m. "Okay, I'm ready to go. It's 11:00 p.m. or 5:00 p.m. I'm going to set off an or 5:00 p.m. I'm going to set off an or 5:00 p.m. I'm going to set off an agent and then come back." And then you agent and then come back." And then you agent and then come back." And then you realize there is a there is a critical realize there is a there is a critical realize there is a there is a critical flaw.

flaw. flaw. Now, the reason I bring this up is it's Now, the reason I bring this up is it's Now, the reason I bring this up is it's very important to have a good mental very important to have a good mental very important to have a good mental model and understanding how accurate model and understanding how accurate model and understanding how accurate models can be in accomplishing tasks. models can be in accomplishing tasks. models can be in accomplishing tasks. Who here has seen this meter graph Who here has seen this meter graph Who here has seen this meter graph before? before? before? I think a decent number of people. So, I think a decent number of people. So, I think a decent number of people. So, this is pretty common on actually this is pretty common on actually this is pretty common on actually mapping how much time models can mapping how much time models can mapping how much time models can complete tasks of certain categories complete tasks of certain categories complete tasks of certain categories for. So, the idea being that as models for. So, the idea being that as models for. So, the idea being that as models get better and better, they can do get better and better, they can do get better and better, they can do longer-running tasks. Now, typically longer-running tasks. Now, typically longer-running tasks. Now, typically this graph is shared with the 50% this graph is shared with the 50% this graph is shared with the 50% accuracy rate, but I think it's much accuracy rate, but I think it's much accuracy rate, but I think it's much better to actually look at the 80% better to actually look at the 80% better to actually look at the 80% accuracy rate or higher. And you can see accuracy rate or higher. And you can see accuracy rate or higher. And you can see there you can still see a similar there you can still see a similar there you can still see a similar exponential trend, but we're no longer exponential trend, but we're no longer exponential trend, but we're no longer claiming that models can accomplish claiming that models can accomplish claiming that models can accomplish tasks that would take a human 18+ hours. tasks that would take a human 18+ hours. tasks that would take a human 18+ hours. Now, I actually think it's much better Now, I actually think it's much better Now, I actually think it's much better to measure the accuracy at 90% or 99% to measure the accuracy at 90% or 99% to measure the accuracy at 90% or 99% because this is where the mental model because this is where the mental model because this is where the mental model is most efficient. You construct a plan, is most efficient. You construct a plan, is most efficient. You construct a plan, you create a spec, [snorts] you hand it you create a spec, [snorts] you hand it you create a spec, [snorts] you hand it off to an agent, and you're pretty sure off to an agent, and you're pretty sure off to an agent, and you're pretty sure that it'll get things done, right? You that it'll get things done, right? You that it'll get things done, right? You don't want to be creating a plan or a don't want to be creating a plan or a don't want to be creating a plan or a spec and then have a 50/50 chance of spec and then have a 50/50 chance of spec and then have a 50/50 chance of coming back and knowing that you wasted coming back and knowing that you wasted coming back and knowing that you wasted compute and your attention span. compute and your attention span. compute and your attention span. Now, if you're kicking off a process Now, if you're kicking off a process Now, if you're kicking off a process that is going to take an hour and it has that is going to take an hour and it has that is going to take an hour and it has a 50% chance of completing, there's a a 50% chance of completing, there's a a 50% chance of completing, there's a very high chance you just wasted that very high chance you just wasted that very high chance you just wasted that hour and you could have been doing hour and you could have been doing hour and you could have been doing something different.

something different. something different. Now, if you think about uh broader Now, if you think about uh broader Now, if you think about uh broader evaluation, so Meter does have some more evaluation, so Meter does have some more evaluation, so Meter does have some more information about their frontier models. information about their frontier models. information about their frontier models. So, this is one for for Metis' preview So, this is one for for Metis' preview So, this is one for for Metis' preview that they did roughly a month ago, and that they did roughly a month ago, and that they did roughly a month ago, and you can see here that generally the you can see here that generally the you can see here that generally the success uh rate starts to decline success uh rate starts to decline success uh rate starts to decline significantly at that 4-hour mark, but significantly at that 4-hour mark, but significantly at that 4-hour mark, but even before then at the 15-second mark even before then at the 15-second mark even before then at the 15-second mark or even before the 15-minute mark, there or even before the 15-minute mark, there or even before the 15-minute mark, there are certain tasks that Metis, uh in in are certain tasks that Metis, uh in in are certain tasks that Metis, uh in in all its glory, cannot complete uh all its glory, cannot complete uh all its glory, cannot complete uh effectively and consistently. So, we're effectively and consistently. So, we're effectively and consistently. So, we're making rapid progress in in the AI model making rapid progress in in the AI model making rapid progress in in the AI model space, but we're still not there where space, but we're still not there where space, but we're still not there where you can just kick off an agent and have you can just kick off an agent and have you can just kick off an agent and have something be completed reliably. something be completed reliably. something be completed reliably. So, again, this is really important for So, again, this is really important for So, again, this is really important for your software engineering teams and for your software engineering teams and for your software engineering teams and for you as an IC to understand what is your you as an IC to understand what is your you as an IC to understand what is your mental model and how are you going to mental model and how are you going to mental model and how are you going to contribute to that. I think what's really important is I I think what's really important is I think you've been hearing this think you've been hearing this think you've been hearing this throughout this conference is that there throughout this conference is that there throughout this conference is that there are a number of different frameworks and are a number of different frameworks and are a number of different frameworks and primitives that you need to have primitives that you need to have primitives that you need to have implemented in order to have good implemented in order to have good implemented in order to have good agentic development. And this is no agentic development. And this is no agentic development. And this is no different from from what we found. As we different from from what we found. As we different from from what we found. As we were continuing to mature as an were continuing to mature as an were continuing to mature as an organization and going through our organization and going through our organization and going through our refactor, these are the things that made refactor, these are the things that made refactor, these are the things that made sure that we can implement the solutions sure that we can implement the solutions sure that we can implement the solutions effectively and not waste our time just effectively and not waste our time just effectively and not waste our time just running in loops with models.

So let's go to chapter three. Let's talk So let's go to chapter three. Let's talk about the refactor itself and some of about the refactor itself and some of about the refactor itself and some of the productivity gains that we saw. the productivity gains that we saw. the productivity gains that we saw. So the core idea is that we had these 10 So the core idea is that we had these 10 So the core idea is that we had these 10 repositories, we put them into a mono repositories, we put them into a mono repositories, we put them into a mono repo, and we wanted to build additional repo, and we wanted to build additional repo, and we wanted to build additional features on top of it. So this is this is the result. The So this is this is the result. The previous repos had been around for more previous repos had been around for more previous repos had been around for more than six years, and you can see the than six years, and you can see the than six years, and you can see the progress that was being made it is progress that was being made it is progress that was being made it is pretty slow. Part of it was because of pretty slow. Part of it was because of pretty slow. Part of it was because of the tech debt that was taken on. Other the tech debt that was taken on. Other the tech debt that was taken on. Other parts were because we didn't have AI parts were because we didn't have AI parts were because we didn't have AI coding tools. coding tools. coding tools. And you can see that within the first And you can see that within the first And you can see that within the first six months of the rebuild when we got to six months of the rebuild when we got to six months of the rebuild when we got to parity that we had before, that parity that we had before, that parity that we had before, that steepness of that curve is immense. And steepness of that curve is immense. And steepness of that curve is immense. And it didn't slow down after we we kept it didn't slow down after we we kept it didn't slow down after we we kept shipping. So after that that dotted line shipping. So after that that dotted line shipping. So after that that dotted line in the middle there, we kept adding new in the middle there, we kept adding new in the middle there, we kept adding new and new features into the repository. and new features into the repository. and new features into the repository. And we shipped a lot faster both in And we shipped a lot faster both in And we shipped a lot faster both in terms of the amount of code even though terms of the amount of code even though terms of the amount of code even though that's not a great metric, but also the that's not a great metric, but also the that's not a great metric, but also the commit rate that we had among commit rate that we had among commit rate that we had among developers. And we actually saw that a developers. And we actually saw that a developers. And we actually saw that a lot more developers actually joined in lot more developers actually joined in lot more developers actually joined in into the contributions. So this is a log into the contributions. So this is a log into the contributions. So this is a log graph on the commits that we had from graph on the commits that we had from graph on the commits that we had from the repository initially, the repository initially, the repository initially, and then we slowly on-boarded more and and then we slowly on-boarded more and and then we slowly on-boarded more and more people. And we had fewer commits more people. And we had fewer commits more people. And we had fewer commits because it's much easier to commit code because it's much easier to commit code because it's much easier to commit code when you're just refactoring and when you're just refactoring and when you're just refactoring and replicating something, but we still kept replicating something, but we still kept replicating something, but we still kept up that velocity as we were adding up that velocity as we were adding up that velocity as we were adding product features towards the end. And product features towards the end. And product features towards the end. And now almost every developer within the now almost every developer within the now almost every developer within the company is committing to this new mono company is committing to this new mono company is committing to this new mono repo even though it might not be their repo even though it might not be their repo even though it might not be their area of expertise, but they might need area of expertise, but they might need area of expertise, but they might need to make changes to schemas, API calls, to make changes to schemas, API calls, to make changes to schemas, API calls, and other parts of of the stack.

So, let's go into chapter four. So, let's go into chapter four. Can a modern LLM zero-shot this problem? Can a modern LLM zero-shot this problem? Can a modern LLM zero-shot this problem? Can I say, "Hey, amazing LLM, go Can I say, "Hey, amazing LLM, go Can I say, "Hey, amazing LLM, go refactor this code base." So, I ran this refactor this code base." So, I ran this refactor this code base." So, I ran this experiment with GPT 5.5 extra high, and experiment with GPT 5.5 extra high, and experiment with GPT 5.5 extra high, and I gave it this goal, I gave it this goal, I gave it this goal, giving some of the names of the giving some of the names of the giving some of the names of the repositories with the underlying models repositories with the underlying models repositories with the underlying models and other components, and it completed and other components, and it completed and other components, and it completed its goal in in 10 minutes and 22 its goal in in 10 minutes and 22 its goal in in 10 minutes and 22 seconds. seconds. seconds. And it only wrote 2,000 lines of code, And it only wrote 2,000 lines of code, And it only wrote 2,000 lines of code, which was a little bit fishy. So, I I which was a little bit fishy. So, I I which was a little bit fishy. So, I I dug deeper. dug deeper. dug deeper. Um and it actually just implemented a Um and it actually just implemented a Um and it actually just implemented a bunch of scaffolding and didn't bunch of scaffolding and didn't bunch of scaffolding and didn't implement the models. So, you can see implement the models. So, you can see implement the models. So, you can see here, I did not add a ray serve here, I did not add a ray serve here, I did not add a ray serve deployment or bootstrap command yet, deployment or bootstrap command yet, deployment or bootstrap command yet, right? So, we're still not there where right? So, we're still not there where right? So, we're still not there where models can self-validate and just models can self-validate and just models can self-validate and just one-shot these kinds of problems, but one-shot these kinds of problems, but one-shot these kinds of problems, but we're getting close. we're getting close. we're getting close. I think in in 6 months, we'll get to the I think in in 6 months, we'll get to the I think in in 6 months, we'll get to the point that we can complete pretty point that we can complete pretty point that we can complete pretty substantial refactors, as we saw in the substantial refactors, as we saw in the substantial refactors, as we saw in the in the Stripe example, consistently in the Stripe example, consistently in the Stripe example, consistently across the board. So, get to the core question. Was this So, get to the core question. Was this refactor worthwhile? Should we have refactor worthwhile? Should we have refactor worthwhile? Should we have waited a year to do this refactor as waited a year to do this refactor as waited a year to do this refactor as models and harnesses continue to get models and harnesses continue to get models and harnesses continue to get better, or did it make sense to do it at better, or did it make sense to do it at better, or did it make sense to do it at the time?

the time? the time? Now, Now, Now, I'll say the the other side of the I'll say the the other side of the I'll say the the other side of the argument, right? Things are getting argument, right? Things are getting argument, right? Things are getting substantially better. Models are getting substantially better. Models are getting substantially better. Models are getting better. They can call tools better. We better. They can call tools better. We better. They can call tools better. We have a lot more infrastructure like have a lot more infrastructure like have a lot more infrastructure like sandboxes and monitoring frameworks in sandboxes and monitoring frameworks in sandboxes and monitoring frameworks in order for us to actually understand order for us to actually understand order for us to actually understand what's happening under the hood with what's happening under the hood with what's happening under the hood with these models. So, taking on technical these models. So, taking on technical these models. So, taking on technical debt and refactoring later is getting debt and refactoring later is getting debt and refactoring later is getting exponentially easier as the days go by. exponentially easier as the days go by. exponentially easier as the days go by. Now, the problem is that a lot of times Now, the problem is that a lot of times Now, the problem is that a lot of times when when you build a lot of code and when when you build a lot of code and when when you build a lot of code and you do this kind of development in an AI you do this kind of development in an AI you do this kind of development in an AI AI-native world, it starts looking like AI-native world, it starts looking like AI-native world, it starts looking like some of the legacy code we've we've seen some of the legacy code we've we've seen some of the legacy code we've we've seen in the past. There's a lot of code in the past. There's a lot of code in the past. There's a lot of code written. It's written with low written. It's written with low written. It's written with low performance or quality, and the broader performance or quality, and the broader performance or quality, and the broader problem is people don't actually problem is people don't actually problem is people don't actually understand what's happening there. So, understand what's happening there. So, understand what's happening there. So, if you have some issues within the code if you have some issues within the code if you have some issues within the code base or you want to adjust based on base or you want to adjust based on base or you want to adjust based on customer requirements, it's actually customer requirements, it's actually customer requirements, it's actually much harder to do so. much harder to do so. much harder to do so. So, you do have to make sure that there So, you do have to make sure that there So, you do have to make sure that there are appropriate guardrails, whether or are appropriate guardrails, whether or are appropriate guardrails, whether or not you do a full refactor or only a not you do a full refactor or only a not you do a full refactor or only a partial one. partial one. partial one. So, if you ask me, was it worthwhile? So, if you ask me, was it worthwhile? So, if you ask me, was it worthwhile? I'd say yes. Uh we had built out the I'd say yes. Uh we had built out the I'd say yes. Uh we had built out the patterns that we had earlier with the patterns that we had earlier with the patterns that we had earlier with the number of different repos in in order to number of different repos in in order to number of different repos in in order to match customer requirements and demands.

match customer requirements and demands. match customer requirements and demands. Uh it took an amount of time, but we Uh it took an amount of time, but we Uh it took an amount of time, but we ultimately achieved the goals of the ultimately achieved the goals of the ultimately achieved the goals of the business. Then we came back and business. Then we came back and business. Then we came back and refactored and we were able to refactored and we were able to refactored and we were able to accelerate. Uh we were able to actually accelerate. Uh we were able to actually accelerate. Uh we were able to actually reduce the amount of time the pipeline reduce the amount of time the pipeline reduce the amount of time the pipeline took. We were able to reduce the costs. took. We were able to reduce the costs. took. We were able to reduce the costs. We could support larger files, and now We could support larger files, and now We could support larger files, and now we can ship features that would take we can ship features that would take we can ship features that would take multiple months in under a week. So, the multiple months in under a week. So, the multiple months in under a week. So, the monorepo refactor, the cleanup was monorepo refactor, the cleanup was monorepo refactor, the cleanup was worthwhile, and we have some of the worthwhile, and we have some of the worthwhile, and we have some of the productivity metrics we saw there. productivity metrics we saw there. productivity metrics we saw there. The other part is that beyond just The other part is that beyond just The other part is that beyond just shipping velocity, developers actually shipping velocity, developers actually shipping velocity, developers actually want to work in this codebase. So, want to work in this codebase. So, want to work in this codebase. So, everybody comes along and says, "Hey, everybody comes along and says, "Hey, everybody comes along and says, "Hey, can I work in this codebase? It's much can I work in this codebase? It's much can I work in this codebase? It's much cleaner compared to the other ones. Uh cleaner compared to the other ones. Uh cleaner compared to the other ones. Uh can we actually contribute in a way that can we actually contribute in a way that can we actually contribute in a way that makes sense?" And a lot of the patterns makes sense?" And a lot of the patterns makes sense?" And a lot of the patterns we have adopted here have spread to we have adopted here have spread to we have adopted here have spread to other repos within the company. Now, whether or not you refactor, uh Now, whether or not you refactor, uh the AI delivery system is is a layered the AI delivery system is is a layered the AI delivery system is is a layered approach. Uh you can isolate different approach. Uh you can isolate different approach. Uh you can isolate different parts of your codebase to avoid a full parts of your codebase to avoid a full parts of your codebase to avoid a full refactor, but there's so many components refactor, but there's so many components refactor, but there's so many components that you need to keep in mind. And that you need to keep in mind. And that you need to keep in mind. And hopefully throughout this conference, hopefully throughout this conference, hopefully throughout this conference, you've heard more details about this. you've heard more details about this. you've heard more details about this. But I I really encourage everybody to But I I really encourage everybody to But I I really encourage everybody to think about uh the the business value of think about uh the the business value of think about uh the the business value of delivering a a big refactor and the delivering a a big refactor and the delivering a a big refactor and the tradeoffs of doing it now versus in the tradeoffs of doing it now versus in the tradeoffs of doing it now versus in the future. So, models will continue to get future. So, models will continue to get future. So, models will continue to get better, uh but sometimes it's good to better, uh but sometimes it's good to better, uh but sometimes it's good to pause, build a monorepo, and forge pause, build a monorepo, and forge pause, build a monorepo, and forge ahead.

ahead. ahead. So, thank you everybody. Uh happy to So, thank you everybody. Uh happy to So, thank you everybody. Uh happy to take any questions. take any questions. take any questions. >> [applause] >> Yeah, so the question was before we had >> Yeah, so the question was before we had multiple repos and did we move into multiple repos and did we move into multiple repos and did we move into monorepo? Yes, uh we did that. One of monorepo? Yes, uh we did that. One of monorepo? Yes, uh we did that. One of the things we found now was that models the things we found now was that models the things we found now was that models are much better at navigating multiple are much better at navigating multiple are much better at navigating multiple repos. So, if you put it into a repos. So, if you put it into a repos. So, if you put it into a higher-level folder, right, they could higher-level folder, right, they could higher-level folder, right, they could navigate the the file directory. But, navigate the the file directory. But, navigate the the file directory. But, for doing that end-to-end testing and for doing that end-to-end testing and for doing that end-to-end testing and verification and deployment, it's still verification and deployment, it's still verification and deployment, it's still much harder to do with multiple repos. much harder to do with multiple repos. much harder to do with multiple repos. Uh and if you're building a sandbox Uh and if you're building a sandbox Uh and if you're building a sandbox environment to run sort of a full AI environment to run sort of a full AI environment to run sort of a full AI factory, it's it also takes more time to factory, it's it also takes more time to factory, it's it also takes more time to clone repos and get everything set up. clone repos and get everything set up. clone repos and get everything set up. So. So. So. Yep.

Yeah, so the question was uh when we Yeah, so the question was uh when we define certain features and requirements define certain features and requirements define certain features and requirements that we go back and check them and make that we go back and check them and make that we go back and check them and make changes uh as well as sort of the changes uh as well as sort of the changes uh as well as sort of the guardrails framework. Uh we did. I think guardrails framework. Uh we did. I think guardrails framework. Uh we did. I think we got uh we got uh we got uh 15 out of 17 requirements right when we 15 out of 17 requirements right when we 15 out of 17 requirements right when we were going ahead with the refactor. Um were going ahead with the refactor. Um were going ahead with the refactor. Um and some of the processes that we added and some of the processes that we added and some of the processes that we added for the actual AI engineering for the actual AI engineering for the actual AI engineering verification, that evolved over time. verification, that evolved over time. verification, that evolved over time. So, for example, when we started, plan So, for example, when we started, plan So, for example, when we started, plan mode was just barely coming into Cloud mode was just barely coming into Cloud mode was just barely coming into Cloud Code and didn't exist in Cursor, but we Code and didn't exist in Cursor, but we Code and didn't exist in Cursor, but we adopted it as part of our development adopted it as part of our development adopted it as part of our development life cycle. life cycle. life cycle. Yep. So, our PR reviews were all all human PR So, our PR reviews were all all human PR reviews during that refactor. Uh we did reviews during that refactor. Uh we did reviews during that refactor. Uh we did some local checks where we ran skills to some local checks where we ran skills to some local checks where we ran skills to say, "Hey, review this code. Make sure say, "Hey, review this code. Make sure say, "Hey, review this code. Make sure that it's good." And they're continuing that it's good." And they're continuing that it's good." And they're continuing to get more autonomous as time goes on. to get more autonomous as time goes on. to get more autonomous as time goes on. But, at that point, PRs were really good But, at that point, PRs were really good But, at that point, PRs were really good way for us to build context uh for that way for us to build context uh for that way for us to build context uh for that repo as we only had a few developers repo as we only had a few developers repo as we only had a few developers working on it, and we wanted to make working on it, and we wanted to make working on it, and we wanted to make sure people understood what had gone sure people understood what had gone sure people understood what had gone into the refactor. into the refactor. into the refactor. Yep.

Um in terms of factors, I think that Um in terms of factors, I think that the complexity of the task you can give the complexity of the task you can give the complexity of the task you can give to a model uh is going to be different, to a model uh is going to be different, to a model uh is going to be different, and many more companies will have more and many more companies will have more and many more companies will have more scaffolding in terms of actually doing a scaffolding in terms of actually doing a scaffolding in terms of actually doing a refactor. So, for example, when I showed refactor. So, for example, when I showed refactor. So, for example, when I showed the the life cycle of doing the the the life cycle of doing the the the life cycle of doing the research, the POC work, um research, the POC work, um research, the POC work, um validating the code quality, checking validating the code quality, checking validating the code quality, checking hidden assumptions, like you you thought hidden assumptions, like you you thought hidden assumptions, like you you thought an open-source library had this feature, an open-source library had this feature, an open-source library had this feature, but it was actually in uh a beta, for but it was actually in uh a beta, for but it was actually in uh a beta, for example. I think that is going to be example. I think that is going to be example. I think that is going to be much much faster on top of sort of the much much faster on top of sort of the much much faster on top of sort of the standard refactoring of hey, here's a standard refactoring of hey, here's a standard refactoring of hey, here's a file, rewrite it to to match this uh set file, rewrite it to to match this uh set file, rewrite it to to match this uh set of requirements. All right. Great. Thank everybody. Have All right. Great. Thank everybody. Have a great rest of the conference.