Building uReview, Uber’s Multi-Agent Code Review Engine — Will Bond & Ameya Ketkar, Uber

>> All right, hello everyone. >> All right, hello everyone. My name is Will and uh I'm here to talk My name is Will and uh I'm here to talk My name is Will and uh I'm here to talk to you about automated code review. Uh to you about automated code review. Uh to you about automated code review. Uh my teammate Amir and I work at Uber and my teammate Amir and I work at Uber and my teammate Amir and I work at Uber and we're going to be walking through U we're going to be walking through U we're going to be walking through U Review, a system that Uber has built uh Review, a system that Uber has built uh Review, a system that Uber has built uh to help increase the velocity of our to help increase the velocity of our to help increase the velocity of our software engineering teams. software engineering teams. software engineering teams. Um for a little bit of context about Um for a little bit of context about Um for a little bit of context about what software engineering org at Uber what software engineering org at Uber what software engineering org at Uber looks like, we have thousands of looks like, we have thousands of looks like, we have thousands of software engineers who work across software engineers who work across software engineers who work across hundreds of teams uh located across 12 hundreds of teams uh located across 12 hundreds of teams uh located across 12 different sites and uh they work in different sites and uh they work in different sites and uh they work in primarily one of six language-specific primarily one of six language-specific primarily one of six language-specific monorepos. monorepos. monorepos. As many of you have probably noticed As many of you have probably noticed As many of you have probably noticed over the past 24 months, the volume of over the past 24 months, the volume of over the past 24 months, the volume of PRs, the size of PRs has been growing. PRs, the size of PRs has been growing. PRs, the size of PRs has been growing. One of the ways that that's been exposed One of the ways that that's been exposed One of the ways that that's been exposed to us has been through uh the metric to us has been through uh the metric to us has been through uh the metric that we track of the first time to that we track of the first time to that we track of the first time to review. Back in 2024, we were seeing review. Back in 2024, we were seeing review. Back in 2024, we were seeing that engineers would get their first that engineers would get their first that engineers would get their first review within 3 hours. review within 3 hours. review within 3 hours. Now in 2026, that has grown to 9 hours Now in 2026, that has grown to 9 hours Now in 2026, that has grown to 9 hours uh in addition to all of the volume uh in addition to all of the volume uh in addition to all of the volume changes. So, in short, code review is changes. So, in short, code review is changes. So, in short, code review is now the bottleneck that we are running now the bottleneck that we are running now the bottleneck that we are running into.

into. into. Um specifically around automated code Um specifically around automated code Um specifically around automated code review, uh there are there are various review, uh there are there are various review, uh there are there are various options available in the industry, uh options available in the industry, uh options available in the industry, uh but Uber spent the time to invest in but Uber spent the time to invest in but Uber spent the time to invest in building an in-house solution due to building an in-house solution due to building an in-house solution due to some of the constraints that we have. some of the constraints that we have. some of the constraints that we have. One of those is uh we currently use One of those is uh we currently use One of those is uh we currently use Fabricator and have for a long time and Fabricator and have for a long time and Fabricator and have for a long time and are in the process of migrating to are in the process of migrating to are in the process of migrating to GitHub. Uh most of the solutions do not GitHub. Uh most of the solutions do not GitHub. Uh most of the solutions do not provide support for Fabricator. Um in provide support for Fabricator. Um in provide support for Fabricator. Um in addition, if you were at the previous addition, if you were at the previous addition, if you were at the previous talk, you saw Uday and Adam talking talk, you saw Uday and Adam talking talk, you saw Uday and Adam talking about the agentic SDLC. A big part of about the agentic SDLC. A big part of about the agentic SDLC. A big part of what we want to do is bring a consistent what we want to do is bring a consistent what we want to do is bring a consistent code review experience to the inner loop code review experience to the inner loop code review experience to the inner loop so that our agents are getting the same so that our agents are getting the same so that our agents are getting the same code review, the same rules, everything code review, the same rules, everything code review, the same rules, everything applied as our humans do. applied as our humans do. applied as our humans do. With hundreds of teams across the With hundreds of teams across the With hundreds of teams across the company, we can't have centralized company, we can't have centralized company, we can't have centralized management of our code reviews, our management of our code reviews, our management of our code reviews, our customizations, and our rules, and even customizations, and our rules, and even customizations, and our rules, and even the knowledge that goes into those code the knowledge that goes into those code the knowledge that goes into those code reviews. We need to distribute that. So, reviews. We need to distribute that. So, reviews. We need to distribute that. So, we have a need for we have a need for we have a need for plugging into existing team ownership plugging into existing team ownership plugging into existing team ownership system rather than trying to replicate system rather than trying to replicate system rather than trying to replicate that externally.

that externally. that externally. Uh finally, with the volume of code Uh finally, with the volume of code Uh finally, with the volume of code reviews that we perform, we need the reviews that we perform, we need the reviews that we perform, we need the ability to take factors like the risk ability to take factors like the risk ability to take factors like the risk profile and the complexity of a code profile and the complexity of a code profile and the complexity of a code change and factor that in when deciding change and factor that in when deciding change and factor that in when deciding how we're going to run a code review. how we're going to run a code review. how we're going to run a code review. Not all code gets the exact same review. Not all code gets the exact same review. Not all code gets the exact same review. And then finally, consistency. We need And then finally, consistency. We need And then finally, consistency. We need to make sure that we have security and to make sure that we have security and to make sure that we have security and compliance reviews run across compliance reviews run across compliance reviews run across everything. We can't rely on teams everything. We can't rely on teams everything. We can't rely on teams hoping to run the skill the code review hoping to run the skill the code review hoping to run the skill the code review skill that happens. We need reliability skill that happens. We need reliability skill that happens. We need reliability there. there. there. With all that said, I wanted to give you With all that said, I wanted to give you With all that said, I wanted to give you an overview of the architecture of what an overview of the architecture of what an overview of the architecture of what you review looks like. We'll talk about you review looks like. We'll talk about you review looks like. We'll talk about a couple of the big pieces, and then a couple of the big pieces, and then a couple of the big pieces, and then we're going to dive into a few focus we're going to dive into a few focus we're going to dive into a few focus areas. areas. areas. At the top, you'll notice that we have At the top, you'll notice that we have At the top, you'll notice that we have our code review surface areas, GitHub, our code review surface areas, GitHub, our code review surface areas, GitHub, Fabricator, and the agent loop. Fabricator, and the agent loop. Fabricator, and the agent loop. These all feed into you review service. These all feed into you review service. These all feed into you review service. These This takes in requests for These This takes in requests for These This takes in requests for reviews. reviews. reviews. It brings in feedback from users, and it It brings in feedback from users, and it It brings in feedback from users, and it routes it. We have a number of different routes it. We have a number of different routes it. We have a number of different generators. Now, these generators are generators. Now, these generators are generators. Now, these generators are tuned for different performance and cost tuned for different performance and cost tuned for different performance and cost avenues. There are We also have the avenues. There are We also have the avenues. There are We also have the ability to plug into third-party code ability to plug into third-party code ability to plug into third-party code review systems so that we can compare review systems so that we can compare review systems so that we can compare ourselves to what's available more ourselves to what's available more ourselves to what's available more broadly.

broadly. broadly. Finally, with all these different Finally, with all these different Finally, with all these different generators, we might be might be generators, we might be might be generators, we might be might be duplicating comments, and we can duplicating comments, and we can duplicating comments, and we can actually create quite a high volume of actually create quite a high volume of actually create quite a high volume of comments. If you've ever used AI to to comments. If you've ever used AI to to comments. If you've ever used AI to to run a code review, you've probably seen run a code review, you've probably seen run a code review, you've probably seen that. So, we run through a number of that. So, we run through a number of that. So, we run through a number of steps in the post-processing where we steps in the post-processing where we steps in the post-processing where we both rate, categorize, filter, and both rate, categorize, filter, and both rate, categorize, filter, and deduplicate comments so that our deduplicate comments so that our deduplicate comments so that our engineers get only the highest engineers get only the highest engineers get only the highest confidence comments that are actionable confidence comments that are actionable confidence comments that are actionable for them to work on. for them to work on. for them to work on. You'll also notice along the bottom we You'll also notice along the bottom we You'll also notice along the bottom we talk a little bit about feedback in our talk a little bit about feedback in our talk a little bit about feedback in our evaluation. But, with this context of evaluation. But, with this context of evaluation. But, with this context of the overall system, I'm now going to the overall system, I'm now going to the overall system, I'm now going to hand it off to Ameya to dive into our hand it off to Ameya to dive into our hand it off to Ameya to dive into our first focus area. >> Hello. >> Hello. Hello, everyone. So, I will be talking Hello, everyone. So, I will be talking Hello, everyone. So, I will be talking about how we evolve U review with about how we evolve U review with about how we evolve U review with observability and evaluation. observability and evaluation. observability and evaluation. So, U review had a very humble So, U review had a very humble So, U review had a very humble beginning. Basically, it was a single beginning. Basically, it was a single beginning. Basically, it was a single prompt that you should do logic checks prompt that you should do logic checks prompt that you should do logic checks per file, a simple agent which used to per file, a simple agent which used to per file, a simple agent which used to do thorough review. And we had a do thorough review. And we had a do thorough review. And we had a dispatcher to decide whether to go which dispatcher to decide whether to go which dispatcher to decide whether to go which generator to choose.

generator to choose. generator to choose. Even what we used to collect as Even what we used to collect as Even what we used to collect as observability was very surface-level. We observability was very surface-level. We observability was very surface-level. We used to collect cost. We used to run an used to collect cost. We used to run an used to collect cost. We used to run an NPS survey, have Google Forms being NPS survey, have Google Forms being NPS survey, have Google Forms being filled, Slack support. And with all of filled, Slack support. And with all of filled, Slack support. And with all of this, we saw that our quality to cost this, we saw that our quality to cost this, we saw that our quality to cost ratio was like all over the place. Like, ratio was like all over the place. Like, ratio was like all over the place. Like, our goal is to be in the second our goal is to be in the second our goal is to be in the second quadrant, that is the top left quadrant, quadrant, that is the top left quadrant, quadrant, that is the top left quadrant, but but but you can see we were all over the place. you can see we were all over the place. you can see we were all over the place. Then what we did is that we started Then what we did is that we started Then what we did is that we started collecting more data. So, we started collecting more data. So, we started collecting more data. So, we started collecting the sentiments of the replies collecting the sentiments of the replies collecting the sentiments of the replies that were made to the U review that were made to the U review that were made to the U review that the U review uh call that the U review uh call that the U review uh call you know, the U review agent got from you know, the U review agent got from you know, the U review agent got from the developers. So, we categorized them the developers. So, we categorized them the developers. So, we categorized them into positive, negative. We classified into positive, negative. We classified into positive, negative. We classified them into them into them into various categories, and we found a bunch various categories, and we found a bunch various categories, and we found a bunch a lot of classes of bugs and issues that a lot of classes of bugs and issues that a lot of classes of bugs and issues that we could actually solve. And with that, we could actually solve. And with that, we could actually solve. And with that, we improved the system, and we were able we improved the system, and we were able we improved the system, and we were able to move a large number of PRs to a high to move a large number of PRs to a high to move a large number of PRs to a high quality to cost ratio. quality to cost ratio. quality to cost ratio. Um but, we still felt that this was not Um but, we still felt that this was not Um but, we still felt that this was not enough.

enough. enough. We need to know more of how the review We need to know more of how the review We need to know more of how the review is done. So we started tracking things is done. So we started tracking things is done. So we started tracking things like address rate. So basically when a U like address rate. So basically when a U like address rate. So basically when a U review comment is made, does the review comment is made, does the review comment is made, does the developer go and actually address the developer go and actually address the developer go and actually address the comment? We started tracking that. And comment? We started tracking that. And comment? We started tracking that. And then we also then we also then we also started doing more like a runtime started doing more like a runtime started doing more like a runtime profile, which is like the agent profile, which is like the agent profile, which is like the agent trajectory, trajectory, trajectory, which told us which told us which told us why the agent is doing what it what it why the agent is doing what it what it why the agent is doing what it what it did. We get to know what tools calls it did. We get to know what tools calls it did. We get to know what tools calls it made. We get to know what thinking made. We get to know what thinking made. We get to know what thinking process it had. And then with that process it had. And then with that process it had. And then with that insight, we were able to actually tune insight, we were able to actually tune insight, we were able to actually tune our runtime, tune our performance such our runtime, tune our performance such our runtime, tune our performance such that the agent could very quickly give that the agent could very quickly give that the agent could very quickly give us us us high-quality results at a low cost. high-quality results at a low cost. high-quality results at a low cost. One of the biggest learnings in this One of the biggest learnings in this One of the biggest learnings in this process was like the model doesn't know process was like the model doesn't know process was like the model doesn't know that it's wrong. It always confidently that it's wrong. It always confidently that it's wrong. It always confidently says 100% sure that yeah, this is the says 100% sure that yeah, this is the says 100% sure that yeah, this is the review for your code. Go ahead. But we review for your code. Go ahead. But we review for your code. Go ahead. But we saw that no, it actually needs a lot of saw that no, it actually needs a lot of saw that no, it actually needs a lot of guidance from the teams because each guidance from the teams because each guidance from the teams because each team has its own style guide, its own team has its own style guide, its own team has its own style guide, its own patterns or like anti-patterns that they patterns or like anti-patterns that they patterns or like anti-patterns that they want to look for. So that all should be want to look for. So that all should be want to look for. So that all should be like baked into the agent. And we also like baked into the agent. And we also like baked into the agent. And we also realized that we need to have guardrails realized that we need to have guardrails realized that we need to have guardrails for the agent. So we need to tell the for the agent. So we need to tell the for the agent. So we need to tell the agent what not to waste turns doing.

agent what not to waste turns doing. agent what not to waste turns doing. Like code review is something that has Like code review is something that has Like code review is something that has to happen in like a specific time span. to happen in like a specific time span. to happen in like a specific time span. And then if it starts spending time And then if it starts spending time And then if it starts spending time doing things that it should not be doing things that it should not be doing things that it should not be doing, uh leads to a bad quality code doing, uh leads to a bad quality code doing, uh leads to a bad quality code review. review. review. Uh second focus area for U review has Uh second focus area for U review has Uh second focus area for U review has been been been nations. nations. nations. We We We We went very deep on team customizations We went very deep on team customizations We went very deep on team customizations because as we'll presented that we have because as we'll presented that we have because as we'll presented that we have hundreds of teams and everyone has like hundreds of teams and everyone has like hundreds of teams and everyone has like their own way or their own thing for their own way or their own thing for their own way or their own thing for code review. code review. code review. So our review stack is pretty So our review stack is pretty So our review stack is pretty straightforward. We have single-file straightforward. We have single-file straightforward. We have single-file reviewers and multi-file reviewers. reviewers and multi-file reviewers. reviewers and multi-file reviewers. Uh, we basically do a general purpose Uh, we basically do a general purpose Uh, we basically do a general purpose "Hey, find me all logic bugs per file" "Hey, find me all logic bugs per file" "Hey, find me all logic bugs per file" uh, kind of a review. And uh, then we uh, kind of a review. And uh, then we uh, kind of a review. And uh, then we also do a deep review because we have also do a deep review because we have also do a deep review because we have like six mono repos. So, all these mono like six mono repos. So, all these mono like six mono repos. So, all these mono repos have their own anti-pattern style repos have their own anti-pattern style repos have their own anti-pattern style guides and all baked into this agent guides and all baked into this agent guides and all baked into this agent review which does have a nice multi-file review which does have a nice multi-file review which does have a nice multi-file review. review. review. But, then we extended it further But, then we extended it further But, then we extended it further uh, basically to AI linters. These are uh, basically to AI linters. These are uh, basically to AI linters. These are basically few shot uh, basically few shot uh, basically few shot uh, AI problem uh, or like a few shots uh, AI problem uh, or like a few shots uh, AI problem uh, or like a few shots uh, system where uh, developers can system where uh, developers can system where uh, developers can basically basically basically kind of deterministically get more kind of deterministically get more kind of deterministically get more context and then run rules with that context and then run rules with that context and then run rules with that context and like a file and find some context and like a file and find some context and like a file and find some uh, systematic and mechanical issues.

uh, systematic and mechanical issues. uh, systematic and mechanical issues. And finally uh, the most powerful thing And finally uh, the most powerful thing And finally uh, the most powerful thing is the custom agent uh, where the teams is the custom agent uh, where the teams is the custom agent uh, where the teams could basically define their own custom could basically define their own custom could basically define their own custom agent, link it to like a knowledge base, agent, link it to like a knowledge base, agent, link it to like a knowledge base, uh, link it to their past PRs, have like uh, link it to their past PRs, have like uh, link it to their past PRs, have like a skill to do the review, and so on. a skill to do the review, and so on. a skill to do the review, and so on. But, uh, all of this was not simple But, uh, all of this was not simple But, uh, all of this was not simple because we had to actually uh, piggyback because we had to actually uh, piggyback because we had to actually uh, piggyback on our uh, ownership model which is at on our uh, ownership model which is at on our uh, ownership model which is at Uber uh, so that we can like very Uber uh, so that we can like very Uber uh, so that we can like very logically roll out to all the teams. logically roll out to all the teams. logically roll out to all the teams. Uh, we had to basically do a Uh, we had to basically do a Uh, we had to basically do a uh, what do you say? Co-locate the uh, what do you say? Co-locate the uh, what do you say? Co-locate the customizations next to where the customizations next to where the customizations next to where the developers write their code so that they developers write their code so that they developers write their code so that they can like quickly uh, keep updating these can like quickly uh, keep updating these can like quickly uh, keep updating these customizations. We had to implement a customizations. We had to implement a customizations. We had to implement a smart deterministic uh, routing so that smart deterministic uh, routing so that smart deterministic uh, routing so that we could route which team gets what kind we could route which team gets what kind we could route which team gets what kind of review with which model, what kind of of review with which model, what kind of of review with which model, what kind of generators, and so on. generators, and so on. generators, and so on. And finally uh, the hard thing was like And finally uh, the hard thing was like And finally uh, the hard thing was like we had to actually surface all of this we had to actually surface all of this we had to actually surface all of this observability that I talked before, like observability that I talked before, like observability that I talked before, like the agent trajectory, addressal rate, the agent trajectory, addressal rate, the agent trajectory, addressal rate, uh, sentiment analysis back to the uh, sentiment analysis back to the uh, sentiment analysis back to the teams. So, so that the teams could teams. So, so that the teams could teams. So, so that the teams could actually understand that "Oh, I wrote actually understand that "Oh, I wrote actually understand that "Oh, I wrote this rule, but maybe not a lot of this rule, but maybe not a lot of this rule, but maybe not a lot of developers are liking it in my team, so developers are liking it in my team, so developers are liking it in my team, so let me go and update it." And then we let me go and update it." And then we let me go and update it." And then we had to give Bubble up that kind of had to give Bubble up that kind of had to give Bubble up that kind of observability to all the people who are observability to all the people who are observability to all the people who are contributing to the platform.

contributing to the platform. contributing to the platform. Uh Uh Uh one thing that we learned is that one thing that we learned is that one thing that we learned is that actually writing the skill was very actually writing the skill was very actually writing the skill was very easy. Like teams just very quickly wrote easy. Like teams just very quickly wrote easy. Like teams just very quickly wrote a skill by asking Claude to write one, a skill by asking Claude to write one, a skill by asking Claude to write one, go over my go over my go over my previous PR reviews and write me a previous PR reviews and write me a previous PR reviews and write me a skill. But the hard part was how to run skill. But the hard part was how to run skill. But the hard part was how to run these skills at scale with consistent these skills at scale with consistent these skills at scale with consistent quality and low cost. And that required quality and low cost. And that required quality and low cost. And that required a lot of iterations not only from the a lot of iterations not only from the a lot of iterations not only from the U-Review team side, but also like for U-Review team side, but also like for U-Review team side, but also like for each team who was trying to write these each team who was trying to write these each team who was trying to write these rules. Uh rules. Uh rules. Uh in results, we basically uh see that, in results, we basically uh see that, in results, we basically uh see that, you know, uh U-Review does like around you know, uh U-Review does like around you know, uh U-Review does like around 25,000 comments a week. And uh we get 25,000 comments a week. And uh we get 25,000 comments a week. And uh we get 10% of them actually get some feedback. 10% of them actually get some feedback. 10% of them actually get some feedback. And only 4% of the PRs actually get some And only 4% of the PRs actually get some And only 4% of the PRs actually get some negative feedback. Uh we also saw that negative feedback. Uh we also saw that negative feedback. Uh we also saw that um um um the overall addressal rate was uh around the overall addressal rate was uh around the overall addressal rate was uh around 67% 67% 67% and almost three quarters of the high and almost three quarters of the high and almost three quarters of the high severity issues uh severity issues uh severity issues uh were usually addressed by the were usually addressed by the were usually addressed by the developers, which shows that U-Review developers, which shows that U-Review developers, which shows that U-Review actually adds some value to the entire actually adds some value to the entire actually adds some value to the entire development life cycle. And then uh with development life cycle. And then uh with development life cycle. And then uh with all the observability and all the observability and all the observability and uh evals that I showed that I went uh evals that I showed that I went uh evals that I showed that I went through, we saw that against like a very through, we saw that against like a very through, we saw that against like a very naive implementation, our costs were naive implementation, our costs were naive implementation, our costs were down by 60% and our quality and our down by 60% and our quality and our down by 60% and our quality and our accuracy was up by uh around 70%.

accuracy was up by uh around 70%. accuracy was up by uh around 70%. Uh Uh Uh for a last focus area, I'll give the mic for a last focus area, I'll give the mic for a last focus area, I'll give the mic back to Will and he will go over the back to Will and he will go over the back to Will and he will go over the inner versus outer loop. >> Awesome. So, now that we've talked about >> Awesome. So, now that we've talked about uh some of the details of actually uh some of the details of actually uh some of the details of actually implementing high-quality reviews, it implementing high-quality reviews, it implementing high-quality reviews, it kind of brings us to the last area, kind of brings us to the last area, kind of brings us to the last area, which is where we start talking about which is where we start talking about which is where we start talking about where things are going, right? With where things are going, right? With where things are going, right? With moving to the Agentech SDLC, moving to the Agentech SDLC, moving to the Agentech SDLC, we're moving software into a model where we're moving software into a model where we're moving software into a model where engineers are interacting with the code engineers are interacting with the code engineers are interacting with the code less. less. less. They're often times not as involved in They're often times not as involved in They're often times not as involved in authoring the code. Uh currently, we authoring the code. Uh currently, we authoring the code. Uh currently, we still have uh humans approving the code, still have uh humans approving the code, still have uh humans approving the code, uh but we see a a short path in the near uh but we see a a short path in the near uh but we see a a short path in the near future to a percentage of our code future to a percentage of our code future to a percentage of our code landing automatically, having automatic landing automatically, having automatic landing automatically, having automatic approvals, right? The various parts of approvals, right? The various parts of approvals, right? The various parts of the industry are already moving there. the industry are already moving there. the industry are already moving there. Um Um Um Part of the way along the process was Part of the way along the process was Part of the way along the process was figuring out by having our single code figuring out by having our single code figuring out by having our single code review platform, what did we need to review platform, what did we need to review platform, what did we need to tune for the various audiences that are tune for the various audiences that are tune for the various audiences that are actually getting these code reviews? Um actually getting these code reviews? Um actually getting these code reviews? Um you know, the interface, that's one area you know, the interface, that's one area you know, the interface, that's one area that's sort of intuitive there. Uh one that's sort of intuitive there. Uh one that's sort of intuitive there. Uh one thing that might be less intuitive is thing that might be less intuitive is thing that might be less intuitive is around accuracy. Uh with the inner loop, around accuracy. Uh with the inner loop, around accuracy. Uh with the inner loop, our accuracy needs actually need to go our accuracy needs actually need to go our accuracy needs actually need to go up, up, up, or else we can result in uh dealing with or else we can result in uh dealing with or else we can result in uh dealing with cavitation of an agent where it fixes cavitation of an agent where it fixes cavitation of an agent where it fixes something, goes back, gets another code something, goes back, gets another code something, goes back, gets another code review, and has to kind of like fix review, and has to kind of like fix review, and has to kind of like fix backwards because the quality of the

backwards because the quality of the backwards because the quality of the comment was low. Um comment was low. Um comment was low. Um The one of the other interesting things The one of the other interesting things The one of the other interesting things is agents are more than happy to go is agents are more than happy to go is agents are more than happy to go through and fix 100 nits on a pull through and fix 100 nits on a pull through and fix 100 nits on a pull request where your engineers really get request where your engineers really get request where your engineers really get frustrated in situations like that. Um frustrated in situations like that. Um frustrated in situations like that. Um but probably the most interesting aspect but probably the most interesting aspect but probably the most interesting aspect of this transition is the feedback. As of this transition is the feedback. As of this transition is the feedback. As you can see, quite a bit of what went you can see, quite a bit of what went you can see, quite a bit of what went into getting high-quality code reviews into getting high-quality code reviews into getting high-quality code reviews at Uber was bringing the human feedback at Uber was bringing the human feedback at Uber was bringing the human feedback into the system and using that to figure into the system and using that to figure into the system and using that to figure out how to tune our prompts, how to tune out how to tune our prompts, how to tune out how to tune our prompts, how to tune our agents. our agents. our agents. Uh and so as we move to a model where Uh and so as we move to a model where Uh and so as we move to a model where humans are less in the loop, where humans are less in the loop, where humans are less in the loop, where software engineering is moving to an software engineering is moving to an software engineering is moving to an agentic model, agentic model, agentic model, we're effectively going to a place where we're effectively going to a place where we're effectively going to a place where we're starting to talk about are we we're starting to talk about are we we're starting to talk about are we going to kill the outer loop? Is the going to kill the outer loop? Is the going to kill the outer loop? Is the human engineer not going to be involved human engineer not going to be involved human engineer not going to be involved in the code review. in the code review. in the code review. Some people are already here. Some people are already here. Some people are already here. Now, Now, Now, with the feedback taken into with the feedback taken into with the feedback taken into consideration, consideration, consideration, you start wondering, all right, you start wondering, all right, you start wondering, all right, what could this result in, right? I'll what could this result in, right? I'll what could this result in, right? I'll let your imagination go there in terms let your imagination go there in terms let your imagination go there in terms of quality degradation, slop, and so of quality degradation, slop, and so of quality degradation, slop, and so forth. But, rather than killing the forth. But, rather than killing the forth. But, rather than killing the outer loop, I think that we believe and outer loop, I think that we believe and outer loop, I think that we believe and the industry has just started to really the industry has just started to really the industry has just started to really kind of coalesce on this idea that we're kind of coalesce on this idea that we're kind of coalesce on this idea that we're really expanding the outer loop. Rather really expanding the outer loop. Rather really expanding the outer loop. Rather than removing humans from the code than removing humans from the code than removing humans from the code review process, we are moving their review process, we are moving their review process, we are moving their responsibilities up a layer.

responsibilities up a layer. responsibilities up a layer. Rather than them dealing with the Rather than them dealing with the Rather than them dealing with the details of the implementation, the agent details of the implementation, the agent details of the implementation, the agent is great at writing the software. is great at writing the software. is great at writing the software. The agent is getting much, much better The agent is getting much, much better The agent is getting much, much better at reviewing the software as a human at reviewing the software as a human at reviewing the software as a human would. But now, as software engineers, would. But now, as software engineers, would. But now, as software engineers, we still are going to have an outer we still are going to have an outer we still are going to have an outer loop. It's just going to look a little loop. It's just going to look a little loop. It's just going to look a little different. Instead of you worrying about different. Instead of you worrying about different. Instead of you worrying about the optimization of the performance and the optimization of the performance and the optimization of the performance and the API compatibility, you're going to the API compatibility, you're going to the API compatibility, you're going to be thinking more about architecture in be thinking more about architecture in be thinking more about architecture in your code reviews. You're going to have your code reviews. You're going to have your code reviews. You're going to have time to focus on the domain expertise time to focus on the domain expertise time to focus on the domain expertise that you have and product thinking. So, that you have and product thinking. So, that you have and product thinking. So, we believe that as we adopt this we believe that as we adopt this we believe that as we adopt this automated uh code review, this is going automated uh code review, this is going automated uh code review, this is going to be the result of how our engineers to be the result of how our engineers to be the result of how our engineers are interacting with the system and are interacting with the system and are interacting with the system and guiding it. And that's it. Thank you so much for And that's it. Thank you so much for coming. Thanks.