How to Kill the Code Review — Ankit Jain, Aviator

>> Okay. >> Okay. Ooh, hello. Hey everyone. Ooh, hello. Hey everyone. Ooh, hello. Hey everyone. Thanks for joining in. Today, Thanks for joining in. Today, Thanks for joining in. Today, uh we will be talking about how to kill uh we will be talking about how to kill uh we will be talking about how to kill the code reviews. Everyone's favorite the code reviews. Everyone's favorite the code reviews. Everyone's favorite topic. I'm Ankit, co-founder of Aviator. topic. I'm Ankit, co-founder of Aviator. topic. I'm Ankit, co-founder of Aviator. At Aviator, we are building AI code At Aviator, we are building AI code At Aviator, we are building AI code verification platform. So, we'll bring verification platform. So, we'll bring verification platform. So, we'll bring in some of the ideas and concepts that in some of the ideas and concepts that in some of the ideas and concepts that we talked about in the we talked about in the we talked about in the uh that we build in our product. Uh but uh that we build in our product. Uh but uh that we build in our product. Uh but first, let's dive into a little bit. So, first, let's dive into a little bit. So, first, let's dive into a little bit. So, a few months ago, I wrote a post on a few months ago, I wrote a post on a few months ago, I wrote a post on LinkedIn space about uh how to kill code LinkedIn space about uh how to kill code LinkedIn space about uh how to kill code review, creating a framework, a review, creating a framework, a review, creating a framework, a five-layer trust model. five-layer trust model. five-layer trust model. So, this model was focused around how do So, this model was focused around how do So, this model was focused around how do we actually layer by layer build trust we actually layer by layer build trust we actually layer by layer build trust into the code that can then be merged without needing line-by-line merged without needing line-by-line review. And I got some things right, and review. And I got some things right, and review. And I got some things right, and I got some things wrong. So, this talk I got some things wrong. So, this talk I got some things wrong. So, this talk will be about just really diving a bit will be about just really diving a bit will be about just really diving a bit more more into it. I'm not going to talk more more into it. I'm not going to talk more more into it. I'm not going to talk about specific layers, but we will talk about specific layers, but we will talk about specific layers, but we will talk about some of the concepts that emerged about some of the concepts that emerged about some of the concepts that emerged from this session.

from this session. from this session. So, let's just talk about the problem. So, let's just talk about the problem. So, let's just talk about the problem. We are looking today at the volume of We are looking today at the volume of We are looking today at the volume of code is increasing every day, and we are code is increasing every day, and we are code is increasing every day, and we are struggling to keep up. struggling to keep up. struggling to keep up. So, when we think about like how long So, when we think about like how long So, when we think about like how long will it take us to actually will it take us to actually will it take us to actually stop reading code line by line? stop reading code line by line? stop reading code line by line? And the reality is we've already stopped And the reality is we've already stopped And the reality is we've already stopped reviewing it. reviewing it. reviewing it. There is 861% There is 861% There is 861% code churn. That means we are like code churn. That means we are like code churn. That means we are like producing more code. The incidents to PR producing more code. The incidents to PR producing more code. The incidents to PR ratio is increasing. That means even if ratio is increasing. That means even if ratio is increasing. That means even if you're doing reviews today, they're not you're doing reviews today, they're not you're doing reviews today, they're not effective. effective. effective. So, the medium time of review is So, the medium time of review is So, the medium time of review is increasing. We have just increased the increasing. We have just increased the increasing. We have just increased the bottleneck to from the coding now kind bottleneck to from the coding now kind bottleneck to from the coding now kind of like coding is solved to now of like coding is solved to now of like coding is solved to now reviewing, where everything just gets reviewing, where everything just gets reviewing, where everything just gets stuck there. You're spending four x the stuck there. You're spending four x the stuck there. You're spending four x the time that you were spending before just time that you were spending before just time that you were spending before just waiting for the reviews. waiting for the reviews. waiting for the reviews. And today, over 30% of changes are And today, over 30% of changes are And today, over 30% of changes are actually getting merged without a review actually getting merged without a review actually getting merged without a review at all. at all. at all. So, let's just think about it a little bit.

let's just think about it a little bit. Think about AI reviews. So, everyone is Think about AI reviews. So, everyone is Think about AI reviews. So, everyone is probably using some form of AI reviews probably using some form of AI reviews probably using some form of AI reviews today. today. today. When AI write the code and AI review the When AI write the code and AI review the When AI write the code and AI review the code, why are we doing it in a UI? code, why are we doing it in a UI? code, why are we doing it in a UI? Right? Like we open a GitHub, there's Right? Like we open a GitHub, there's Right? Like we open a GitHub, there's kind of like, you know, maybe two or kind of like, you know, maybe two or kind of like, you know, maybe two or three AI coding agents who are like three AI coding agents who are like three AI coding agents who are like doing the reviews. You know, you the doing the reviews. You know, you the doing the reviews. You know, you the review goes back passed to the user and review goes back passed to the user and review goes back passed to the user and or the agent and it gets resolved and or the agent and it gets resolved and or the agent and it gets resolved and they kind of like you're doing back and they kind of like you're doing back and they kind of like you're doing back and forth with the agent. forth with the agent. forth with the agent. Where is human here in the loop? Like Where is human here in the loop? Like Where is human here in the loop? Like you're eventually just looking at, okay, you're eventually just looking at, okay, you're eventually just looking at, okay, you know, if AI has reviewed it, most of you know, if AI has reviewed it, most of you know, if AI has reviewed it, most of the things probably found, let's just the things probably found, let's just the things probably found, let's just kind of like, you know, do a skimming of kind of like, you know, do a skimming of kind of like, you know, do a skimming of it and merge it. it and merge it. it and merge it. So, when AI reviews and nobody reads, we So, when AI reviews and nobody reads, we So, when AI reviews and nobody reads, we have configured the wrong thing. have configured the wrong thing. have configured the wrong thing. So, let's just take a step back. Code So, let's just take a step back. Code So, let's just take a step back. Code reviews are not very old. They are maybe reviews are not very old. They are maybe reviews are not very old. They are maybe 15 to 20 years ago. In 2006 was when 15 to 20 years ago. In 2006 was when 15 to 20 years ago. In 2006 was when Google launched Mondrian internally and Google launched Mondrian internally and Google launched Mondrian internally and they made formal code review as a thing. they made formal code review as a thing. they made formal code review as a thing. If you think about it, Windows If you think about it, Windows If you think about it, Windows back in the days, the first versions back in the days, the first versions back in the days, the first versions were actually built without reviews.

were actually built without reviews. were actually built without reviews. But if we But if we But if we look carefully, code review is not just look carefully, code review is not just look carefully, code review is not just about code reviews. about code reviews. about code reviews. You know, obviously we are looking at it You know, obviously we are looking at it You know, obviously we are looking at it like catching bugs, understanding kind like catching bugs, understanding kind like catching bugs, understanding kind of conventions, identifying security of conventions, identifying security of conventions, identifying security issues. issues. issues. But, code review is also about But, code review is also about But, code review is also about alignment. alignment. alignment. And this was one piece which was missing And this was one piece which was missing And this was one piece which was missing from my five layers model that I talked from my five layers model that I talked from my five layers model that I talked about a few months ago. about a few months ago. about a few months ago. So, a big part of code reviews is So, a big part of code reviews is So, a big part of code reviews is knowledge sharing, mentorship, knowledge sharing, mentorship, knowledge sharing, mentorship, architectural feedback, architectural feedback, architectural feedback, onboarding, being able to collaborate. onboarding, being able to collaborate. onboarding, being able to collaborate. This is Again, if you're doing This is Again, if you're doing This is Again, if you're doing uh white coding, you're working as a uh white coding, you're working as a uh white coding, you're working as a solo project, this is not a talk for solo project, this is not a talk for solo project, this is not a talk for you. If you are working in teams, which you. If you are working in teams, which you. If you are working in teams, which I believe most of you folks are, I believe most of you folks are, I believe most of you folks are, collaborating in teams, you're not collaborating in teams, you're not collaborating in teams, you're not likely using completely dark factories, likely using completely dark factories, likely using completely dark factories, orchestrators, where nobody looks at the orchestrators, where nobody looks at the orchestrators, where nobody looks at the code. You're actually collaborating in code. You're actually collaborating in code. You're actually collaborating in teams, you need to do knowledge sharing. teams, you need to do knowledge sharing. teams, you need to do knowledge sharing. Which is the alignment part. And that is Which is the alignment part. And that is Which is the alignment part. And that is the part which is the most important the part which is the most important the part which is the most important aspect of release. aspect of release. aspect of release. So, for semantic accuracy, we can build So, for semantic accuracy, we can build So, for semantic accuracy, we can build better tooling, but alignment must better tooling, but alignment must better tooling, but alignment must survive.

survive. survive. So, let's just kind of like dive into So, let's just kind of like dive into So, let's just kind of like dive into alignment. What does it mean? Like, in alignment. What does it mean? Like, in alignment. What does it mean? Like, in today's world, can we actually think of today's world, can we actually think of today's world, can we actually think of better model than aligning just based on better model than aligning just based on better model than aligning just based on reading line by line code? reading line by line code? reading line by line code? So, most folks have probably heard about So, most folks have probably heard about So, most folks have probably heard about spectrum development by now. So, spectrum development by now. So, spectrum development by now. So, spectrum development is like, okay, we spectrum development is like, okay, we spectrum development is like, okay, we write a spec, you know, write a spec, you know, write a spec, you know, it covers all the details, we pass it to it covers all the details, we pass it to it covers all the details, we pass it to an agent, it generates a code, and then an agent, it generates a code, and then an agent, it generates a code, and then we verify. we verify. we verify. So, what's wrong here? If you look back So, what's wrong here? If you look back So, what's wrong here? If you look back in 1970, this is what waterfall model in 1970, this is what waterfall model in 1970, this is what waterfall model was. You know, you have requirements, was. You know, you have requirements, was. You know, you have requirements, you have specification, you implement, you have specification, you implement, you have specification, you implement, and then you verify. But, there's no and then you verify. But, there's no and then you verify. But, there's no feedback loop. feedback loop. feedback loop. You know, the spec is written before we You know, the spec is written before we You know, the spec is written before we identified everything else. Right? identified everything else. Right? identified everything else. Right? That's why today everyone still wants to That's why today everyone still wants to That's why today everyone still wants to use your coding sessions, whether it's use your coding sessions, whether it's use your coding sessions, whether it's cloud code, codex, cursor, whatever cloud code, codex, cursor, whatever cloud code, codex, cursor, whatever you're using, you want to interact with you're using, you want to interact with you're using, you want to interact with the agents. And the reason you're the agents. And the reason you're the agents. And the reason you're interacting with agents is because there interacting with agents is because there interacting with agents is because there were certain things which were not clear were certain things which were not clear were certain things which were not clear in the spec, and we still need to in the spec, and we still need to in the spec, and we still need to capture that. capture that. capture that. And second is, as you implement, you And second is, as you implement, you And second is, as you implement, you identify more issues, and you never go identify more issues, and you never go identify more issues, and you never go back and update the spec, because like, back and update the spec, because like, back and update the spec, because like, if you're doing the spectrum if you're doing the spectrum if you're doing the spectrum development, it's already done. Once the development, it's already done. Once the development, it's already done. Once the spec is done, you expect, like, you spec is done, you expect, like, you spec is done, you expect, like, you know, the code will come know, the code will come know, the code will come deterministically. But guess what? LLM deterministically. But guess what? LLM deterministically. But guess what? LLM is not deterministic. It's going to make is not deterministic. It's going to make is not deterministic. It's going to make decisions itself. So, that's why decisions itself. So, that's why decisions itself. So, that's why spectrum development is a great spectrum development is a great spectrum development is a great uh methodology, but it falls short in uh methodology, but it falls short in uh methodology, but it falls short in day-to-day software development. But day-to-day software development. But day-to-day software development. But there are some interesting aspects of there are some interesting aspects of there are some interesting aspects of this, which we should carry forward. The this, which we should carry forward. The this, which we should carry forward. The most important part is the intent.

most important part is the intent. most important part is the intent. An intent doesn't only live in the spec. An intent doesn't only live in the spec. An intent doesn't only live in the spec. Intent live in your Jira ticket. That's Intent live in your Jira ticket. That's Intent live in your Jira ticket. That's the goal, right? Like it's kind of like the goal, right? Like it's kind of like the goal, right? Like it's kind of like where you express what we want to do. where you express what we want to do. where you express what we want to do. It lives in your PRDs. It's kind of like It lives in your PRDs. It's kind of like It lives in your PRDs. It's kind of like details, like it's a plan. details, like it's a plan. details, like it's a plan. But most importantly, it lives in your But most importantly, it lives in your But most importantly, it lives in your prompts today. prompts today. prompts today. This is where the real decisions are This is where the real decisions are This is where the real decisions are being made. You start with like, okay, being made. You start with like, okay, being made. You start with like, okay, this is a Jira ticket I'm going to look this is a Jira ticket I'm going to look this is a Jira ticket I'm going to look at, at, at, but you're going back and forth with the but you're going back and forth with the but you're going back and forth with the agent, and this is where all the user agent, and this is where all the user agent, and this is where all the user decisions are being made. decisions are being made. decisions are being made. But what we do today is we create a But what we do today is we create a But what we do today is we create a change, we create a pull request, and change, we create a pull request, and change, we create a pull request, and then we throw away the prompts. And this then we throw away the prompts. And this then we throw away the prompts. And this is one of the things that we need to is one of the things that we need to is one of the things that we need to change. change. change. Let us first talk also about semantic Let us first talk also about semantic Let us first talk also about semantic accuracy because like, you know, you're accuracy because like, you know, you're accuracy because like, you know, you're saying, "Hey, Ankit, okay, I understand saying, "Hey, Ankit, okay, I understand saying, "Hey, Ankit, okay, I understand the alignment part, but like there are the alignment part, but like there are the alignment part, but like there are still bugs in the code. Who's going to still bugs in the code. Who's going to still bugs in the code. Who's going to look at that?" LLMs are also not great look at that?" LLMs are also not great look at that?" LLMs are also not great at this, and we already talked about how at this, and we already talked about how at this, and we already talked about how AI agents may reviewers may not always AI agents may reviewers may not always AI agents may reviewers may not always be perfect. be perfect. be perfect. So, this is where I introduce you to the So, this is where I introduce you to the So, this is where I introduce you to the concept of AI slop registry.

concept of AI slop registry. concept of AI slop registry. So, think about this. We are reviewing So, think about this. We are reviewing So, think about this. We are reviewing If you're reviewing code today manually, If you're reviewing code today manually, If you're reviewing code today manually, and I expect everyone should be doing and I expect everyone should be doing and I expect everyone should be doing some degree of this. some degree of this. some degree of this. We are essentially possibly identifying We are essentially possibly identifying We are essentially possibly identifying the same issues over and over again. the same issues over and over again. the same issues over and over again. Can we actually capture these concepts Can we actually capture these concepts Can we actually capture these concepts and codify them so that we don't have to and codify them so that we don't have to and codify them so that we don't have to always create those review feedback always create those review feedback always create those review feedback one by one. You actually also have all one by one. You actually also have all one by one. You actually also have all of those things automatically of those things automatically of those things automatically identified. The beauty of this is if you identified. The beauty of this is if you identified. The beauty of this is if you do it a few times, you now build sort of do it a few times, you now build sort of do it a few times, you now build sort of like a system which actually learns over like a system which actually learns over like a system which actually learns over time. So, think of this as sort of like time. So, think of this as sort of like time. So, think of this as sort of like you know, you know, you're doing more you know, you know, you're doing more you know, you know, you're doing more training on top of the standard LLM that training on top of the standard LLM that training on top of the standard LLM that you have actually extracted like built you have actually extracted like built you have actually extracted like built on top of. So, AI slop registers now can on top of. So, AI slop registers now can on top of. So, AI slop registers now can create better create better create better results results results because it's trained it's learning from because it's trained it's learning from because it's trained it's learning from the review experience that you as humans the review experience that you as humans the review experience that you as humans are providing. are providing. are providing. Every recurring comment is now a Every recurring comment is now a Every recurring comment is now a guardrail that you don't have to review guardrail that you don't have to review guardrail that you don't have to review again.

again. again. Okay, so let's kind of like try to put Okay, so let's kind of like try to put Okay, so let's kind of like try to put both of these alignment and the semantic both of these alignment and the semantic both of these alignment and the semantic accuracy together. accuracy together. accuracy together. It is two halves of the same problem. We It is two halves of the same problem. We It is two halves of the same problem. We are trying to understand what are the are trying to understand what are the are trying to understand what are the core mechanics of review. How do we core mechanics of review. How do we core mechanics of review. How do we actually break it down into two parts actually break it down into two parts actually break it down into two parts which is the alignment and the semantic which is the alignment and the semantic which is the alignment and the semantic accuracy and bring them together into a accuracy and bring them together into a accuracy and bring them together into a single loop. single loop. single loop. So, first thing is you could take your So, first thing is you could take your So, first thing is you could take your session and you capture the session and you capture the session and you capture the user responses and that essentially user responses and that essentially user responses and that essentially forms your acceptance criteria. forms your acceptance criteria. forms your acceptance criteria. The acceptance criteria then tied with The acceptance criteria then tied with The acceptance criteria then tied with your AI slot register that you are now your AI slot register that you are now your AI slot register that you are now constantly maintaining finally creates a constantly maintaining finally creates a constantly maintaining finally creates a test plan. And this is the test plan test plan. And this is the test plan test plan. And this is the test plan which then gets verified. This is part which then gets verified. This is part which then gets verified. This is part of the system that we are building is of the system that we are building is of the system that we are building is the verification system where it spins the verification system where it spins the verification system where it spins up a preview, takes your test plan and up a preview, takes your test plan and up a preview, takes your test plan and make sure it actually works end-to-end. make sure it actually works end-to-end. make sure it actually works end-to-end. Even if the code looks right, does it Even if the code looks right, does it Even if the code looks right, does it actually work? actually work? actually work? So, this verification part has become So, this verification part has become So, this verification part has become interesting. And now the kicker is interesting. And now the kicker is interesting. And now the kicker is this is now your review surface.

this is now your review surface. this is now your review surface. You're not reviewing code line by line, You're not reviewing code line by line, You're not reviewing code line by line, but rather you're looking at the but rather you're looking at the but rather you're looking at the evidence of what was the intent, did the evidence of what was the intent, did the evidence of what was the intent, did the user actually implement the capability user actually implement the capability user actually implement the capability that was defined in the intent, and did that was defined in the intent, and did that was defined in the intent, and did the behavior actually meet the the behavior actually meet the the behavior actually meet the requirements that we had in the as requirements that we had in the as requirements that we had in the as acceptance criteria. acceptance criteria. acceptance criteria. So, you're still having the So, you're still having the So, you're still having the architectural decisions, you're still architectural decisions, you're still architectural decisions, you're still having these arguments, but the review having these arguments, but the review having these arguments, but the review surface changes. So, So, just giving a walk through of like how just giving a walk through of like how just giving a walk through of like how we have built our system. The session we have built our system. The session we have built our system. The session becomes a criteria, so all these becomes a criteria, so all these becomes a criteria, so all these decisions that you are doing here with decisions that you are doing here with decisions that you are doing here with the agent, you're asking the agent, you're asking the agent, you're asking uh you know, you're providing this uh you know, you're providing this uh you know, you're providing this feedback. Like even if there is a simple feedback. Like even if there is a simple feedback. Like even if there is a simple task, many times you're going back and task, many times you're going back and task, many times you're going back and forth. Agent will stop to ask questions. forth. Agent will stop to ask questions. forth. Agent will stop to ask questions. These are the decisions that we need to These are the decisions that we need to These are the decisions that we need to capture. capture. capture. This is the intent. This is what makes This is the intent. This is what makes This is the intent. This is what makes your review, makes your collaboration your review, makes your collaboration your review, makes your collaboration more valuable. This is how you teach more valuable. This is how you teach more valuable. This is how you teach your junior engineers on how to improve your junior engineers on how to improve your junior engineers on how to improve over time. These are the decisions which over time. These are the decisions which over time. These are the decisions which make a software engineer valuable today.

make a software engineer valuable today. make a software engineer valuable today. We convert those into an acceptance We convert those into an acceptance We convert those into an acceptance criteria. This is where you can also criteria. This is where you can also criteria. This is where you can also leverage LLM to do so. So, you capture leverage LLM to do so. So, you capture leverage LLM to do so. So, you capture these user uh these user uh these user uh decisions and make sure you can actually decisions and make sure you can actually decisions and make sure you can actually create a test plan based on this. I know create a test plan based on this. I know create a test plan based on this. I know test plan creation is always painful, test plan creation is always painful, test plan creation is always painful, which is where I would always recommend which is where I would always recommend which is where I would always recommend people to use LLM for this purpose. people to use LLM for this purpose. people to use LLM for this purpose. And finally, the criteria plus invariant And finally, the criteria plus invariant And finally, the criteria plus invariant is what makes the test plan. is what makes the test plan. is what makes the test plan. Uh and we build the verification systems Uh and we build the verification systems Uh and we build the verification systems to actually capture the test plan, run to actually capture the test plan, run to actually capture the test plan, run your previews, and be able to test based your previews, and be able to test based your previews, and be able to test based on this test plan. So, even imagine if on this test plan. So, even imagine if on this test plan. So, even imagine if you're building a new feature, you don't you're building a new feature, you don't you're building a new feature, you don't have to maintain tests at all. This is have to maintain tests at all. This is have to maintain tests at all. This is creating tests in real time, and this is creating tests in real time, and this is creating tests in real time, and this is where you can leverage the power of LLM where you can leverage the power of LLM where you can leverage the power of LLM because the test plan maintenance and because the test plan maintenance and because the test plan maintenance and creation can be really painful, but creation can be really painful, but creation can be really painful, but the value of human in the loop here is the value of human in the loop here is the value of human in the loop here is the governance and the review part, and the governance and the review part, and the governance and the review part, and the part where you're reviewing the test the part where you're reviewing the test the part where you're reviewing the test plan and not the code.

plan and not the code. plan and not the code. Right? I know it's been like over 20 Right? I know it's been like over 20 Right? I know it's been like over 20 years we came up with test-driven years we came up with test-driven years we came up with test-driven development. This in some ways is closer development. This in some ways is closer development. This in some ways is closer to behavior-driven to behavior-driven to behavior-driven development, where the test plan is now development, where the test plan is now development, where the test plan is now something which even you can share with something which even you can share with something which even you can share with your product managers, your designers. your product managers, your designers. your product managers, your designers. Everyone can participate because these Everyone can participate because these Everyone can participate because these things are now in English. things are now in English. things are now in English. At the same time, we have At the same time, we have At the same time, we have now deterministic verification, which now deterministic verification, which now deterministic verification, which actually verifies actually verifies actually verifies whether the particular test criteria has whether the particular test criteria has whether the particular test criteria has been met or not. Let's move on. Let's move on. So, this is where I would say the system So, this is where I would say the system So, this is where I would say the system is not supposed to be perfect. It's is not supposed to be perfect. It's is not supposed to be perfect. It's deterministic where it can be, but LLM deterministic where it can be, but LLM deterministic where it can be, but LLM where you must. Not everything can be where you must. Not everything can be where you must. Not everything can be deterministic. Not every system can be deterministic. Not every system can be deterministic. Not every system can be built in a way which is built in a way which is built in a way which is 100% built on deterministic systems. 100% built on deterministic systems. 100% built on deterministic systems. This is where you use LLM as a fallback.

This is where you use LLM as a fallback. This is where you use LLM as a fallback. Let me give an example. Like if you're Let me give an example. Like if you're Let me give an example. Like if you're making a change in your web application, making a change in your web application, making a change in your web application, what you can do is it creates a test what you can do is it creates a test what you can do is it creates a test plan of what the behavior changes. Let's plan of what the behavior changes. Let's plan of what the behavior changes. Let's say you introduce a new payment form. say you introduce a new payment form. say you introduce a new payment form. So, it creates a new payment form. The So, it creates a new payment form. The So, it creates a new payment form. The verification here is does the payment verification here is does the payment verification here is does the payment system changes? system changes? system changes? Uh Uh Uh an AI agent can go and browse through an AI agent can go and browse through an AI agent can go and browse through your application to fill out a form and your application to fill out a form and your application to fill out a form and capture screenshots as evidence. capture screenshots as evidence. capture screenshots as evidence. And then take those screenshots as well And then take those screenshots as well And then take those screenshots as well as your database snapshots to identify as your database snapshots to identify as your database snapshots to identify whether the criteria was met. whether the criteria was met. whether the criteria was met. So, the screenshot testing or like sand So, the screenshot testing or like sand So, the screenshot testing or like sand flying can then still be done by agents. flying can then still be done by agents. flying can then still be done by agents. But at the same time, you're creating But at the same time, you're creating But at the same time, you're creating more solid evidence, which now a more solid evidence, which now a more solid evidence, which now a reviewer can look at and build more reviewer can look at and build more reviewer can look at and build more confidence that this actually works. So, now reviewers are reviewing the So, now reviewers are reviewing the intent, not the diff. intent, not the diff. intent, not the diff. You're reviewing the intent decisions. You're reviewing the intent decisions. You're reviewing the intent decisions. Uh what we said to build out.

Uh what we said to build out. Uh what we said to build out. What we tried and rejected, and capture What we tried and rejected, and capture What we tried and rejected, and capture all of these things from the sessions. all of these things from the sessions. all of these things from the sessions. Remember, capturing it from the sessions Remember, capturing it from the sessions Remember, capturing it from the sessions is is key. If we try to build it from is is key. If we try to build it from is is key. If we try to build it from the code, you'll end up in the same the code, you'll end up in the same the code, you'll end up in the same situation that we were talking about I situation that we were talking about I situation that we were talking about I think Dex was talking about yesterday, think Dex was talking about yesterday, think Dex was talking about yesterday, which is if your code is built by the which is if your code is built by the which is if your code is built by the same agent which is actually building a same agent which is actually building a same agent which is actually building a test plan, it's not going to test plan, it's not going to test plan, it's not going to build a test plan which will actually build a test plan which will actually build a test plan which will actually catch issues. So, that's why it's catch issues. So, that's why it's catch issues. So, that's why it's important to actually use the session important to actually use the session important to actually use the session information to build out a test plan. information to build out a test plan. information to build out a test plan. You can discuss architectural decisions. You can discuss architectural decisions. You can discuss architectural decisions. So, how you're creating the data models, So, how you're creating the data models, So, how you're creating the data models, how these services interact with each how these services interact with each how these services interact with each other. So, you've moved one level above. other. So, you've moved one level above. other. So, you've moved one level above. So, you're essentially instead of like So, you're essentially instead of like So, you're essentially instead of like reviewing line by line, you're actually reviewing line by line, you're actually reviewing line by line, you're actually having discussions on architecture, having discussions on architecture, having discussions on architecture, which are very critical for any kind of which are very critical for any kind of which are very critical for any kind of collaboration. collaboration. collaboration. And then you look at the evidence, And then you look at the evidence, And then you look at the evidence, everything that was collected from the everything that was collected from the everything that was collected from the verification. So, here's a homework for everyone. So, here's a homework for everyone. Go home and mine your last 1,000 review Go home and mine your last 1,000 review Go home and mine your last 1,000 review comments comments comments and build out a AI slot register for the and build out a AI slot register for the and build out a AI slot register for the things which are repeatable. So, a vast things which are repeatable. So, a vast things which are repeatable. So, a vast majority of the comments that you're majority of the comments that you're majority of the comments that you're providing in your code review are providing in your code review are providing in your code review are something that we repeat over and over something that we repeat over and over something that we repeat over and over again.

again. again. This compounds with every merge PR. This compounds with every merge PR. This compounds with every merge PR. Every time you capture something as a Every time you capture something as a Every time you capture something as a register, you don't have to capture that register, you don't have to capture that register, you don't have to capture that comment again. comment again. comment again. And this is where you can actually And this is where you can actually And this is where you can actually codify some of the best practices of codify some of the best practices of codify some of the best practices of doing code, maintain semantic accuracy, doing code, maintain semantic accuracy, doing code, maintain semantic accuracy, at the same time, do not lose the at the same time, do not lose the at the same time, do not lose the collaboration part of the review. It does follow a J curve. So, pain is It does follow a J curve. So, pain is real. real. real. Uh you will have to spend some time to Uh you will have to spend some time to Uh you will have to spend some time to actually make it pay off because actually make it pay off because actually make it pay off because initially creating a registry can take initially creating a registry can take initially creating a registry can take some time. And this is where I would some time. And this is where I would some time. And this is where I would recommend you folks can come and try out recommend you folks can come and try out recommend you folks can come and try out our product. So, code review is not just our product. So, code review is not just our product. So, code review is not just about code review. It is about uh about code review. It is about uh about code review. It is about uh really getting the alignment. And where really getting the alignment. And where really getting the alignment. And where we can build better tools is we can build better tools is we can build better tools is creating the semantic accuracy and creating the semantic accuracy and creating the semantic accuracy and defining your AI slot register. So, if defining your AI slot register. So, if defining your AI slot register. So, if you remember one thing from today, you remember one thing from today, you remember one thing from today, remember code review is not just about remember code review is not just about remember code review is not just about code review, it is about getting the code review, it is about getting the code review, it is about getting the alignment.

alignment. alignment. And yes, we are And yes, we are And yes, we are piloting our new product called Verify. piloting our new product called Verify. piloting our new product called Verify. Please join and be our early design Please join and be our early design Please join and be our early design partners. We are working with a few partners. We are working with a few partners. We are working with a few companies to companies to companies to uh uh uh pilot out a new verification system. pilot out a new verification system. pilot out a new verification system. This combines both the alignment side of This combines both the alignment side of This combines both the alignment side of things as well as building tools and things as well as building tools and things as well as building tools and capabilities for detecting semantic capabilities for detecting semantic capabilities for detecting semantic accuracy using the AI slot register. accuracy using the AI slot register. accuracy using the AI slot register. Thank you, everyone. Thanks for joining. Thank you, everyone. Thanks for joining. Thank you, everyone. Thanks for joining. >> [applause]