AI Writes More PRs. Who Validates Them? — Ali-Reza Adl-Tabatabai, Sonar

Let's begin. Good day Let's begin. Good day everyone. I am Ali. I am one of the everyone. I am Ali. I am one of the everyone. I am Ali. I am one of the founders of guitar.ai and founders of guitar.ai and was the CEO until last month, when we were was the CEO until last month, when we were acquired by Sonar acquired by Sonar . And now we are part of . And now we are part of . And now we are part of Sonar. In recent days, Sonar. In recent days, Sonar. In recent days, you've heard a lot about you've heard a lot about you've heard a lot about the importance of the importance of the importance of verification in the verification in the verification in the AI ​​era. So today I AI ​​era. So today I AI ​​era. So today I will focus on the code will focus on the code will focus on the code and its verification. and its verification. Almost all Almost all software teams software teams software teams have a have a have a centralized centralized centralized review phase, which review phase, which review phase, which is essentially CI plus is essentially CI plus is essentially CI plus code review. When you code review. When you code review. When you scale scale scale engineering, this stage engineering, this stage engineering, this stage plays a critical plays a critical plays a critical role, especially in role, especially in role, especially in the world of AI. First, the world of AI. First, the world of AI. First, it centrally it centrally it centrally ensures that ensures that ensures that all all all your your your quality standards are met. These are quality standards are met. These are quality standards are met. These are security, compliance, security, compliance, security, compliance, and code quality standards. This is the and code quality standards. This is the and code quality standards. This is the critical path critical path critical path to delivering your changes to to delivering your changes to to delivering your changes to production. Because production. Because production. Because faster is better, but faster is better, but faster is better, but when scaling when scaling when scaling engineering, it becomes a engineering, it becomes a engineering, it becomes a challenge due to challenge due to challenge due to long builds and long builds and long builds and tasks. You have tasks. You have tasks. You have asynchronous asynchronous asynchronous review processes across teams review processes across teams review processes across teams spread across different spread across different spread across different locations and locations and locations and geographies. And everything geographies. And everything geographies. And everything naturally naturally naturally slows down as slows down as slows down as you grow. This is you grow. This is you grow. This is also the stage where also the stage where also the stage where you spend significant amounts of you spend significant amounts of you spend significant amounts of money on money on money on infrastructure for infrastructure for infrastructure for developers. There are developers. There are developers. There are a lot of expensive a lot of expensive a lot of expensive tools here that tools here that tools here that need to be integrated need to be integrated need to be integrated and scaled and scaled and scaled with your with your with your organization. And all

organization. And all organization. And all this runs on very this runs on very this runs on very expensive expensive expensive infrastructure. For these infrastructure. For these infrastructure. For these reasons, this stage reasons, this stage reasons, this stage is often the main focus of is often the main focus of is often the main focus of your platform your platform your platform team. But team. But any failure at this any failure at this any failure at this stage is stage is stage is very costly. Failures very costly. Failures very costly. Failures create an external create an external create an external cycle, usually cycle, usually cycle, usually measured in hours measured in hours measured in hours and days. So, failures and days. So, failures are expensive. They are are expensive. They are are expensive. They are precious. They precious. They precious. They significantly slow down significantly slow down significantly slow down your your your work speed. You are wasting work speed. You are wasting work speed. You are wasting developers' time, developers' time, developers' time, compounded by compounded by compounded by interruptions to interruptions to interruptions to their streaming their streaming their streaming state. Optimizing state. Optimizing state. Optimizing this is also very this is also very this is also very expensive. You have expensive. You have expensive. You have many many many fragmented fragmented fragmented systems integrated systems integrated systems integrated together using together using together using individual individual individual configurations and configurations and configurations and scripts. And you scripts. And you scripts. And you usually have a usually have a usually have a resource-constrained platform resource-constrained platform resource-constrained platform team team team trying trying trying to keep it all up and to keep it all up and to keep it all up and running. running. Trying to solve this Trying to solve this with AI is also with AI is also with AI is also quite complex, as it quite complex, as it quite complex, as it requires complex requires complex requires complex workflow orchestration workflow orchestration workflow orchestration , , , asynchronous asynchronous asynchronous interaction, team interaction, team interaction, team processes, and processes, and processes, and integrations between integrations between integrations between tools.

tools. In fact, AI is even In fact, AI is even making this problem worse. making this problem worse. Thanks to AI, Thanks to AI, more PRs are being generated more PRs are being generated than before, and they are than before, and they are than before, and they are getting larger, getting larger, getting larger, each of which may each of which may each of which may contain defects. So contain defects. So contain defects. So your costs in the your costs in the your costs in the development cycle shift development cycle shift development cycle shift to the right. You need to to the right. You need to to the right. You need to review a lot review a lot review a lot more code. Now more code. Now more code. Now there is there is there is the potential for significantly the potential for significantly the potential for significantly more more more production failures. production failures. production failures. So, your choice here is So, your choice here is So, your choice here is quite complicated. Either quite complicated. Either quite complicated. Either you slow down and you slow down and you slow down and ask developers ask developers ask developers to review each PR to review each PR to review each PR very carefully, or you very carefully, or you very carefully, or you approve PRs approve PRs approve PRs automatically and automatically and automatically and risk risk risk incidents in incidents in incidents in production. Either production. Either way, you end way, you end way, you end up with less up with less up with less satisfied satisfied satisfied developers, lower developers, lower developers, lower morale, and morale, and morale, and much less much less much less productivity than productivity than productivity than you expected. So what's the you expected. So what's the you expected. So what's the answer here? Of course answer here? Of course , this is more AI. This is an , this is more AI. This is an , this is more AI. This is an agent-based approach to agent-based approach to agent-based approach to validation that validation that validation that we actually created in GitAr, and we actually created in GitAr, and we actually created in GitAr, and now we're also now we're also now we're also implementing it as implementing it as implementing it as part of Sonar. Essentially, part of Sonar. Essentially, part of Sonar. Essentially, we have an agent that we have an agent that we have an agent that fully automates fully automates fully automates all validation steps.

all validation steps. all validation steps. As a result, you As a result, you As a result, you get "green" PRs get "green" PRs get "green" PRs ready to merge, or ready to merge, or ready to merge, or they are merged they are merged they are merged automatically. So, how does automatically. So, how does automatically. So, how does it work? First of all it work? First of all , when you create a PR, , when you create a PR, , when you create a PR, it checks it and it checks it and it checks it and adds notes and adds notes and adds notes and comments in the code. comments in the code. comments in the code. Nothing strange. But Nothing strange. But Nothing strange. But our goal is to make our goal is to make our goal is to make this experience as this experience as noise-free as possible, noise-free as possible, both through UX and both through UX and both through UX and by publishing by publishing by publishing only those issues that only those issues that only those issues that are real and are real and are real and important. You can important. You can important. You can customize customize customize the inspection the inspection the inspection with your own with your own with your own rules, add your rules, add your rules, add your own checks, and own checks, and own checks, and run your own run your own run your own automated automated automated workflows. You workflows. You workflows. You can also can also can also configure the system to configure the system to block merges block merges if there are unresolved if there are unresolved if there are unresolved concerns after a concerns after a concerns after a code review. It code review. It code review. It also analyzes CI failures also analyzes CI failures also analyzes CI failures and publishes a summary of the and publishes a summary of the and publishes a summary of the root causes of these root causes of these root causes of these failures. In particular, it is failures. In particular, it is failures. In particular, it is very good at detecting very good at detecting very good at detecting unstable (flaky) unstable (flaky) unstable (flaky) tests. You tests. You tests. You can even set can even set can even set rules that rules that rules that automatically automatically automatically restart restart restart unstable tests unstable tests unstable tests if you wish. This is a very if you wish. This is a very if you wish. This is a very popular feature. You popular feature. You popular feature. You can also can also can also automatically automatically automatically fix any fix any code review issues or code review issues or CI failures—any CI failures—any CI failures—any errors that occur errors that occur errors that occur in CI can be in CI can be in CI can be fixed for you by the system, fixed for you by the system, fixed for you by the system, as well as any as well as any issues it finds. You can issues it finds. You can either ask it either ask it either ask it to fix a specific to fix a specific to fix a specific issue, or, more issue, or, more issue, or, more interestingly, interestingly, interestingly, set it to set it to set it to automatically loop automatically loop automatically loop until all issues in the PR until all issues in the PR until all issues in the PR are fixed are fixed are fixed and you have a " and you have a " green" PR ready green" PR ready green" PR ready for merging. Finally, for merging. Finally, for merging. Finally, you can you can you can automatically automatically automatically approve and approve and approve and merge PRs that are " merge PRs that are " green". You can

green". You can green". You can define rules and define rules and define rules and conditions that conditions that conditions that control when PRs are control when PRs are control when PRs are automatically automatically automatically approved by agents approved by agents approved by agents and merged and merged and merged independently. We are independently. We are independently. We are observing a very observing a very observing a very interesting trend interesting trend interesting trend among our among our among our users. There is a users. There is a users. There is a clear trajectory towards clear trajectory towards clear trajectory towards fully fully fully automated PR. As automated PR. As the system is used and the system is used and confidence in the accuracy of AI grows confidence in the accuracy of AI grows , we see , we see , we see users users users moving towards full moving towards full moving towards full automation. automation. automation. First, they First, they First, they receive the checks, receive the checks, receive the checks, are are are satisfied, and satisfied, and satisfied, and begin to trust begin to trust begin to trust the accuracy of the results the accuracy of the results . They see that these . They see that these . They see that these checks are not only checks are not only checks are not only accurate, but also accurate, but also accurate, but also raise really raise really raise really important questions. So important questions. So , then they give the , then they give the , then they give the agent authority, agent authority, agent authority, allowing it to allowing it to allowing it to block PR. And then block PR. And then block PR. And then they start they start they start using using auto-fix capabilities to auto-fix capabilities to automatically automatically automatically move PRs to move PRs to move PRs to “green” status and fix “green” status and fix “green” status and fix any issues any issues any issues found during found during found during code reviews or code reviews or code reviews or any CI errors. And the any CI errors. And the any CI errors. And the really interesting step really interesting step really interesting step that starts that starts that starts now is when now is when now is when users users users set up rules set up rules that determine when the that determine when the that determine when the agent automatically agent automatically agent automatically approves and merges PRs.

approves and merges PRs. So, over time, as So, over time, as they they they start to trust start to trust start to trust the system, they the system, they the system, they unlock new unlock new unlock new levels of automation, levels of automation, levels of automation, and with additional and with additional and with additional levels of automation levels of automation levels of automation and trust, they and trust, they and trust, they get get get more out of the system more out of the system more out of the system . Because . Because . Because the system processes the system processes the system processes each individual PR each individual PR each individual PR using AI, it is using AI, it is using AI, it is also capable of also capable of also capable of discovering new types of discovering new types of discovering new types of analytical data. analytical data. For example, one For example, one metric that is metric that is metric that is very useful for very useful for very useful for platform teams is platform teams is platform teams is the classification of the classification of the classification of major CI failures. Where are major CI failures. Where are major CI failures. Where are you seeing you seeing you seeing failures in your failures in your failures in your CI system? What errors do CI system? What errors do CI system? What errors do your your your developers encounter? Are these developers encounter? Are these instability issues that instability issues that need to be addressed need to be addressed need to be addressed by improving by improving by improving task reliability, are task reliability, are task reliability, are these these these infrastructure infrastructure infrastructure issues that issues that issues that need to be fixed, or are need to be fixed, or are need to be fixed, or are they experiencing they experiencing they experiencing other types of other types of other types of errors? Very, very errors? Very, very errors? Very, very useful for useful for useful for platform teams. platform teams. platform teams. We also have another We also have another We also have another example of what we example of what we example of what we have in terms of have in terms of AI-based analytics—that is PR classification . What exactly is . What exactly is . What exactly is going on in your going on in your going on in your engineering team in engineering team in engineering team in terms of PRs that terms of PRs that terms of PRs that are in the works?

are in the works? are in the works? This is extremely This is extremely This is extremely useful for management useful for management useful for management teams. For example, teams. For example, teams. For example, how much PR how much PR how much PR goes to goes to goes to feature development compared to feature development compared to feature development compared to bug fixes bug fixes , routine , routine , routine tasks, tasks, tasks, refactoring, etc. refactoring, etc. If you look under the If you look under the hood of the system, you hood of the system, you hood of the system, you can see a number of can see a number of can see a number of components that components that components that make up Guitar. There make up Guitar. There make up Guitar. There are three interesting elements here are three interesting elements here . The first is the . The first is the . The first is the control plane, which acts as an control plane, which acts as an control plane, which acts as an orchestration layer orchestration layer orchestration layer for handling for handling for handling PR workflows. And then there is the PR workflows. And then there is the agent runtime or "binding ". We created our own ". We created our own ". We created our own binding that handles binding that handles binding that handles multiple multiple multiple agent execution, agent execution, agent execution, context management, memory context management, memory , , , tool calls, all tool calls, all tool calls, all integrations, and more. integrations, and more. Our own binding Our own binding allowed us to allowed us to allowed us to specifically specifically specifically optimize it for a optimize it for a validation usage scenario. And this validation usage scenario. And this allowed us allowed us allowed us to optimize the to optimize the to optimize the token cost, token cost, token cost, accuracy, and coverage of the accuracy, and coverage of the accuracy, and coverage of the agent itself. This agent itself. This agent itself. This allows us to essentially allows us to essentially allows us to essentially guarantee guarantee guarantee results at a results at a results at a fixed price for PR fixed price for PR fixed price for PR while delivering while delivering while delivering very high quality very high quality very high quality results. We also results. We also results. We also have an have an have an LLM proxy that handles LLM proxy that handles LLM proxy that handles routing between routing between routing between different models, different models, different models, as well as things like as well as things like as well as things like fault tolerance.

fault tolerance. Now that we are Now that we are part of Sonar, we are part of Sonar, we are part of Sonar, we are integrating Guitar with integrating Guitar with integrating Guitar with SonarQube. There are many SonarQube. There are many SonarQube. There are many opportunities here to opportunities here to opportunities here to combine an agent-based combine an agent-based combine an agent-based approach to validation approach to validation approach to validation with programmatic with programmatic with programmatic analysis. So, analysis. So, analysis. So, combining agents with combining agents with combining agents with things like things like things like contamination analysis, contamination analysis, control flow analysis, control flow analysis, data flow analysis, and data flow analysis, and software composition analysis. So, software composition analysis. So, all the traditional all the traditional all the traditional approaches combined with approaches combined with approaches combined with agents, in our opinion, agents, in our opinion, agents, in our opinion, open up a lot of open up a lot of open up a lot of possibilities, and together possibilities, and together possibilities, and together you get you get you get much better much better much better accuracy and better accuracy and better accuracy and better coverage at a really coverage at a really coverage at a really great price. So the great price. So the great price. So the overall results, overall results, overall results, the value we the value we the value we see in all of this see in all of this , is much higher , is much higher , is much higher quality, much better quality, much better quality, much better security, especially with security, especially with AI-generated code, faster AI-generated code, faster delivery times due to delivery times due to delivery times due to automation from automation from automation from creating creating creating pull requests to pull requests to pull requests to approving and merging them, approving and merging them, approving and merging them, and much higher and much higher developer productivity and satisfaction. So, developer productivity and satisfaction. So, less time is less time is less time is spent spent spent reviewing code and reviewing code and reviewing code and worrying about what's worrying about what's worrying about what's happening in happening in happening in production, with production, with production, with more confidence, more confidence, more confidence, and a better and a better and a better mood because now mood because now mood because now you have a lot of you have a lot of you have a lot of automation and automation and automation and less routine less routine less routine work. Finally, work. Finally, work. Finally, the features we’ve the features we’ve the features we’ve built for built for built for platform teams give them platform teams give them platform teams give them much more much more much more leverage to leverage to leverage to fix issues, fix issues, fix issues, add add add customizations, and customizations, and customizations, and see insights in this see insights in this see insights in this critical part of the critical part of the software development lifecycle software development lifecycle .

. In summary, In summary, validation of validation of validation of fetch requests is very fetch requests is very fetch requests is very important, especially important, especially important, especially for AI, but AI for AI, but AI for AI, but AI turns validation turns validation turns validation into a point of friction. And Guitar, what into a point of friction. And Guitar, what into a point of friction. And Guitar, what we've built, we've built, we've built, allows you to allows you to allows you to automate that automate that automate that step, completely step, completely step, completely automate automate automate query validation, query validation, query validation, go from go from go from creation to merge, creation to merge, creation to merge, completely removing that completely removing that completely removing that friction, and helps friction, and helps friction, and helps you keep up with the you keep up with the you keep up with the speed of speed of speed of AI-powered code generation. AI-powered code generation. And finally, the combination of And finally, the combination of using agents using agents , i.e. agent , i.e. agent , i.e. agent verification, plus verification, plus verification, plus algorithmic algorithmic algorithmic analysis of programs, will analysis of programs, will analysis of programs, will give you something that, in give you something that, in give you something that, in my opinion, will be my opinion, will be my opinion, will be much better than much better than much better than either of them alone. either of them alone. Want to talk Want to talk more and see more and see more and see more? If you more? If you more? If you want to see want to see want to see a demonstration, a demonstration, a demonstration, come visit us at come visit us at come visit us at booth #P 7, right there in the booth #P 7, right there in the booth #P 7, right there in the hall. Thank you.