[music] >> All right. >> [applause] >> Thank you. That's very helpful. My name is Anirban Chatterjee. I do product marketing at Sonar. I'm really excited to be talking to this group today. It's actually my first time here at this conference. Um and so I've been having a blast uh along with my team here uh meeting a whole bunch of AI engineers as well as leaders and you know, influencers and founders. Uh there's a lot going on in this space. I think this year there's really been a turning point from experimentation to engineering. And that makes me that warms my heart very deeply because I started my career many many many many many years ago as a software engineer uh writing code for for servers, if you can believe it. Um and I think there's a turning point that's happening right now where we're starting to add the capabilities that we need to add to these systems in order to make them repeatable, make them scalable, make them consistent much in the way we were doing with cloud computing not too long ago in order to expand the access that IT technology gave to small businesses and other innovators. I think AI is going to do the same thing for software development going forward. But in order to do that, in order to get there, we need to start adding safety and trust to these systems so that they can be used more widely across a wide variety of use cases so we can build new things and solve bigger problems. And how we get there is what we're going to talk about today. And uh for those of you who were in uh Tarik's keynote yesterday, uh he presented some of this data and I'm going to talk about it a little bit deeper today. So there was a study that Carnegie Mellon did um where they actually looked at projects that were posted on GitHub. And they were able to use the metadata to support to sort them into projects where uh there were just traditional tools that were being used and projects where an AI tool was used to write the code. In this case, it was Cursor, although it could have been any AI tool. And what they found was interesting. They found that there was, in fact, a temporary spike in productivity, um but it lasted about 3 months and then it went back down. And the reason for that, we think, is because there was also a persistent increase in static analysis warnings and code complexity. They're actually using SonarQube uh to actually collect the data on this and they saw that there was a persistent increase in these types of issues that went beyond the 3-month mark and persisted well into the future. And so, it's these types of issues that end up actually slowing developers down even more and and making And this is what makes it a challenge to deliver high-quality code using AI tools. The reason for this is that there's a differing need for quality depending on the criticality of the application, right? If you're experimenting, if you're playing around, if you're just one person building things to see what's possible, um if it's an internal non-critical application which has a few users, maybe it's just you, maybe it's a small team, maybe it's just a short-lived project that's not going to last very long, the gap between the quality that you're getting from the AI tool and the quality you need from the application is quite small, right? And so, you can And you can live with uh that gap. But as you move to higher levels of criticality, as you run into situations where you're supporting many, many users, it's a larger code base with many lines of code and many changes happening across that code base all the time. Uh you have many, many users. Some of them could be adversaries that are actively trying to break your software, right? And so, in those cases, the quality level you need is quite a bit higher than the quality level you're getting by default from these AI tools. And that's where this verification debt comes in. That's where you have to bring the humans in, bring your software engineers in to try to close that gap and make sure that the quality level is is up to an acceptable level before you ship that code into production. So, why is this happening? Why is this gap actually occurring? We know these models are excellent. They're getting better and better all the time. I'm really excited to start playing with Fable now that that's out to see what levels of of code we can get out of Fable going forward. But, we do know that because of the technology, because of the way that these models are built, they will still make mistakes. They will still have quality issues. They are still somewhat error-prone. And if you let these errors go into production code, you could have a catastrophic effect to your organization. They're also missing context, right? They only know what you tell it. They don't know the broader things that are happening elsewhere in the code base. They don't know what's happening with your business. They don't know what happened in the meeting you had with somebody else 2 weeks ago that's going to influence the code you're writing today. They don't have all of the context that you have as an engineer, right? And so, they don't always know your objectives the way you do. And that is going to also cause gaps between what you need from the software and the way it's built. We also know that models are diverse. No two models are the same and they have diverse quality issues. And we actually want to explore this. And so, we actually have a leaderboard that you can go to on our website right now. It's called the LLM leaderboard. And what we do is we take all of the major new models that come out and we evaluate them. We give them, you know, 4,000 or so coding tasks and we evaluate them using all of the metrics that SonarCube uses to evaluate code, right? We look at their correctness, complexity, the way the rate at which they're solving the tasks we assign them, and then our classic things maintainability, reliability, and security. And we're able to kind of graph all of these models across these different axes and show you where models perform well and where they have room to improve. And what you're looking at on the screen right now is actually Claude Opus 4.6 and Claude Sonnet 4.6. If you're a Claude customer, you might be toggling between these two models to control your token burn rates. And you'll see that Claude Sonnet is actually quite good from a correctness standpoint, from from solving tasks, and from a standpoint. But if you're requiring higher levels of maintainability or higher levels of security, if you're trying to get a lower complexity out of your code, you might benefit from switching to Opus for tasks like that. And so we run these kinds of analyses across a lot of different models, and you're always able to go to our website to get the latest analyses that we run. I think we're actually doing the latest Claude and and OpenAI models pretty soon. But this kind of data is helpful, right? Because it tells you what where models are good and where models are not good. It also serves to put some sunlight on the fact that you still need to be vigilant with these models, right? None of these models are ever going to be perfect. You're always going to have some kind of need for verification in the loop to make sure that the code that you're getting is the code you actually want to ship. Now, classically, that verification can be human verification. It can be you. Your own eyes reading the code, your own intellect reviewing the code to make sure that it's successful. But we know, based on experience and now based on research, that human review can also be compromised. This is a study that was done earlier this year by Wharton, and they actually gave quite a lot of human participants tasks to complete, and they gave those human participants the use of an AI tool to complete those tasks. But unbeknownst to those participants, the AI was told to confidently lie to these participants some of the time. And what they found in the data is that while participants did follow the AI advice 92.7% of the time when the AI was correct, they unfortunately also listened to the AI nearly 80% of the time when the AI was wrong. This is almost surely happening in code review as well, right? Especially when there's higher amounts of code being written, when there's multiple agents writing code simultaneously, when you now have to bring all those pieces together into a single software application, the load is just too great. There's there's only so many hours in the day, right? And you still have to ship something. And so there's a lot of rubber stamping that I'm sure is happening in all of your organizations. It's happening everywhere. And so we need to backstop that somehow with an automated verification tool. What can we do about it, right? As Tarek was talking about yesterday, you know, I think all of us sort of when we got involved with software, one of the things that we find most attractive about it is that code is is quite, you know, once you write code properly, it's going to run the same way every single time and there's a certain level of comfort in that, right? There's a certain level of comfort in knowing that if I write this function the right way, it is going to work this way every single time. And there's there's a clarity that comes to that and there's there's a certain confidence you get out of being able to build something that you know is going to work well for every user going forward, but we also know that code's written by humans, humans have requirements, and those requirements and externalities have impact in how this code functions. And as you add more and more and more code to the application, they can interact in unpredictable ways. As you now allow users to use those applications, those users can do all kinds of things you didn't expect, and so software is not provable in the same way that code is provable. Software can break in interesting and novel ways, and as you're using AI to write more and more software to solve bigger and bigger problems, um you're going to run into these limitations more and more often. And so having automatic verification as part of this process is an important part of the solution. It's going to help you control some of the risks that you're introducing by maybe releasing some of the control you have over the code that's actually being written. And so we believe that verification is going to be a key enabler and a and a key unblocker for all of the amazing things that we're going to be able to achieve with with the AI-driven software development going forward. And so when we say verification, what do we mean, right? We think there's two core elements to successful automated verification when it comes to AI coding. One is that needs to be zero trust. What do we mean by that? Zero trust in this context basically means that the code could really have come from anywhere. It could still be written by a human, it could be written by an AI as I just showed you a few slides ago. Different AIs will write code in different ways, and you're not going to want to be able You're not going to want to use that same AI to to validate the code because you're going to want a diversity of of tools being used to make sure that you're catching all the different issues that can happen. And so, no matter where the code is coming from, you want to have a a similar comprehensive regime to verify that code that works the same no matter how that code was written, right? Use a different methodology to review the code that was used to write the code. Uh it's completely auditable, completely explainable, so you can prove that verification was run the same way every single time, and it's algorithmic and repeatable and consistent no matter how you run it. It also needs to be multi-layered. You need to have multiple ways uh a multiple techniques being used, multiple approaches being used to review the code that is being generated, right? Um because you're never going to be able to find every single problem that can occur in software by just using one or two methods. You need to use computational review, you also need to use LLM driven review, and everything else in between, right? You heard a little bit about agentic I'm I'm really I've been hearing a lot about agentic loops this week, uh and Tarik uh talked yesterday about our framework for agentic loops. We call it ACDC, or agentic agent-centric development cycle. And there's three phases in in the ACDC that we that we like to talk about. The the easiest one By easiest, I mean the the fastest one to implement now, the one that many of you are probably already on a path to implementing, is the verification step, which is front which is top center, right? This is the most important piece that allows you to to write code in these agentic loops in a way that is going to be easily shippable. Uh it needs to be multi-layered, it needs to be reasoning based, and it needs to cut across quality issues, security issues, and compliance issues to make sure that you're shipping quality that you can stand behind, right? Um before the verification step, there's a guidance step. And what guide allows you to do is provide guardrails and context and constraints to make sure that the agent has everything it needs up front to write better code uh uh uh the first time, right? And then finally, after verification, you need to solve the issues that come up, right? And that's where the solve state come. That's where you can remediate any issues that are found in the code. You Hopefully, you're allowing the agent to have the agency to do so itself by providing access to the tools it needs to to find the issues and fix them itself and then just repeat the loop. And so these these agentic loops with verification at the core is how you can get to shipping quality software using AI agents. And there are prior there's different reasons why you would want to do this. And when we talk to customers, and we've talked to a lot of customers about this, uh the driving functions that are forcing them to adopt verification across all of their AI coding processes are are very similar, right? They want to make sure that AI code is verified consistently. They don't want to have different methods of verification applying to different projects or different teams. They want to have a standard rulebook that applies everywhere, no matter what tool is being used. They all also want to make sure they're using their AI tools effectively, right? Some of that a big part of this is token efficiency or just efficiency in general, but also just making sure that the tools are being used for the things they're being designed to do in ways that we know they're good at doing, right? So we talk a lot about token efficiency, and we talk a lot about making sure that the right models are used for the right projects, and so on. Finally, third, catching issues from a security standpoint as early as possible in development cycle. Shifting left on security issues has been very important for a number of years now, and now that AI is writing more and more code, catching security issues up front is extremely critical, especially now that we're in a world where CVEs are announced and then immediately exploited uh by bad actors almost often the same day. And so you need to make sure that your code is as hardened as possible from those types of issues creeping into production. And finally, maintaining compliance. Many of you, I'm sure, work in a regulated industry, uh and for those types of situations where you need to be able to prove that verification is run constantly and consistently across the board. Maintaining an audit trail that allows you to prove that is extremely important. We have a number of solutions that that help with that. SonarQube has been around for quite a while. There are probably quite a few of you that are already SonarQube users. It is a zero trust multi-layered verification platform that works across syntax issues, data flow issues, architectural issues, control flow issues. And it works across basically any language you would be using. We have a lot of deep hooks that I'm going to take you through in a moment that allow agents to have first party access to the SonarQube verification so they can more effectively write high quality code. And we also just recently, and by recently I mean just a few weeks ago, acquired a company called Gitarr, based right here in San Mateo. And they do AI code review. And more than that, they actually build a full they build and automate the full CI workflow so that you can not only find issues using an using our LM approach, but you can also automatically block if those issues cause a quality issue that you would want to push forward. It can write fixes and it can approve those fixes and merge those PRs completely automatically if you wanted to. Now, we'd want to earn that trust. It doesn't happen that way by default usually. By default they'll just find the issues and show them to you and enter a dialogue with you so you can have those issues fixed. But as you use it more and more and gain confidence, you can turn on more and more features and completely automate the PR review workflow if you like using Gitarr. So, the other big news from earlier this week, and Tarik alluded to this during his talk yesterday, is that we also announced a new agentic loop capability with a product called Sonar Vortex. And Sonar Vortex, I'm going to I'm going to show another flowchart that shows what it does, but basically it is providing your agents with tools in the inner loop, in the agentic loop, to not only get constraints and guardrails up front to write better code, but also run verification as it's writing code in real time so it can find and fix the issues that are being created. And then we also released the remediation agent. And the remediation agent allows you to tackle backlog issues and and take down your tech debt at a scale that you might not have the bandwidth to do now with human developers, right? You can basically take your your tech debt or your older issues, your legacy code, point them at remediation agent, and it can then improve your code base almost in the background while you focus on the on the innovation work at the front end that you're working on now. And those are both GA as of this week. Now, there was a very detailed chart that was shown during the keynote yesterday that I'm going to kind of break down for you and really explain what the different pieces of this chart mean. This is kind of how we see the ACDC applying not only to the inner agentic loops, but also the outer CICD loops that we're all working in to develop code. Before you start either of those loops though, it is really important to have a sense of the specifications of what you actually are going to want to accomplish with the software that's being written. This is This is your architectural constraints. This is the way your desired architecture for the software. This is your coding standards and your coding patterns that are acceptable. These are the list of dependencies that you are and are not allowed to use. These are your coding standards and and syntax standards that you obey in your organization. This could be your your logging practices or your observability and tracing practices. All of that goes into your specs and then you also need to define what your quality criteria are, right? And we have quality criteria that we ship with that you can use by default or you can adjust them as it makes sense for your organization, but this is like what are the levels of security, of quality, of maintainability that you're willing to accept in your code that you're pushing into production. You need to basically write that down and encode it. And now you're ready to start using LLM coding agents to write code. And whenever you're initiating a coding task with an with an with an agent, one of the first things that we can help is providing context and constraints so that the agent starts from the ground floor with an understanding of the code base and the guardrails that is relevant to it in that moment. You have to manage the context window of the agent you can't just throw your entire code base at the agent up front. It's going to spend a lot of time thrashing and exploring and burning tokens while it's doing it. We are able to efficiently provide just the context that it needs based on the work that it's being given so that it can get to work writing productive code very quickly. It then generates the source code as you can see and we also provide in loop verification to the agent as it's writing. It can call in to us and provide and get a list of issues that we are finding in real time in the code that's being written and the great thing about that is those issues can then be fixed immediately by the agent so they don't propagate into future agentic loops that are going to run in order to fully build out the software project that you're doing. This is being enabled by Sonar Vortex as of this week. Once all of the inner loops have run you get you reach a point later on when you have to start entering the formal review and shipping process for the code and this is the CICD process, right? And so there's a PR flow that gets initiated that I'm sure we're all familiar with. Guitar can live in that flow, SonarQube also lives in that flow in order to run a broad automated review of all of the code that is in the PR and an automated verification that actually returns issues across quality, security and maintainability. There's a superhuman review that is LLM driven by Guitar and there's a computational review that is run by SonarQube that actually assigns grades for all three of those things and won't allow the PR to go past into production unless it gets a passing grade across that criteria. So if there are issues that come up, you can actually use a fix agent to fix all the issues that are discovered there and then once you're actually able to pass that quality gate, that is when you're able to proceed in the test and build and deploy in the application. The verification needs to run in both the inner agentic loop and also in the outer loop for CICD. I'm running low on time so I'm going to hope this video completes. This is basically a video demo of the inner loop, the agentic loop. It's a it's a demo of Sonar Vortex, basically. And what you've just seen happen is we've given it a task and in starting that task, this is this is actually Cursor that's running right now. Cursor called into our Sonar Vortex context tool to get some context up front to to get give it an understanding of the code that's working and so it knows how to um how to write the code. It is now writing the code and once it's completed uh the initial write, it's going to call into our verification process to get a list of issues that it finds. And then if it and if issues get provided, it will actually fix those issues immediately in the inner loop. This is all happening automatically through an integration that we have directly with Cursor. We have similar integrations with uh with Claude Code, with Codex, with Any Gravity, with basically any major AI coding tool that you would have. And you can sort of get the gist. It It flagged an issue. It's got a plan. It's going to fix it immediately and it's going to run uh the analysis again and then it will not proceed until it actually is able to get a passing grade from us on the verification pass. I'm going to move past this cuz I'm out of time now. So, uh if you take one thing away from this presentation, it's that, you know, we believe very strongly, we're very convicted about this, that a governance and verification regime is a engine is extremely critical to unlock the next level of success that we need to be able to get from AI coding tools so we can solve bigger and bigger problems. And we know, based on our data, uh that Sonar customers and Sonar users are able to get higher levels of success from AI coding tools. I can I'm not going to go through all of these now cuz I'm out of time, but you can stop by the our big red booth downstairs and we'll be happy to talk to you about any of these. Um but, you know, the good news is I think many of you probably have access to this stuff already. Uh Sonar Cube is one of the most widely adopted verification tools in existence today. We have over 7 million developers around the world using us and we analyze close to 750 billion lines of code across our solutions every single day. And if there are a few of you in the room who care about Gartner at all, it's nice to know that we're a Gartner market uh magic quadrant leader as well. So, uh final slide, key takeaways, right? What are the things that we want to walk away from this? First, you know, establish some bounded autonomy guidelines for your AI agents. Give them the freedom to generate code, but also make sure that you're enforcing a centralized scheme of verification and constraints. Implement the ACDC, right? Give them context. Verify that they're doing the right work with independent metrics, and then use agents to solve their own mistakes. Empower them to do that. Make sure your developers are equipped with the orchestration tools that they need in order to design the context frameworks and the processes that are going to enable them to be successful and use AI most effectively. And finally, standardize on a single independent multi-layered verification platform that is used consistently across all projects, across all teams, across all developers, and all AI coding tools so that you can eliminate any blind spots that might be created by using silos of tools across the organization. Um that's all for me. Uh you can learn more by Oh, it's off the screen now. But uh you can come down to our big red booth downstairs, and we'd be happy to talk to you. Thank you very much. >> [music]