Through the AI Fog: The Architectural Decision Agentic Security Depends On — Manoj Nair, Snyk

>> What's up, everyone? Good to see you all >> What's up, everyone? Good to see you all here in the very first ever security here in the very first ever security here in the very first ever security track at the World's Fair. track at the World's Fair. track at the World's Fair. Um pretty exciting day, honestly, Um pretty exciting day, honestly, Um pretty exciting day, honestly, because uh like all of you, I genuinely because uh like all of you, I genuinely because uh like all of you, I genuinely love this stuff, and having looked love this stuff, and having looked love this stuff, and having looked through the agenda for the speakers we through the agenda for the speakers we through the agenda for the speakers we have today, have today, have today, um it's going to be absolutely um it's going to be absolutely um it's going to be absolutely mind-blowingly useful and fun mind-blowingly useful and fun mind-blowingly useful and fun information. So, hopefully information. So, hopefully information. So, hopefully if you stick around all day, by the end if you stick around all day, by the end if you stick around all day, by the end of the day, you'll be able to leave of the day, you'll be able to leave of the day, you'll be able to leave here, go back to your hotel rooms, and here, go back to your hotel rooms, and here, go back to your hotel rooms, and build some genuinely cool software, build some genuinely cool software, build some genuinely cool software, hopefully without humans in the loop, hopefully without humans in the loop, hopefully without humans in the loop, because that's the ultimate goal, right? because that's the ultimate goal, right? because that's the ultimate goal, right? Like, how do we build truly safe Like, how do we build truly safe Like, how do we build truly safe autonomous software at scale? And it's autonomous software at scale? And it's autonomous software at scale? And it's not something easy to do. not something easy to do. not something easy to do. So, with that being said, I'm very So, with that being said, I'm very So, with that being said, I'm very excited to welcome my my good friend and excited to welcome my my good friend and excited to welcome my my good friend and colleague, Manoj Nair. colleague, Manoj Nair. colleague, Manoj Nair. He is Sneak's Chief Innovation Officer He is Sneak's Chief Innovation Officer He is Sneak's Chief Innovation Officer and CTO. and CTO. and CTO. Um before Sneak, he was the Chief Cloud Um before Sneak, he was the Chief Cloud Um before Sneak, he was the Chief Cloud Officer at Convo IT. He founded and ran Officer at Convo IT. He founded and ran Officer at Convo IT. He founded and ran HyperGrid. He did product and security HyperGrid. He did product and security HyperGrid. He did product and security leadership at HPE, Dell, and RSA. And leadership at HPE, Dell, and RSA. And leadership at HPE, Dell, and RSA. And he's got something like a dozen patents he's got something like a dozen patents he's got something like a dozen patents to his name.

to his name. to his name. So, he's kind of a legend in the space. So, he's kind of a legend in the space. So, he's kind of a legend in the space. Uh he also personally had a hand in Uh he also personally had a hand in Uh he also personally had a hand in curating this entire track. So, if you curating this entire track. So, if you curating this entire track. So, if you like the talks today, please go up and like the talks today, please go up and like the talks today, please go up and say thank you to him after, but if not, say thank you to him after, but if not, say thank you to him after, but if not, then just don't blame me, basically. Um then just don't blame me, basically. Um then just don't blame me, basically. Um but yeah, welcome to the stage, Manoj. but yeah, welcome to the stage, Manoj. but yeah, welcome to the stage, Manoj. >> Woo! >> Woo! >> Woo! >> [applause] >> [applause] >> [applause] >> Thank >> Thank >> Thank Thank you, Randall. I was not expecting Thank you, Randall. I was not expecting Thank you, Randall. I was not expecting a bio. Hi, everyone. Um a bio. Hi, everyone. Um a bio. Hi, everyone. Um really appreciate uh you all joining really appreciate uh you all joining really appreciate uh you all joining here um right after those great keynotes here um right after those great keynotes here um right after those great keynotes up front. Uh I'm Manoj Nair, and uh I up front. Uh I'm Manoj Nair, and uh I up front. Uh I'm Manoj Nair, and uh I have the uh pleasure of leading an have the uh pleasure of leading an have the uh pleasure of leading an amazing team that is helping secure amazing team that is helping secure amazing team that is helping secure about, you know, 5,000 enterprise about, you know, 5,000 enterprise about, you know, 5,000 enterprise customers around the globe. Uh so, I get customers around the globe. Uh so, I get customers around the globe. Uh so, I get to look good about all of that, but some to look good about all of that, but some to look good about all of that, but some of what I'm going to show is real data of what I'm going to show is real data of what I'm going to show is real data from those customers. Half of Fortune 2 from those customers. Half of Fortune 2 from those customers. Half of Fortune 2 Fortune 500 runs on Sneak, and some of Fortune 500 runs on Sneak, and some of Fortune 500 runs on Sneak, and some of the data is, you know, from those the data is, you know, from those the data is, you know, from those learnings. learnings. learnings. But before that, I I also want to like But before that, I I also want to like But before that, I I also want to like talk about this, you know, the title of talk about this, you know, the title of talk about this, you know, the title of the talk was cutting through the AI fog, the talk was cutting through the AI fog, the talk was cutting through the AI fog, I think. But the way we do that is by I think. But the way we do that is by I think. But the way we do that is by having tracks like this.

having tracks like this. having tracks like this. So last year we stood here, there were So last year we stood here, there were So last year we stood here, there were 3,000 people at the AI Engineer was 3,000 people at the AI Engineer was 3,000 people at the AI Engineer was there, and you know, it really felt like there, and you know, it really felt like there, and you know, it really felt like security was missing in the room. And security was missing in the room. And security was missing in the room. And thanks to Swig and amazing partnership thanks to Swig and amazing partnership thanks to Swig and amazing partnership with the AI Engineer organization, we with the AI Engineer organization, we with the AI Engineer organization, we created the AI Security Summit in created the AI Security Summit in created the AI Security Summit in partnership with them. partnership with them. partnership with them. And we're creating this track, so it is And we're creating this track, so it is And we're creating this track, so it is really good to see this and all the really good to see this and all the really good to see this and all the great speakers who are going to, you great speakers who are going to, you great speakers who are going to, you know, talk um here today with some know, talk um here today with some know, talk um here today with some fantastic knowledge. But that is in our fantastic knowledge. But that is in our fantastic knowledge. But that is in our mind, like that's how we cut to the fog, mind, like that's how we cut to the fog, mind, like that's how we cut to the fog, right? Security needs to be very much right? Security needs to be very much right? Security needs to be very much part of the room. We're very passionate part of the room. We're very passionate part of the room. We're very passionate about it not slowing things down, but about it not slowing things down, but about it not slowing things down, but really that's how you build trusted really that's how you build trusted really that's how you build trusted systems. So, systems. So, systems. So, one thing you're going to hear from from one thing you're going to hear from from one thing you're going to hear from from me quite a bit of, you know, in this me quite a bit of, you know, in this me quite a bit of, you know, in this like, you know, what if you take one like, you know, what if you take one like, you know, what if you take one thing, it's this notion of thing, it's this notion of thing, it's this notion of our learning from this real-life data our learning from this real-life data our learning from this real-life data and working with the biggest frontier and working with the biggest frontier and working with the biggest frontier labs in the world and the biggest labs in the world and the biggest labs in the world and the biggest companies in the world companies in the world companies in the world is this concept that has really been, is this concept that has really been, is this concept that has really been, you know, not questioned in security you know, not questioned in security you know, not questioned in security before, but it is being asked now, before, but it is being asked now, before, but it is being asked now, right? Can, you know, the generator and right? Can, you know, the generator and right? Can, you know, the generator and the validator be the same?

the validator be the same? the validator be the same? And our point is, you know, in some of And our point is, you know, in some of And our point is, you know, in some of the data you'll show for all kinds of the data you'll show for all kinds of the data you'll show for all kinds of reasons why not, right? And and almost reasons why not, right? And and almost reasons why not, right? And and almost like if you know Sneak, don't think like if you know Sneak, don't think like if you know Sneak, don't think about the supply chain security company about the supply chain security company about the supply chain security company that shifted left. Like a lot of what that shifted left. Like a lot of what that shifted left. Like a lot of what you'll see is the last 18 months of what you'll see is the last 18 months of what you'll see is the last 18 months of what we have been doing with some of these we have been doing with some of these we have been doing with some of these very large enterprises in in adopting, very large enterprises in in adopting, very large enterprises in in adopting, you know, these complex uh systems that you know, these complex uh systems that you know, these complex uh systems that are we're all in joy and we love, and are we're all in joy and we love, and are we're all in joy and we love, and what What are we learning from that? what What are we learning from that? what What are we learning from that? Um, there are three problems that, you Um, there are three problems that, you Um, there are three problems that, you know, we hear when we talk to these know, we hear when we talk to these know, we hear when we talk to these customers. And I want to share those and customers. And I want to share those and customers. And I want to share those and I want to like kind of show some of the I want to like kind of show some of the I want to like kind of show some of the data behind that. Um, but, you know, data behind that. Um, but, you know, data behind that. Um, but, you know, fundamentally, it's it's really looking fundamentally, it's it's really looking fundamentally, it's it's really looking at this as um at this as um at this as um you think about this room and everyone you think about this room and everyone you think about this room and everyone that you support, you know, you're that you support, you know, you're that you support, you know, you're building fast at the frontier. building fast at the frontier. building fast at the frontier. The question nobody is answering is, can The question nobody is answering is, can The question nobody is answering is, can you trust what your agents just shipped you trust what your agents just shipped you trust what your agents just shipped and how they did it, right? And so if and how they did it, right? And so if and how they did it, right? And so if you'd like really take, you know, the you'd like really take, you know, the you'd like really take, you know, the gist of and I I have the joy of talking gist of and I I have the joy of talking gist of and I I have the joy of talking to all of these customers a lot, this is to all of these customers a lot, this is to all of these customers a lot, this is pretty much every like one, two, or pretty much every like one, two, or pretty much every like one, two, or three three three or all of them. It's really the con- or all of them. It's really the con- or all of them. It's really the con- conversation that happens. So, conversation that happens. So, conversation that happens. So, autonomous attacks, like, you know, it's autonomous attacks, like, you know, it's autonomous attacks, like, you know, it's not mythos, it's not, you know, I know I not mythos, it's not, you know, I know I not mythos, it's not, you know, I know I said the M word, but sorry. Uh, you said the M word, but sorry. Uh, you said the M word, but sorry. Uh, you know, it's it's not, you know, uh, GPT know, it's it's not, you know, uh, GPT know, it's it's not, you know, uh, GPT 55 Cybers. Like, we're seeing that you 55 Cybers. Like, we're seeing that you 55 Cybers. Like, we're seeing that you can have these attacks. All the frontier can have these attacks. All the frontier can have these attacks. All the frontier labs have been used in automated labs have been used in automated labs have been used in automated attacks. You can do that even without attacks. You can do that even without attacks. You can do that even without having frontier models. We have shown having frontier models. We have shown having frontier models. We have shown it, so have the attackers, it, so have the attackers, it, so have the attackers, unfortunately. With good context and

unfortunately. With good context and unfortunately. With good context and good harness, now you have an attacker good harness, now you have an attacker good harness, now you have an attacker that never sleeps. What does it do? The that never sleeps. What does it do? The that never sleeps. What does it do? The fundamentals of things like application fundamentals of things like application fundamentals of things like application security that was already something we security that was already something we security that was already something we tried to disrupt for 10 years have tried to disrupt for 10 years have tried to disrupt for 10 years have completely, you know, gone away. Like, completely, you know, gone away. Like, completely, you know, gone away. Like, oh, you cannot have contextual risk oh, you cannot have contextual risk oh, you cannot have contextual risk management and say, I fixed my criticals management and say, I fixed my criticals management and say, I fixed my criticals and I fixed my highs and I'm pretty good and I fixed my highs and I'm pretty good and I fixed my highs and I'm pretty good because everything else is too hard. No, because everything else is too hard. No, because everything else is too hard. No, you can string low vulnerabilities and you can string low vulnerabilities and you can string low vulnerabilities and and, you know, create exploits. You can and, you know, create exploits. You can and, you know, create exploits. You can do it without a lot of a harness with do it without a lot of a harness with do it without a lot of a harness with the mythos class model, as we've seen. the mythos class model, as we've seen. the mythos class model, as we've seen. And and but with a little bit of, you And and but with a little bit of, you And and but with a little bit of, you know, effort, you can do it with know, effort, you can do it with know, effort, you can do it with everything else. So, so that's a big everything else. So, so that's a big everything else. So, so that's a big big, you know, sea change in how, you big, you know, sea change in how, you big, you know, sea change in how, you know, we are seeing our customers know, we are seeing our customers know, we are seeing our customers actually react real time. actually react real time. actually react real time. And then you go, wow, a thousand And then you go, wow, a thousand And then you go, wow, a thousand enterprises spend over a million dollars enterprises spend over a million dollars enterprises spend over a million dollars for, you know, a cloud code roll out. for, you know, a cloud code roll out. for, you know, a cloud code roll out. So, this is the answer, right? Well, mhm So, this is the answer, right? Well, mhm So, this is the answer, right? Well, mhm maybe. Let's let's let's pull that maybe. Let's let's let's pull that maybe. Let's let's let's pull that string a little bit. string a little bit. string a little bit. The There are existing classes of The There are existing classes of The There are existing classes of problems that are getting worse. problems that are getting worse. problems that are getting worse. The quality of code is unfortunately The quality of code is unfortunately The quality of code is unfortunately worse than human-generated code. It's worse than human-generated code. It's worse than human-generated code. It's not like humans we I'm an engineer like not like humans we I'm an engineer like not like humans we I'm an engineer like I don't think I wrote perfect code, none I don't think I wrote perfect code, none I don't think I wrote perfect code, none of us do. So, but it is actually a of us do. So, but it is actually a of us do. So, but it is actually a little worse. And so, well, if that was little worse. And so, well, if that was little worse. And so, well, if that was the only problem, that's okay.

the only problem, that's okay. the only problem, that's okay. But what about the environment? All the But what about the environment? All the But what about the environment? All the things that we find as magical are based things that we find as magical are based things that we find as magical are based on skills and MCP servers and all these on skills and MCP servers and all these on skills and MCP servers and all these things that we want to share and use. things that we want to share and use. things that we want to share and use. And those are intentionally or And those are intentionally or And those are intentionally or unintentionally both, unintentionally both, unintentionally both, you know, being poisoned and, you know, you know, being poisoned and, you know, you know, being poisoned and, you know, malware's injected in there and we'll malware's injected in there and we'll malware's injected in there and we'll show some of the research on that. show some of the research on that. show some of the research on that. And then the behavior of the agent, And then the behavior of the agent, And then the behavior of the agent, right? And so, so we're having these right? And so, so we're having these right? And so, so we're having these these, you know, new patterns of these, you know, new patterns of these, you know, new patterns of problems compounding on top. problems compounding on top. problems compounding on top. And then who here doesn't want to become And then who here doesn't want to become And then who here doesn't want to become a a a an AI company or in any room in the an AI company or in any room in the an AI company or in any room in the world, right? Every company every world, right? Every company every world, right? Every company every board's like we're going to transform board's like we're going to transform board's like we're going to transform our business, our workflows, or you're our business, our workflows, or you're our business, our workflows, or you're starting brand new, you are fully starting brand new, you are fully starting brand new, you are fully agentic. And when you build with agents agentic. And when you build with agents agentic. And when you build with agents and models, that's an entirely new and models, that's an entirely new and models, that's an entirely new threat surface that was not part of the threat surface that was not part of the threat surface that was not part of the prior one. And so, these are really the prior one. And so, these are really the prior one. And so, these are really the fundamental problems, right? And And fundamental problems, right? And And fundamental problems, right? And And this is, you know, I want to share some this is, you know, I want to share some this is, you know, I want to share some of the real data. This is 4,800 plus of the real data. This is 4,800 plus of the real data. This is 4,800 plus customers customers customers in the last year. Their actual backlog in the last year. Their actual backlog in the last year. Their actual backlog quarter over quarter quarter over quarter quarter over quarter is like 108% more backlog.

is like 108% more backlog. is like 108% more backlog. So, So, So, remember that what I said about remember that what I said about remember that what I said about attackers? It's not just the existing attackers? It's not just the existing attackers? It's not just the existing vulnerabilities. vulnerabilities. vulnerabilities. And unfortunately, there's millions of And unfortunately, there's millions of And unfortunately, there's millions of them if you're an enterprise of any them if you're an enterprise of any them if you're an enterprise of any size. It's the fact that we are growing size. It's the fact that we are growing size. It's the fact that we are growing them despite the best agents, despite them despite the best agents, despite them despite the best agents, despite things that we have done and the things that we have done and the things that we have done and the industry has done. This trend is real industry has done. This trend is real industry has done. This trend is real and it's it's not good. and it's it's not good. and it's it's not good. And, you know, And, you know, And, you know, you have novel exploits, but they're not you have novel exploits, but they're not you have novel exploits, but they're not novel. Like the light LLM exploit, you novel. Like the light LLM exploit, you novel. Like the light LLM exploit, you think about like it's really taking think about like it's really taking think about like it's really taking existing vulnerabilities in the new existing vulnerabilities in the new existing vulnerabilities in the new surface and and chaining them together. surface and and chaining them together. surface and and chaining them together. And so, they create a much bigger blast And so, they create a much bigger blast And so, they create a much bigger blast radius right now at the pace at which radius right now at the pace at which radius right now at the pace at which work's being done. And yes, speed is a work's being done. And yes, speed is a work's being done. And yes, speed is a big part of this, but it's not just big part of this, but it's not just big part of this, but it's not just speed. speed. speed. And you know, just this um last week we And you know, just this um last week we And you know, just this um last week we had the Five Eyes uh you know, uh these had the Five Eyes uh you know, uh these had the Five Eyes uh you know, uh these are the the the Western world's uh are the the the Western world's uh are the the the Western world's uh intelligence leaders talking about AI intelligence leaders talking about AI intelligence leaders talking about AI will bypass cyber security systems will bypass cyber security systems will bypass cyber security systems in months, not years. Now, I don't share in months, not years. Now, I don't share in months, not years. Now, I don't share you know, share this I hate to have the you know, share this I hate to have the you know, share this I hate to have the scare tactic. It's just a fact. It's scare tactic. It's just a fact. It's scare tactic. It's just a fact. It's getting ready for getting ready for getting ready for you know, whatever is coming and we have you know, whatever is coming and we have you know, whatever is coming and we have already seen what's coming. It's just already seen what's coming. It's just already seen what's coming. It's just not widespread enough. And so, trying to not widespread enough. And so, trying to not widespread enough. And so, trying to like you know, chase systems and like like you know, chase systems and like like you know, chase systems and like the specific model will be the cure for the specific model will be the cure for the specific model will be the cure for all this is not the way all this is not the way all this is not the way is our point. The data on the second is our point. The data on the second is our point. The data on the second pain point, the untrusted output pain point, the untrusted output pain point, the untrusted output environment and behavior of agents.

environment and behavior of agents. environment and behavior of agents. And these are again facts. These are And these are again facts. These are And these are again facts. These are benchmark data facts. All of them the QR benchmark data facts. All of them the QR benchmark data facts. All of them the QR codes at the top are are if you want to codes at the top are are if you want to codes at the top are are if you want to like go check out the studies and the like go check out the studies and the like go check out the studies and the research behind it. research behind it. research behind it. This is you know, the amount of This is you know, the amount of This is you know, the amount of vulnerable code coming from the latest vulnerable code coming from the latest vulnerable code coming from the latest models. models. models. Plus the skills, there's toxic skills. Plus the skills, there's toxic skills. Plus the skills, there's toxic skills. We found the research the seminal We found the research the seminal We found the research the seminal research around how skills a third of research around how skills a third of research around how skills a third of them or more than third of them in all them or more than third of them in all them or more than third of them in all of the skills, not just you know, open of the skills, not just you know, open of the skills, not just you know, open claw, clawed, code acts, these skills claw, clawed, code acts, these skills claw, clawed, code acts, these skills actually have malware and they have actually have malware and they have actually have malware and they have vulnerabilities that are being Three vulnerabilities that are being Three vulnerabilities that are being Three lines of English are able to now bring a lines of English are able to now bring a lines of English are able to now bring a system down. system down. system down. So, you have to really understand the So, you have to really understand the So, you have to really understand the intent behind it. intent behind it. intent behind it. And the MCP servers, how do you connect And the MCP servers, how do you connect And the MCP servers, how do you connect to enterprise data? This is great to enterprise data? This is great to enterprise data? This is great protocol, very little security built in. protocol, very little security built in. protocol, very little security built in. It's getting better, It's getting better, It's getting better, but the foundations behind it is you but the foundations behind it is you but the foundations behind it is you know, this is the GitHub MCP server know, this is the GitHub MCP server know, this is the GitHub MCP server exploit that we highlighted to the year exploit that we highlighted to the year exploit that we highlighted to the year a year ago. a year ago. a year ago. And what did some of our customers do? And what did some of our customers do? And what did some of our customers do? Immediately shut down all MCP servers Immediately shut down all MCP servers Immediately shut down all MCP servers then they figured out that all of their then they figured out that all of their then they figured out that all of their dev screens and then how do you actually dev screens and then how do you actually dev screens and then how do you actually go back safely enabling things that are go back safely enabling things that are go back safely enabling things that are very powerful.

very powerful. very powerful. And then on the behavior, everyone knows And then on the behavior, everyone knows And then on the behavior, everyone knows the pocket OS example, right? What I the pocket OS example, right? What I the pocket OS example, right? What I have is real data in our own environment have is real data in our own environment have is real data in our own environment in our Fortune 100 customers. These in our Fortune 100 customers. These in our Fortune 100 customers. These agents go and create copies of PII data. agents go and create copies of PII data. agents go and create copies of PII data. Why? Somebody shared the PII data and Why? Somebody shared the PII data and Why? Somebody shared the PII data and and was trying to solve a real customer and was trying to solve a real customer and was trying to solve a real customer problem. problem. problem. The agent thought that maybe I should The agent thought that maybe I should The agent thought that maybe I should create a squirrel away copy of this in a create a squirrel away copy of this in a create a squirrel away copy of this in a database just in case I needed it again. database just in case I needed it again. database just in case I needed it again. That database is untrusted. Great. You That database is untrusted. Great. You That database is untrusted. Great. You now have an unknown attack surface that now have an unknown attack surface that now have an unknown attack surface that is not part of any enterprise security, is not part of any enterprise security, is not part of any enterprise security, you know, um, coverage. you know, um, coverage. you know, um, coverage. This is happening. So, how do you not This is happening. So, how do you not This is happening. So, how do you not put gates on it? How do you steer them, put gates on it? How do you steer them, put gates on it? How do you steer them, right? And so, you start going into, you right? And so, you start going into, you right? And so, you start going into, you know, know, know, the last part problem is you can't the last part problem is you can't the last part problem is you can't govern what you don't know exists. This govern what you don't know exists. This govern what you don't know exists. This is when you start building with agents. is when you start building with agents. is when you start building with agents. Now, this is real data from 3,000 plus Now, this is real data from 3,000 plus Now, this is real data from 3,000 plus customers who have used our abilities to customers who have used our abilities to customers who have used our abilities to really find the intelligence of what's really find the intelligence of what's really find the intelligence of what's in their code bases, what they're in their code bases, what they're in their code bases, what they're building. building. building. And for every model that we find in a And for every model that we find in a And for every model that we find in a repo, you have three times more agentic repo, you have three times more agentic repo, you have three times more agentic components in there. You have agents in components in there. You have agents in components in there. You have agents in the tools and everything that they use.

the tools and everything that they use. the tools and everything that they use. You have to figure out the full You have to figure out the full You have to figure out the full landscape because the risk is not just landscape because the risk is not just landscape because the risk is not just at one layer. And then once you find it, at one layer. And then once you find it, at one layer. And then once you find it, what do you how do you know how risky is what do you how do you know how risky is what do you how do you know how risky is it? What is your independent data it? What is your independent data it? What is your independent data verification? So, this is from this verification? So, this is from this verification? So, this is from this weekend from our risk BB that we have weekend from our risk BB that we have weekend from our risk BB that we have built our own attacks and red teaming built our own attacks and red teaming built our own attacks and red teaming capability. As new new models and new, capability. As new new models and new, capability. As new new models and new, uh, you know, components come out, we uh, you know, components come out, we uh, you know, components come out, we check them. check them. check them. The first two are your favorite frontier The first two are your favorite frontier The first two are your favorite frontier models. models. models. You can guess which ones those are. They You can guess which ones those are. They You can guess which ones those are. They did awesome on PI extraction like they did awesome on PI extraction like they did awesome on PI extraction like they didn't used to be so good on with our didn't used to be so good on with our didn't used to be so good on with our attacks. attacks. attacks. No PI extraction with our attacks. No PI extraction with our attacks. No PI extraction with our attacks. The third one is is your is is the hot The third one is is your is is the hot The third one is is your is is the hot new model in Silicon Valley especially new model in Silicon Valley especially new model in Silicon Valley especially or this you know last few weeks rhymes or this you know last few weeks rhymes or this you know last few weeks rhymes with LLM with LLM with LLM 100% 100% of the time our attacks were 100% 100% of the time our attacks were 100% 100% of the time our attacks were able to extract PI. able to extract PI. able to extract PI. But when you check a different test But when you check a different test But when you check a different test decision override decision override decision override the frontier models the frontier models the frontier models did worse. did worse. did worse. The open model The open model The open model 0% of the time you were able to override 0% of the time you were able to override 0% of the time you were able to override the decision at least with our attacks.

the decision at least with our attacks. the decision at least with our attacks. Knowing this allows you to know what to Knowing this allows you to know what to Knowing this allows you to know what to use when to use that and how do you use when to use that and how do you use when to use that and how do you control and these things are changing control and these things are changing control and these things are changing dynamically. dynamically. dynamically. So again going back to the original So again going back to the original So again going back to the original point right if none of that convinces point right if none of that convinces point right if none of that convinces you like the generator validator you like the generator validator you like the generator validator separation separation separation this is fresh new research from just I this is fresh new research from just I this is fresh new research from just I think yesterday's when we went public think yesterday's when we went public think yesterday's when we went public with this. It's a benchmark that no no with this. It's a benchmark that no no with this. It's a benchmark that no no model has been trained so we can trust model has been trained so we can trust model has been trained so we can trust it for now and we'll have to keep it for now and we'll have to keep it for now and we'll have to keep updating the benchmarks. updating the benchmarks. updating the benchmarks. But this is you know just a very simple But this is you know just a very simple But this is you know just a very simple thing. You're we're asking the latest thing. You're we're asking the latest thing. You're we're asking the latest models and we have access to everything models and we have access to everything models and we have access to everything as you can imagine. We're asking them to as you can imagine. We're asking them to as you can imagine. We're asking them to find you know the same vulnerability find you know the same vulnerability find you know the same vulnerability run it five times run it five times run it five times and only 50% of those ones are found and only 50% of those ones are found and only 50% of those ones are found across those five tests. That's not how across those five tests. That's not how across those five tests. That's not how you can run an enterprise system if you you can run an enterprise system if you you can run an enterprise system if you just use the LLM without any anything just use the LLM without any anything just use the LLM without any anything else. This is the latest models. Only else. This is the latest models. Only else. This is the latest models. Only 75% 75% 75% of the issues were found versus a good of the issues were found versus a good of the issues were found versus a good old boring deterministic check.

old boring deterministic check. old boring deterministic check. And you know 40% was the F1 score. And you know 40% was the F1 score. And you know 40% was the F1 score. So this whole like what did you actually So this whole like what did you actually So this whole like what did you actually miss? miss? miss? And we're talking about the latest And we're talking about the latest And we're talking about the latest models that are not even publicly models that are not even publicly models that are not even publicly available to people right? So what does available to people right? So what does available to people right? So what does this mean? It doesn't mean that they're this mean? It doesn't mean that they're this mean? It doesn't mean that they're not good. It just means they need to be not good. It just means they need to be not good. It just means they need to be you really need to use them for what you really need to use them for what you really need to use them for what they're really good at together and they're really good at together and they're really good at together and carefully carefully carefully to find the surface that your to find the surface that your to find the surface that your deterministic check cannot use not just deterministic check cannot use not just deterministic check cannot use not just think about probabilistic systems will think about probabilistic systems will think about probabilistic systems will solve everything. solve everything. solve everything. And so, you know, to net it out, what And so, you know, to net it out, what And so, you know, to net it out, what we've been working on, we've been working on, we've been working on, we don't have all the answers. And I'll we don't have all the answers. And I'll we don't have all the answers. And I'll share where we're going to. And share where we're going to. And share where we're going to. And but what we have answers for right now but what we have answers for right now but what we have answers for right now on the automated attacks that are on the automated attacks that are on the automated attacks that are working in some of these very large working in some of these very large working in some of these very large environments, just you know, how do I environments, just you know, how do I environments, just you know, how do I prevent new issues from coming into the prevent new issues from coming into the prevent new issues from coming into the agentic loop? agentic loop? agentic loop? To put put security context right there, To put put security context right there, To put put security context right there, that's studio. You can go check it out that's studio. You can go check it out that's studio. You can go check it out on our website. And we're on our website. And we're on our website. And we're you know, everyone's worried about you know, everyone's worried about you know, everyone's worried about packages that they download. Like how we packages that they download. Like how we packages that they download. Like how we have we know what the health of the have we know what the health of the have we know what the health of the packages are. We know the vulnerability packages are. We know the vulnerability packages are. We know the vulnerability information. We know it's malware. So, information. We know it's malware. So, information. We know it's malware. So, we're able to prevent the the agent we're able to prevent the the agent we're able to prevent the the agent from, you know, picking a package like from, you know, picking a package like from, you know, picking a package like that or writing code that, you know, that or writing code that, you know, that or writing code that, you know, inherently is a sequel injection.

inherently is a sequel injection. inherently is a sequel injection. But great, so prevention will prevent But great, so prevention will prevent But great, so prevention will prevent that hockey stick, which we all want to that hockey stick, which we all want to that hockey stick, which we all want to prevent. Well, what happens with, you prevent. Well, what happens with, you prevent. Well, what happens with, you know, um the fact that, you know, you know, um the fact that, you know, you know, um the fact that, you know, you have this this mountain of have this this mountain of have this this mountain of vulnerability. We've been able to take vulnerability. We've been able to take vulnerability. We've been able to take organizations like Labelbox to zero organizations like Labelbox to zero organizations like Labelbox to zero once, zero backlog. once, zero backlog. once, zero backlog. Which is very hard in security due to Which is very hard in security due to Which is very hard in security due to because it breaks applications. So, you because it breaks applications. So, you because it breaks applications. So, you need to know concepts like breakability. need to know concepts like breakability. need to know concepts like breakability. And that data is super important that And that data is super important that And that data is super important that we're able to get from our base to know we're able to get from our base to know we're able to get from our base to know this is a safe upgrade. And so, that, this is a safe upgrade. And so, that, this is a safe upgrade. And so, that, you know, just last week a max seven you know, just last week a max seven you know, just last week a max seven company remediated 16,000 critical company remediated 16,000 critical company remediated 16,000 critical issues issues issues using this remediation agent. Again, using this remediation agent. Again, using this remediation agent. Again, something you can go try out. So, something you can go try out. So, something you can go try out. So, you know, this is how, you know, I you know, this is how, you know, I you know, this is how, you know, I talked about, you know, kind of from the talked about, you know, kind of from the talked about, you know, kind of from the untrusted agentic development. We just untrusted agentic development. We just untrusted agentic development. We just GA'd our agentic dev security offering GA'd our agentic dev security offering GA'd our agentic dev security offering yesterday. It's looking at the yesterday. It's looking at the yesterday. It's looking at the environment, the output, the skills, the environment, the output, the skills, the environment, the output, the skills, the MCP servers, and the behavior of coding MCP servers, and the behavior of coding MCP servers, and the behavior of coding agents like Cursor Cloud, um Codex, and agents like Cursor Cloud, um Codex, and agents like Cursor Cloud, um Codex, and others. others. others. And when you're building ungoverned AI And when you're building ungoverned AI And when you're building ungoverned AI apps, apps, apps, that AI governance cannot live in a that AI governance cannot live in a that AI governance cannot live in a confluence page or PDF. So, how do you confluence page or PDF. So, how do you confluence page or PDF. So, how do you real-time look at everything that is real-time look at everything that is real-time look at everything that is happening in a very fast moving complex happening in a very fast moving complex happening in a very fast moving complex code repositories and understand risk code repositories and understand risk code repositories and understand risk and have policies enforced in the loops and have policies enforced in the loops and have policies enforced in the loops that the agents and the devs are.

that the agents and the devs are. that the agents and the devs are. So, seeing is believing. I would love to So, seeing is believing. I would love to So, seeing is believing. I would love to bring Ezra up here to do a quick demo. bring Ezra up here to do a quick demo. bring Ezra up here to do a quick demo. Um and uh Um and uh Um and uh you know, Ezra's going to show you a you know, Ezra's going to show you a you know, Ezra's going to show you a couple things and you can come by to our couple things and you can come by to our couple things and you can come by to our booth and uh share a few more booth and uh share a few more booth and uh share a few more later. later. later. >> Thanks, Manoj. >> Thanks, Manoj. >> Thanks, Manoj. Let's flip over to the CLI here. I'm Let's flip over to the CLI here. I'm Let's flip over to the CLI here. I'm going to do a few rapid-fire demos. going to do a few rapid-fire demos. going to do a few rapid-fire demos. Don't have time to do everything Manoj Don't have time to do everything Manoj Don't have time to do everything Manoj just talked about here, but we'd love just talked about here, but we'd love just talked about here, but we'd love for you to come and and talk to us. Stop for you to come and and talk to us. Stop for you to come and and talk to us. Stop by Stop by the booth. Find us after this by Stop by the booth. Find us after this by Stop by the booth. Find us after this talk here. Uh the first thing that I'm talk here. Uh the first thing that I'm talk here. Uh the first thing that I'm going to do here is I'm going to ask going to do here is I'm going to ask going to do here is I'm going to ask Claude to generate Claude to generate Claude to generate a tool a CLI tool that can create a QR a tool a CLI tool that can create a QR a tool a CLI tool that can create a QR code image from uh from a prompt. code image from uh from a prompt. code image from uh from a prompt. >> Start Codex or Claude. Uh there we go. >> Start Codex or Claude. Uh there we go. >> Start Codex or Claude. Uh there we go. Please start Claude. Thank you. Please start Claude. Thank you. Please start Claude. Thank you. Live demos. Live demos. Live demos. Let's try that one more time. All right. Uh try number two here. We All right. Uh try number two here. We are uh are uh are uh going to try to get Claude to generate a going to try to get Claude to generate a going to try to get Claude to generate a CLI tool um CLI tool um CLI tool um to ultimately create that that QR code to ultimately create that that QR code to ultimately create that that QR code image. I'm asking it to use an open image. I'm asking it to use an open image. I'm asking it to use an open source dependency and if you notice this source dependency and if you notice this source dependency and if you notice this fourth uh fourth line that I have here fourth uh fourth line that I have here fourth uh fourth line that I have here um um um I'm being pretty verbose. I'm telling it I'm being pretty verbose. I'm telling it I'm being pretty verbose. I'm telling it to use the Snyk package health check to use the Snyk package health check to use the Snyk package health check tool. Um and so we're going to be able tool. Um and so we're going to be able tool. Um and so we're going to be able to see that here in the demo, but if you to see that here in the demo, but if you to see that here in the demo, but if you use this in practice, we really use this in practice, we really use this in practice, we really encourage you encourage you encourage you to leverage a uh a skill or a hook to to leverage a uh a skill or a hook to to leverage a uh a skill or a hook to make this just a deterministic automated make this just a deterministic automated make this just a deterministic automated part of your workflow.

part of your workflow. part of your workflow. So, the first thing that it's going to So, the first thing that it's going to So, the first thing that it's going to do is going to see, you know, ultimately do is going to see, you know, ultimately do is going to see, you know, ultimately what what does the project look like and what what does the project look like and what what does the project look like and what are some dependencies some open what are some dependencies some open what are some dependencies some open source packages that I might be able to source packages that I might be able to source packages that I might be able to use. use. use. And let's see if we can expand this. And let's see if we can expand this. And let's see if we can expand this. I don't know if conference Wi-Fi is I don't know if conference Wi-Fi is I don't know if conference Wi-Fi is going to play nice with us. going to play nice with us. going to play nice with us. Now, we can switch to a recorded demo Now, we can switch to a recorded demo Now, we can switch to a recorded demo here, but was really hoping to to do here, but was really hoping to to do here, but was really hoping to to do this live. Is that a little better? Is that a little better? That even more? That even more? That even more? All right. Great. Uh so, we can see now that it is Great. Uh so, we can see now that it is calling this uh package health tool uh calling this uh package health tool uh calling this uh package health tool uh for two different open source for two different open source for two different open source dependencies. It's looking at a QR code dependencies. It's looking at a QR code dependencies. It's looking at a QR code package and a QR image package. Both package and a QR image package. Both package and a QR image package. Both viable packages that can help accomplish viable packages that can help accomplish viable packages that can help accomplish this task. this task. this task. Uh results are returning. Now, it looks like conference Wi-Fi is Now, it looks like conference Wi-Fi is really really not playing in our favor really really not playing in our favor really really not playing in our favor here. Uh so, I'm going to quickly jump here. Uh so, I'm going to quickly jump here. Uh so, I'm going to quickly jump over to recorded demo. Apologies. Come over to recorded demo. Apologies. Come over to recorded demo. Apologies. Come find us after. We can go to hopefully a find us after. We can go to hopefully a find us after. We can go to hopefully a space where there's not as much of this space where there's not as much of this space where there's not as much of this happening.

happening. happening. Uh if we can get this live here. There we are. So, same prompt. There we are. So, same prompt. Maybe I can fast forward a bit so you Maybe I can fast forward a bit so you Maybe I can fast forward a bit so you don't need to hear me me banter on this don't need to hear me me banter on this don't need to hear me me banter on this a little bit. a little bit. a little bit. Uh but, ultimately uh we see here that Uh but, ultimately uh we see here that Uh but, ultimately uh we see here that the two images Excuse me, the two the the two images Excuse me, the two the the two images Excuse me, the two the two packages returned. Both are actually two packages returned. Both are actually two packages returned. Both are actually uh vulnerability free. There's no active uh vulnerability free. There's no active uh vulnerability free. There's no active CVEs that could be exploited in these CVEs that could be exploited in these CVEs that could be exploited in these versions. versions. versions. Uh but, the first one, QR code, is Uh but, the first one, QR code, is Uh but, the first one, QR code, is healthy, meaning it's actively being healthy, meaning it's actively being healthy, meaning it's actively being maintained. maintained. maintained. Um and there's a ton of a ton of active Um and there's a ton of a ton of active Um and there's a ton of a ton of active usage downloads of this. Whereas, QR usage downloads of this. Whereas, QR usage downloads of this. Whereas, QR image, no CVEs today, but it's not image, no CVEs today, but it's not image, no CVEs today, but it's not actively being maintained. It was really actively being maintained. It was really actively being maintained. It was really released 10 years ago for the first released 10 years ago for the first released 10 years ago for the first time. And so, if I were to deploy time. And so, if I were to deploy time. And so, if I were to deploy software with this right now, I may not software with this right now, I may not software with this right now, I may not get exploited today if I use either get exploited today if I use either get exploited today if I use either package, but if there was a new package, but if there was a new package, but if there was a new vulnerability identified in the future, vulnerability identified in the future, vulnerability identified in the future, um there's a much higher likelihood that um there's a much higher likelihood that um there's a much higher likelihood that if I'm using this QR code package here, if I'm using this QR code package here, if I'm using this QR code package here, that a patch would be released sooner, that a patch would be released sooner, that a patch would be released sooner, within a day or two, and I would be able within a day or two, and I would be able within a day or two, and I would be able to continue uh building on this right to continue uh building on this right to continue uh building on this right now.

Uh the second demo that I wanted to show Uh the second demo that I wanted to show here was exploring that uh assessment of here was exploring that uh assessment of here was exploring that uh assessment of risk from skills uh or MCP servers that risk from skills uh or MCP servers that risk from skills uh or MCP servers that a my agent might be using here. And so, a my agent might be using here. And so, a my agent might be using here. And so, somebody shared with me a skill uh somebody shared with me a skill uh somebody shared with me a skill uh called Do we have it here? called Do we have it here? called Do we have it here? Um Um Um a competitive analysis skill. Um and I a competitive analysis skill. Um and I a competitive analysis skill. Um and I ran the skill assessment against ran the skill assessment against ran the skill assessment against Excuse me, the risk assessment against Excuse me, the risk assessment against Excuse me, the risk assessment against the skill here, and we saw there were the skill here, and we saw there were the skill here, and we saw there were four findings returned. And some of four findings returned. And some of four findings returned. And some of these are fairly problematic. We can see these are fairly problematic. We can see these are fairly problematic. We can see in the actual skill itself, it's asking in the actual skill itself, it's asking in the actual skill itself, it's asking me to echo the authorization header, me to echo the authorization header, me to echo the authorization header, which is a big no-no. That's which is a big no-no. That's which is a big no-no. That's That's not something that we want a That's not something that we want a That's not something that we want a skill ultimately to be doing. We also skill ultimately to be doing. We also skill ultimately to be doing. We also see that in some cases, it's going to be see that in some cases, it's going to be see that in some cases, it's going to be pulling from live content from Reddit, pulling from live content from Reddit, pulling from live content from Reddit, from Twitter. That may be totally fine, from Twitter. That may be totally fine, from Twitter. That may be totally fine, but you really want to go into that eyes but you really want to go into that eyes but you really want to go into that eyes wide open and making make sure that you wide open and making make sure that you wide open and making make sure that you know that it's pulling in this know that it's pulling in this know that it's pulling in this information, the way you're using the information, the way you're using the information, the way you're using the skill is going to be appropriate. For skill is going to be appropriate. For skill is going to be appropriate. For competitive analysis, that's probably competitive analysis, that's probably competitive analysis, that's probably probably reasonable. probably reasonable. probably reasonable. Um but what's especially problematic, if Um but what's especially problematic, if Um but what's especially problematic, if we see in this line here, we see in this line here, we see in this line here, is that it's actually looking to pull is that it's actually looking to pull is that it's actually looking to pull from a YAML file that's hosted on the from a YAML file that's hosted on the from a YAML file that's hosted on the internet, instructions on how to monitor internet, instructions on how to monitor internet, instructions on how to monitor these targets and some of the these targets and some of the these targets and some of the classification rules. So, it's really classification rules. So, it's really classification rules. So, it's really giving it the logic to actually execute giving it the logic to actually execute giving it the logic to actually execute the skill from a third-party website.

the skill from a third-party website. the skill from a third-party website. And if that gets changed, even if my And if that gets changed, even if my And if that gets changed, even if my skill file doesn't change at all, that skill file doesn't change at all, that skill file doesn't change at all, that is a a chance for an exploit to occur. is a a chance for an exploit to occur. is a a chance for an exploit to occur. It's worth noting that we've been It's worth noting that we've been It's worth noting that we've been refining the logic associated with refining the logic associated with refining the logic associated with the findings that we return here, the findings that we return here, the findings that we return here, addressing the signal-to-noise ratio addressing the signal-to-noise ratio addressing the signal-to-noise ratio and kind of giving some policy and kind of giving some policy and kind of giving some policy configuration capabilities as part of configuration capabilities as part of configuration capabilities as part of our Evo product. our Evo product. our Evo product. And we're going to see the gap kind of And we're going to see the gap kind of And we're going to see the gap kind of get closed here with this open-source get closed here with this open-source get closed here with this open-source tool as well in the future. We encourage tool as well in the future. We encourage tool as well in the future. We encourage you to check these out. Go to sneak.io, you to check these out. Go to sneak.io, you to check these out. Go to sneak.io, come talk to us in the booth. Find Find come talk to us in the booth. Find Find come talk to us in the booth. Find Find me in the hallway. We can actually do me in the hallway. We can actually do me in the hallway. We can actually do this live instead of looking at a this live instead of looking at a this live instead of looking at a recording recording recording if you want. But I think given the time, if you want. But I think given the time, if you want. But I think given the time, we probably should wrap the demo. we probably should wrap the demo. we probably should wrap the demo. >> Cool. >> [applause] >> [applause] >> Live demos are always fun, but Murphy's >> Live demos are always fun, but Murphy's >> Live demos are always fun, but Murphy's striking always, but you had a plan B, striking always, but you had a plan B, striking always, but you had a plan B, so that's awesome. So, look, so that's awesome. So, look, so that's awesome. So, look, just to wrap it up, right? This is just just to wrap it up, right? This is just just to wrap it up, right? This is just the beginning. Yes, we have built some the beginning. Yes, we have built some the beginning. Yes, we have built some very interesting agents for some very interesting agents for some very interesting agents for some specific sets of those problems. specific sets of those problems. specific sets of those problems. In the end, it's a you know, a system In the end, it's a you know, a system In the end, it's a you know, a system that we talked about that really like that we talked about that really like that we talked about that really like it's it's rooms like this that we're it's it's rooms like this that we're it's it's rooms like this that we're building with building with building with in our customer base and and we would in our customer base and and we would in our customer base and and we would love to partner with the rest of the love to partner with the rest of the love to partner with the rest of the industry while we're doing this track.

industry while we're doing this track. industry while we're doing this track. And we learned, you know, this as I And we learned, you know, this as I And we learned, you know, this as I mentioned Evo, what is Evo? Evo is the mentioned Evo, what is Evo? Evo is the mentioned Evo, what is Evo? Evo is the system that we we brought to the world system that we we brought to the world system that we we brought to the world late last year in terms of a concept and late last year in terms of a concept and late last year in terms of a concept and we've constantly built these different we've constantly built these different we've constantly built these different agents. agents. agents. We learned from other systems, you know, We learned from other systems, you know, We learned from other systems, you know, you have fighter pilots with 5G fighters you have fighter pilots with 5G fighters you have fighter pilots with 5G fighters and how are they trained? It is this and how are they trained? It is this and how are they trained? It is this notion of you need to really observe, notion of you need to really observe, notion of you need to really observe, orient, decide and act and that's how orient, decide and act and that's how orient, decide and act and that's how you go and in the constant learning from you go and in the constant learning from you go and in the constant learning from that kind of loop is how you become, you that kind of loop is how you become, you that kind of loop is how you become, you know, a super pilot. And so we want to know, a super pilot. And so we want to know, a super pilot. And so we want to enable this room and other rooms. We had enable this room and other rooms. We had enable this room and other rooms. We had a workshop yesterday training hundreds a workshop yesterday training hundreds a workshop yesterday training hundreds of AI security engineers. We want to of AI security engineers. We want to of AI security engineers. We want to enable AI security engineers to be able enable AI security engineers to be able enable AI security engineers to be able to have their own powerful system and to have their own powerful system and to have their own powerful system and tools and yes, there are problems that tools and yes, there are problems that tools and yes, there are problems that are not solved yet fully in terms of are not solved yet fully in terms of are not solved yet fully in terms of coordinating multiple agents with coordinating multiple agents with coordinating multiple agents with systems like this. Yes, we all know how, systems like this. Yes, we all know how, systems like this. Yes, we all know how, you know, the harnesses and and and you know, the harnesses and and and you know, the harnesses and and and shared memory and and we're working shared memory and and we're working shared memory and and we're working these problems. This is how we're these problems. This is how we're these problems. This is how we're building Evo. We're building it with the building Evo. We're building it with the building Evo. We're building it with the community and we want this to be an open community and we want this to be an open community and we want this to be an open vision. That's really empowering AI vision. That's really empowering AI vision. That's really empowering AI security engineers. You know, from from security engineers. You know, from from security engineers. You know, from from these rooms the AI engineers became 10x these rooms the AI engineers became 10x these rooms the AI engineers became 10x engineers. Our goal here is to have that engineers. Our goal here is to have that engineers. Our goal here is to have that 10x superpower in the hands of AI 10x superpower in the hands of AI 10x superpower in the hands of AI security engineers so we can build security engineers so we can build security engineers so we can build trusted systems together. So let's trusted systems together. So let's trusted systems together. So let's continue to do that and you know, come continue to do that and you know, come continue to do that and you know, come by, talk to us or if you guys are, you by, talk to us or if you guys are, you by, talk to us or if you guys are, you know, have have know, have have know, have have I think there was like 50 plus events I think there was like 50 plus events I think there was like 50 plus events around AI engineer but we're we're

around AI engineer but we're we're around AI engineer but we're we're having this rooftop fan zone. We're having this rooftop fan zone. We're having this rooftop fan zone. We're going to do some some more things going to do some some more things going to do some some more things together with the community. So this is together with the community. So this is together with the community. So this is down the road in our office here in SF. down the road in our office here in SF. down the road in our office here in SF. Uh community jam. Come join us uh if you Uh community jam. Come join us uh if you Uh community jam. Come join us uh if you if you have time this evening. We'll if you have time this evening. We'll if you have time this evening. We'll continue the conversation. Thank you continue the conversation. Thank you continue the conversation. Thank you all. all. all. Let's continue building. Let's continue building. Let's continue building. >> [applause]