The Joy of Unplugging Cables: Kelly Shortridge on Security Resilience

I remember back in the days with I remember back in the days with Twitter, there was cyber squirrel where Twitter, there was cyber squirrel where Twitter, there was cyber squirrel where it talked about all the power plant it talked about all the power plant it talked about all the power plant failures caused by squirrels just doing failures caused by squirrels just doing failures caused by squirrels just doing things squirrels do, you know, and how things squirrels do, you know, and how things squirrels do, you know, and how they were almost the more threatening they were almost the more threatening they were almost the more threatening advanced persistent threat because of advanced persistent threat because of advanced persistent threat because of the damage that they wrought. I think the damage that they wrought. I think the damage that they wrought. I think there's just so many examples of like there's just so many examples of like there's just so many examples of like your best intentions, you know, reality your best intentions, you know, reality your best intentions, you know, reality is stranger than fiction. You're not is stranger than fiction. You're not is stranger than fiction. You're not going to be able to dream up every going to be able to dream up every going to be able to dream up every scenario that's possible, so you have to scenario that's possible, so you have to scenario that's possible, so you have to prepare for the idea of okay, things prepare for the idea of okay, things prepare for the idea of okay, things will go wrong. How do we minimize impact will go wrong. How do we minimize impact will go wrong. How do we minimize impact to make sure we can evolve to like meet to make sure we can evolve to like meet to make sure we can evolve to like meet the moment? Yeah. Hi, I'm Scott the moment? Yeah. Hi, I'm Scott the moment? Yeah. Hi, I'm Scott Hanselman. This is another episode of Hanselman. This is another episode of Hanselman. This is another episode of Hanselminutes in association with the Hanselminutes in association with the Hanselminutes in association with the ACM Bytecast. And today I have the honor ACM Bytecast. And today I have the honor ACM Bytecast. And today I have the honor of speaking with Kelly Shortridge. She's of speaking with Kelly Shortridge. She's of speaking with Kelly Shortridge. She's a chief product officer at Fastly. How's a chief product officer at Fastly. How's a chief product officer at Fastly. How's it going? it going? it going? It's going very well. It's a beautiful It's going very well. It's a beautiful It's going very well. It's a beautiful spring day here in New York. It is a spring day here in New York. It is a spring day here in New York. It is a beautiful spring day. I've gotten some beautiful spring day. I've gotten some beautiful spring day. I've gotten some sunshine today and I feel a lot better. sunshine today and I feel a lot better. sunshine today and I feel a lot better. Um Um Um everything sucks, but it just sucks everything sucks, but it just sucks everything sucks, but it just sucks slightly less when it's sunny. slightly less when it's sunny. slightly less when it's sunny. That is true and things are blooming. I That is true and things are blooming. I That is true and things are blooming. I can't complain. Yeah, absolutely. So you can't complain. Yeah, absolutely. So you can't complain. Yeah, absolutely. So you are the author of security chaos are the author of security chaos are the author of security chaos engineering, sustaining resilience in engineering, sustaining resilience in engineering, sustaining resilience in software and systems and I spent the software and systems and I spent the software and systems and I spent the weekend reading the book and trying to weekend reading the book and trying to weekend reading the book and trying to understand where the intersection of understand where the intersection of understand where the intersection of chaos engineering and security chaos engineering and security chaos engineering and security engineering is because like I remember engineering is because like I remember engineering is because like I remember when Chaos Monkey was a thing and I just when Chaos Monkey was a thing and I just when Chaos Monkey was a thing and I just got to imagine all the Netflix people got to imagine all the Netflix people got to imagine all the Netflix people running around pulling cables and the running around pulling cables and the running around pulling cables and the monkey was just messing up their stuff.

monkey was just messing up their stuff. monkey was just messing up their stuff. And now I'm trying to understand the And now I'm trying to understand the And now I'm trying to understand the intersection of security engineering intersection of security engineering intersection of security engineering with chaos engineering. I wonder if you with chaos engineering. I wonder if you with chaos engineering. I wonder if you could help me understand that. could help me understand that. could help me understand that. Yes, I think it's better characterized Yes, I think it's better characterized Yes, I think it's better characterized by the umbrella of resilience by the umbrella of resilience by the umbrella of resilience engineering if anything. engineering if anything. engineering if anything. I've actually had the rare and I've actually had the rare and I've actually had the rare and delectable pleasure of unplugging cables delectable pleasure of unplugging cables delectable pleasure of unplugging cables from Fastly's pops. Of course, network from Fastly's pops. Of course, network from Fastly's pops. Of course, network continued working perfectly. It is a continued working perfectly. It is a continued working perfectly. It is a thrill though, I will admit. Part of the thrill though, I will admit. Part of the thrill though, I will admit. Part of the title with the book is with a little title with the book is with a little title with the book is with a little behind the scenes tea is chaos behind the scenes tea is chaos behind the scenes tea is chaos engineering, especially at the time was engineering, especially at the time was engineering, especially at the time was a big buzzword. The book is certainly a big buzzword. The book is certainly a big buzzword. The book is certainly more than just chaos engineering. That more than just chaos engineering. That more than just chaos engineering. That is one tool and kind of the resilience is one tool and kind of the resilience is one tool and kind of the resilience engineering toolkit. When we think about engineering toolkit. When we think about engineering toolkit. When we think about resilience, it's ultimately about how do resilience, it's ultimately about how do resilience, it's ultimately about how do you recover from failure of any kind and you recover from failure of any kind and you recover from failure of any kind and prepare for what's next? That what's prepare for what's next? That what's prepare for what's next? That what's next could be a threat, but equally it next could be a threat, but equally it next could be a threat, but equally it could be a business opportunity. It could be a business opportunity. It could be a business opportunity. It could be, you know, massive traffic could be, you know, massive traffic could be, you know, massive traffic growth for good reasons or it's a DDoS. growth for good reasons or it's a DDoS. growth for good reasons or it's a DDoS. So security really is a subset of the So security really is a subset of the So security really is a subset of the both surprises, stressors, both surprises, stressors, both surprises, stressors, opportunities, and threats in a very opportunities, and threats in a very opportunities, and threats in a very broad sense that we need to think about.

broad sense that we need to think about. broad sense that we need to think about. This might be a dumb question. It could This might be a dumb question. It could This might be a dumb question. It could be a spicy question, but like why call be a spicy question, but like why call be a spicy question, but like why call it security chaos engineering? Is it security chaos engineering? Is it security chaos engineering? Is because those are fun words? Because because those are fun words? Because because those are fun words? Because like resilience engineering didn't like resilience engineering didn't like resilience engineering didn't wouldn't fly off the shelves? Cuz it wouldn't fly off the shelves? Cuz it wouldn't fly off the shelves? Cuz it seems very clear that like resilience is seems very clear that like resilience is seems very clear that like resilience is really what we want, but it's just not a really what we want, but it's just not a really what we want, but it's just not a sexy term. sexy term. sexy term. I mean, I think this is a classic I mean, I think this is a classic I mean, I think this is a classic tension always when you're trying to tension always when you're trying to tension always when you're trying to publish, you know, a book or frankly a publish, you know, a book or frankly a publish, you know, a book or frankly a movie, you know, you [clears throat] movie, you know, you [clears throat] movie, you know, you [clears throat] have to have some sort of catchy name. have to have some sort of catchy name. have to have some sort of catchy name. No, that's a great point. Chaos would be No, that's a great point. Chaos would be No, that's a great point. Chaos would be an awesome movie name, but resilience is an awesome movie name, but resilience is an awesome movie name, but resilience is like it's more of an A24 movie. like it's more of an A24 movie. like it's more of an A24 movie. Exactly. A24 vibe. It's also, you know, Exactly. A24 vibe. It's also, you know, Exactly. A24 vibe. It's also, you know, you know, talked about it Davos and it's you know, talked about it Davos and it's you know, talked about it Davos and it's in, you know, the National Association in, you know, the National Association in, you know, the National Association of Corporate Directors book around of Corporate Directors book around of Corporate Directors book around organizational resilience. It is a great organizational resilience. It is a great organizational resilience. It is a great Latin root word for international Latin root word for international Latin root word for international appeal, but I think chaos people like appeal, but I think chaos people like appeal, but I think chaos people like wait a second, chaos can be a good wait a second, chaos can be a good wait a second, chaos can be a good thing? That doesn't sound right. thing? That doesn't sound right. thing? That doesn't sound right. Absolutely. I love the title. I want to Absolutely. I love the title. I want to Absolutely. I love the title. I want to engineer chaos. That'll be a very very engineer chaos. That'll be a very very engineer chaos. That'll be a very very exciting. Exactly. exciting. Exactly. exciting. Exactly. >> [snorts] >> [snorts] >> [snorts] >> Now you have said that security should >> Now you have said that security should >> Now you have said that security should be designed for failure, not for be designed for failure, not for be designed for failure, not for prevention and I think that's a really prevention and I think that's a really prevention and I think that's a really cool way to think about that. Can you cool way to think about that. Can you cool way to think about that. Can you think of an example where there's a think of an example where there's a think of an example where there's a perfectly secure system that still perfectly secure system that still perfectly secure system that still failed in the real world? Like what's an failed in the real world? Like what's an failed in the real world? Like what's an example where oh, it still happened and example where oh, it still happened and example where oh, it still happened and we couldn't stop it? I mean, I feel like we couldn't stop it? I mean, I feel like we couldn't stop it? I mean, I feel like tons. First, no such thing as a tons. First, no such thing as a tons. First, no such thing as a perfectly secure system, right? I think perfectly secure system, right? I think perfectly secure system, right? I think there's so many esoteric failures out there's so many esoteric failures out there's so many esoteric failures out there. I think about there. I think about there. I think about the airline industry has learned many the airline industry has learned many the airline industry has learned many years ahead of software about the

years ahead of software about the years ahead of software about the intricate nature of complex systems and intricate nature of complex systems and intricate nature of complex systems and all the failures that could go wrong, all the failures that could go wrong, all the failures that could go wrong, but the example I always think about is but the example I always think about is but the example I always think about is the fact that they designed the fact that they designed the fact that they designed I forget which airplane it was, which I forget which airplane it was, which I forget which airplane it was, which model. They designed it with safety in model. They designed it with safety in model. They designed it with safety in mind to almost every degree except for mind to almost every degree except for mind to almost every degree except for the fact that in a very bizarre the fact that in a very bizarre the fact that in a very bizarre scenario, if you spilled scenario, if you spilled scenario, if you spilled or not spilled, if you somehow exploded or not spilled, if you somehow exploded or not spilled, if you somehow exploded the coffee maker the coffee maker the coffee maker by boiling it too hot or something, it by boiling it too hot or something, it by boiling it too hot or something, it happened to be close enough to like the happened to be close enough to like the happened to be close enough to like the panel with some cables that it could panel with some cables that it could panel with some cables that it could cause a like critical failure while the cause a like critical failure while the cause a like critical failure while the plane was in air. Oh my god. plane was in air. Oh my god. plane was in air. Oh my god. Right. You wouldn't think about that as Right. You wouldn't think about that as Right. You wouldn't think about that as like that is the trigger to like a like that is the trigger to like a like that is the trigger to like a massive failure that means there has to massive failure that means there has to massive failure that means there has to be like an emergency landing, but yet be like an emergency landing, but yet be like an emergency landing, but yet there they were. So I think looking at there they were. So I think looking at there they were. So I think looking at real world systems and all the just real world systems and all the just real world systems and all the just bizarre ways that they can fall apart. bizarre ways that they can fall apart. bizarre ways that they can fall apart. >> Yeah. I think it's so important also to >> Yeah. I think it's so important also to >> Yeah. I think it's so important also to remember that like remember that like remember that like and maybe this is also a little spicy. and maybe this is also a little spicy. and maybe this is also a little spicy. Like it's on you to be responsible for Like it's on you to be responsible for Like it's on you to be responsible for your own resilience. And I remember in your own resilience. And I remember in your own resilience. And I remember in the early days of the cloud when we were the early days of the cloud when we were the early days of the cloud when we were all trying to get five nines out of all trying to get five nines out of all trying to get five nines out of Azure and five nines out of AWS. It's Azure and five nines out of AWS. It's Azure and five nines out of AWS. It's like like like okay, Azure went down. I'm going to call okay, Azure went down. I'm going to call okay, Azure went down. I'm going to call somebody and yell at them and it's to somebody and yell at them and it's to somebody and yell at them and it's to say, but how badly do you want your site say, but how badly do you want your site say, but how badly do you want your site to be up all the time? Do you want it to be up all the time? Do you want it to be up all the time? Do you want it badly enough that you're going to put it badly enough that you're going to put it badly enough that you're going to put it in both Azure and AWS?

in both Azure and AWS? in both Azure and AWS? How much do you want a copy of like how How much do you want a copy of like how How much do you want a copy of like how do you make a plane that doesn't crash? do you make a plane that doesn't crash? do you make a plane that doesn't crash? Do you fly two planes next to each Do you fly two planes next to each Do you fly two planes next to each other? And then when one fails like you other? And then when one fails like you other? And then when one fails like you jump to the other plane? Like it is all jump to the other plane? Like it is all jump to the other plane? Like it is all admittedly on us, is it not? And we just admittedly on us, is it not? And we just admittedly on us, is it not? And we just need to decide how hard to squeeze. need to decide how hard to squeeze. need to decide how hard to squeeze. I think there is usually a trade-off if I think there is usually a trade-off if I think there is usually a trade-off if you want to really simplify it between you want to really simplify it between you want to really simplify it between cost and resilience. To your point, you cost and resilience. To your point, you cost and resilience. To your point, you know, ultimately redundancy is multiple know, ultimately redundancy is multiple know, ultimately redundancy is multiple paths to get to the same goal. In paths to get to the same goal. In paths to get to the same goal. In practice, you need polyglot applications practice, you need polyglot applications practice, you need polyglot applications and systems. That's pretty expensive to and systems. That's pretty expensive to and systems. That's pretty expensive to pull off. I do think though that pull off. I do think though that pull off. I do think though that software has the beautiful luxury we software has the beautiful luxury we software has the beautiful luxury we sometimes don't leverage. To your point sometimes don't leverage. To your point sometimes don't leverage. To your point about planes, sometimes you can run two about planes, sometimes you can run two about planes, sometimes you can run two instances of a service to like offload instances of a service to like offload instances of a service to like offload capacity in a way you just can't do with capacity in a way you just can't do with capacity in a way you just can't do with physical systems. Same thing with physical systems. Same thing with physical systems. Same thing with simulating failures, too. Again, I think simulating failures, too. Again, I think simulating failures, too. Again, I think that's a very responsible thing to do. that's a very responsible thing to do. that's a very responsible thing to do. Um Um Um a lot of complex systems wish they a lot of complex systems wish they a lot of complex systems wish they could. For instance, you can actually could. For instance, you can actually could. For instance, you can actually instantiate, [clears throat] you know, a instantiate, [clears throat] you know, a instantiate, [clears throat] you know, a real kind of clone of the production real kind of clone of the production real kind of clone of the production system. You can't replicate a realistic system. You can't replicate a realistic system. You can't replicate a realistic clone of New York City to see, you know, clone of New York City to see, you know, clone of New York City to see, you know, if there's a certain level of trash if there's a certain level of trash if there's a certain level of trash blocking sewer drains, like what level blocking sewer drains, like what level blocking sewer drains, like what level of flooding will cause like deaths. Like of flooding will cause like deaths. Like of flooding will cause like deaths. Like you can't simulate that with any degree you can't simulate that with any degree you can't simulate that with any degree of ethics, but you can in the computer of ethics, but you can in the computer of ethics, but you can in the computer world around doing it. So I think that's world around doing it. So I think that's world around doing it. So I think that's that's part of my call to action that that's part of my call to action that that's part of my call to action that was big in the book is like okay, how do was big in the book is like okay, how do was big in the book is like okay, how do we start taking this more serious we start taking this more serious we start taking this more serious seriously and really leveraging the seriously and really leveraging the seriously and really leveraging the benefits that the flexibility software benefits that the flexibility software benefits that the flexibility software begets begets begets gives us. So I do think there's

gives us. So I do think there's gives us. So I do think there's to your point, yes, some of it is more to your point, yes, some of it is more to your point, yes, some of it is more expensive to do, but in another sense, expensive to do, but in another sense, expensive to do, but in another sense, you know, maybe we should be allocating you know, maybe we should be allocating you know, maybe we should be allocating more spend towards some of that more spend towards some of that more spend towards some of that simulation or just understanding like simulation or just understanding like simulation or just understanding like the resilience contours of our systems the resilience contours of our systems the resilience contours of our systems better when other industries are just better when other industries are just better when other industries are just looking at us shaking us like why aren't looking at us shaking us like why aren't looking at us shaking us like why aren't you doing this? We wish we could do you doing this? We wish we could do you doing this? We wish we could do this. Yeah. I I've been I've been this. Yeah. I I've been I've been this. Yeah. I I've been I've been thinking about resilience in my own kind thinking about resilience in my own kind thinking about resilience in my own kind of personal IT life. I assume you have a of personal IT life. I assume you have a of personal IT life. I assume you have a home lab and you know, various sorts. home lab and you know, various sorts. home lab and you know, various sorts. Right now as I talk to you because I had Right now as I talk to you because I had Right now as I talk to you because I had an appointment with you, I am on my an appointment with you, I am on my an appointment with you, I am on my backup internet. It turns out I'm backup internet. It turns out I'm backup internet. It turns out I'm looking at my my UniFi here. My my WAN looking at my my UniFi here. My my WAN looking at my my UniFi here. My my WAN failed over at um at 4:38 a.m. and I failed over at um at 4:38 a.m. and I failed over at um at 4:38 a.m. and I have yet to diagnose it. So I'm on have yet to diagnose it. So I'm on have yet to diagnose it. So I'm on backup internet right now. backup internet right now. backup internet right now. And when I mention that to people like And when I mention that to people like And when I mention that to people like like muggles, like regular people, like muggles, like regular people, like muggles, like regular people, they're like you have two internet at they're like you have two internet at they're like you have two internet at your house? I'm like like this is my your house? I'm like like this is my your house? I'm like like this is my job, bro. Like I'm here. Like I got I've job, bro. Like I'm here. Like I got I've job, bro. Like I'm here. Like I got I've been doing this at this house for 18 been doing this at this house for 18 been doing this at this house for 18 years. It cost me 45 bucks for Comcast years. It cost me 45 bucks for Comcast years. It cost me 45 bucks for Comcast as backup internet. I have my fiber as backup internet. I have my fiber as backup internet. I have my fiber with the backup for $45. It only has to with the backup for $45. It only has to with the backup for $45. It only has to fail once like today fail once like today fail once like today and that made it worth the money for the and that made it worth the money for the and that made it worth the money for the year because otherwise I would have had year because otherwise I would have had year because otherwise I would have had to cancel on you.

to cancel on you. to cancel on you. And that wouldn't happen. And that wouldn't happen. And that wouldn't happen. >> Exactly. Yeah, so it's like it was a >> Exactly. Yeah, so it's like it was a >> Exactly. Yeah, so it's like it was a choice and I feel like there are choice and I feel like there are choice and I feel like there are [clears throat] teams that think that [clears throat] teams that think that [clears throat] teams that think that they are resilient until the thing they are resilient until the thing they are resilient until the thing happens. Um what's the difference happens. Um what's the difference happens. Um what's the difference between a security team that thinks between a security team that thinks between a security team that thinks they're resilient and maybe one that they're resilient and maybe one that they're resilient and maybe one that actually is? Like I feel like there's a actually is? Like I feel like there's a actually is? Like I feel like there's a lot of false confidence and metrics lot of false confidence and metrics lot of false confidence and metrics theater that happens. theater that happens. theater that happens. Metrics theater, that could be an Metrics theater, that could be an Metrics theater, that could be an episode in itself. The like oh, what episode in itself. The like oh, what episode in itself. The like oh, what percentage security coverage do we have? percentage security coverage do we have? percentage security coverage do we have? Nobody knows what that means. It's a Nobody knows what that means. It's a Nobody knows what that means. It's a meaningless metric. That is a great meaningless metric. That is a great meaningless metric. That is a great question. I think the giveaway is when question. I think the giveaway is when question. I think the giveaway is when the security team feels a sense of the security team feels a sense of the security team feels a sense of control, probably means that they don't control, probably means that they don't control, probably means that they don't have a lot of resilience. Cuz part of have a lot of resilience. Cuz part of have a lot of resilience. Cuz part of resilience is embracing the fact that resilience is embracing the fact that resilience is embracing the fact that like there will be things well outside like there will be things well outside like there will be things well outside of your control, so it's like how do you of your control, so it's like how do you of your control, so it's like how do you prepare for that? If you were trying to prepare for that? If you were trying to prepare for that? If you were trying to control everything and make things as control everything and make things as control everything and make things as deterministic as possible, you've deterministic as possible, you've deterministic as possible, you've already failed in my view cuz the world already failed in my view cuz the world already failed in my view cuz the world is not deterministic. Humans aren't is not deterministic. Humans aren't is not deterministic. Humans aren't deterministic. We would like computers deterministic. We would like computers deterministic. We would like computers to be deterministic, but they aren't to be deterministic, but they aren't to be deterministic, but they aren't fully at least. It's one of the hardest fully at least. It's one of the hardest fully at least. It's one of the hardest problems in computer science is problems in computer science is problems in computer science is verifying that the software works the verifying that the software works the verifying that the software works the way that the designer of the program way that the designer of the program way that the designer of the program intended it to. So whenever I hear a intended it to. So whenever I hear a intended it to. So whenever I hear a security team say like well, we have security team say like well, we have security team say like well, we have full control over the software delivery full control over the software delivery full control over the software delivery life cycle. Like life cycle. Like life cycle. Like mm mm mm are you sure?

are you sure? are you sure? >> [laughter] >> [laughter] >> [laughter] >> You know? >> You know? >> You know? >> saw a meme I just imagined a meme >> saw a meme I just imagined a meme >> saw a meme I just imagined a meme version of you and that one guy from HBO version of you and that one guy from HBO version of you and that one guy from HBO is like you sure about that? is like you sure about that? is like you sure about that? You sure about that? You sure about that? You sure about that? >> Yep. Yeah, are you sure that's the right >> Yep. Yeah, are you sure that's the right >> Yep. Yeah, are you sure that's the right thing to do? Yeah, you're exactly right. thing to do? Yeah, you're exactly right. thing to do? Yeah, you're exactly right. It probably means you're investing in It probably means you're investing in It probably means you're investing in things that make you feel good and give things that make you feel good and give things that make you feel good and give you that sense of control and not the you that sense of control and not the you that sense of control and not the things that minimize impact. things that minimize impact. things that minimize impact. Yep. Like and that's not directly Yep. Like and that's not directly Yep. Like and that's not directly security, but that old joke of like security, but that old joke of like security, but that old joke of like backups always succeed, it's restores backups always succeed, it's restores backups always succeed, it's restores that fail. that fail. that fail. Mm, yes. So it makes me think about like Mm, yes. So it makes me think about like Mm, yes. So it makes me think about like chaos engineering in in infrastructure chaos engineering in in infrastructure chaos engineering in in infrastructure is about, you know, pulling wires and is about, you know, pulling wires and is about, you know, pulling wires and yanking wires is very exciting, but I yanking wires is very exciting, but I yanking wires is very exciting, but I feel like there's a lot of pull the wire feel like there's a lot of pull the wire feel like there's a lot of pull the wire moments that are that can happen in moments that are that can happen in moments that are that can happen in security, but people are too scared to security, but people are too scared to security, but people are too scared to try. try. try. Fear? I mean, fear is pervasive in the Fear? I mean, fear is pervasive in the Fear? I mean, fear is pervasive in the culture and it's a disservice to the culture and it's a disservice to the culture and it's a disservice to the industry and the mission for sure. I industry and the mission for sure. I industry and the mission for sure. I think there are also cases where think there are also cases where think there are also cases where you know, you could be starting with you know, you could be starting with you know, you could be starting with smaller experiments or just testing more smaller experiments or just testing more smaller experiments or just testing more basic hypotheses. My favorite leveraging basic hypotheses. My favorite leveraging basic hypotheses. My favorite leveraging actually fastly compute which is kind of actually fastly compute which is kind of actually fastly compute which is kind of like a high performance serverless you like a high performance serverless you like a high performance serverless you can think of it that way. It's just a can think of it that way. It's just a can think of it that way. It's just a little function that strips out um little function that strips out um little function that strips out um cookies just to see like hey, does your cookies just to see like hey, does your cookies just to see like hey, does your login site work? Same with like off login site work? Same with like off login site work? Same with like off headers. It's just those basic headers. It's just those basic headers. It's just those basic [clears throat] assumptions you hold [clears throat] assumptions you hold [clears throat] assumptions you hold like of course like we're always going like of course like we're always going like of course like we're always going to require this for the login page. It's to require this for the login page. It's to require this for the login page. It's like well, is that true? Are you sure?

like well, is that true? Are you sure? like well, is that true? Are you sure? And the And the And the especially when you can like duplicate especially when you can like duplicate especially when you can like duplicate the requests um which this prototype the requests um which this prototype the requests um which this prototype did, you know, it's pretty low impact to did, you know, it's pretty low impact to did, you know, it's pretty low impact to the business to run that experiment. the business to run that experiment. the business to run that experiment. There are of course things where it's There are of course things where it's There are of course things where it's like hey, RMRF like the customer like hey, RMRF like the customer like hey, RMRF like the customer database. Yeah, that's going to be a database. Yeah, that's going to be a database. Yeah, that's going to be a pretty poorly designed experiment with pretty poorly designed experiment with pretty poorly designed experiment with high consequences, but there's like such high consequences, but there's like such high consequences, but there's like such a range in between a range in between a range in between that I think it's very unfortunate that that I think it's very unfortunate that that I think it's very unfortunate that security practitioners are security practitioners are security practitioners are too hesitant to try those experiments too hesitant to try those experiments too hesitant to try those experiments especially they're very hesitant to especially they're very hesitant to especially they're very hesitant to reach out to their peers across the reach out to their peers across the reach out to their peers across the island like platform engineering and be island like platform engineering and be island like platform engineering and be like hey, can we co-conspire on like hey, can we co-conspire on like hey, can we co-conspire on developing some of these experiments cuz developing some of these experiments cuz developing some of these experiments cuz there are a lot of jointly held there are a lot of jointly held there are a lot of jointly held assumptions too that aren't always poked assumptions too that aren't always poked assumptions too that aren't always poked and prodded. and prodded. and prodded. You mentioned about determinism and how You mentioned about determinism and how You mentioned about determinism and how computers and software is not as computers and software is not as computers and software is not as deterministic as we would love to think deterministic as we would love to think deterministic as we would love to think that it is. Not just because, you know, that it is. Not just because, you know, that it is. Not just because, you know, the software pretty much always runs as the software pretty much always runs as the software pretty much always runs as you wrote it, but whether or not your you wrote it, but whether or not your you wrote it, but whether or not your intent was well expressed. That intent was well expressed. That intent was well expressed. That certainly is a problem and then the certainly is a problem and then the certainly is a problem and then the environment within which it runs you environment within which it runs you environment within which it runs you can't always count on. But I'm finding can't always count on. But I'm finding can't always count on. But I'm finding that people seem to be spackling or that people seem to be spackling or that people seem to be spackling or puttying over their systems now with puttying over their systems now with puttying over their systems now with what I'm calling ambiguity loops which what I'm calling ambiguity loops which what I'm calling ambiguity loops which are basically using an LLM to deal with are basically using an LLM to deal with are basically using an LLM to deal with ambiguity by letting it fill the ambiguity by letting it fill the ambiguity by letting it fill the ambiguity with randomness. And I'm ambiguity with randomness. And I'm ambiguity with randomness. And I'm curious in your business when now people curious in your business when now people curious in your business when now people are like running playbooks that aren't are like running playbooks that aren't are like running playbooks that aren't scripts, they are prose.

scripts, they are prose. scripts, they are prose. Like a markdown file is not a script I Like a markdown file is not a script I Like a markdown file is not a script I think you would agree. Yeah. think you would agree. Yeah. think you would agree. Yeah. >> Uh how do you feel about that? Is there >> Uh how do you feel about that? Is there >> Uh how do you feel about that? Is there a place for LLMs to live in security and a place for LLMs to live in security and a place for LLMs to live in security and in resilience and in chaos or do they in resilience and in chaos or do they in resilience and in chaos or do they just increase just increase just increase increase chaos and and entropy? increase chaos and and entropy? increase chaos and and entropy? It depends. I think It depends. I think It depends. I think you know, you know, you know, they're only going to be as good as the they're only going to be as good as the they're only going to be as good as the corpus that went into them for one and corpus that went into them for one and corpus that went into them for one and so unless you can verify like really so unless you can verify like really so unless you can verify like really clean code went into it. It's like well, clean code went into it. It's like well, clean code went into it. It's like well, you know, it can maybe be a good basis you know, it can maybe be a good basis you know, it can maybe be a good basis for actually in some cases chaos for actually in some cases chaos for actually in some cases chaos experiments or you know, specific experiments or you know, specific experiments or you know, specific configurations, integration tests, etc. configurations, integration tests, etc. configurations, integration tests, etc. What I will say though What I will say though What I will say though is to me the more important litmus test is to me the more important litmus test is to me the more important litmus test is is this replacing human judgment? And is is this replacing human judgment? And is is this replacing human judgment? And if it is, that's probably not a good if it is, that's probably not a good if it is, that's probably not a good case for an LLM. I am pro human judgment case for an LLM. I am pro human judgment case for an LLM. I am pro human judgment and creativity. I am pretty anti like and creativity. I am pretty anti like and creativity. I am pretty anti like rep like very repetitive work, very rep like very repetitive work, very rep like very repetitive work, very tedious work where you don't need that tedious work where you don't need that tedious work where you don't need that kind of judgment call. LLMs can be very kind of judgment call. LLMs can be very kind of judgment call. LLMs can be very helpful there. I think you know, there helpful there. I think you know, there helpful there. I think you know, there there are also document intelligence there are also document intelligence there are also document intelligence examples, you know, who loves going examples, you know, who loves going examples, you know, who loves going through your compliance documents and through your compliance documents and through your compliance documents and pulling out relevant information like pulling out relevant information like pulling out relevant information like LLMs can shine there and that way you LLMs can shine there and that way you LLMs can shine there and that way you can focus more on strategically like can focus more on strategically like can focus more on strategically like are we sustaining resilience? Like what are we sustaining resilience? Like what are we sustaining resilience? Like what are the indicators we should be looking are the indicators we should be looking are the indicators we should be looking at for that? But asking LLM like is our at for that? But asking LLM like is our at for that? But asking LLM like is our system resilient probably is not going system resilient probably is not going system resilient probably is not going to be a great outcome. I think the to be a great outcome. I think the to be a great outcome. I think the markdown example is a little interesting markdown example is a little interesting markdown example is a little interesting cuz I am also pro cuz I am also pro cuz I am also pro making security more accessible. I think

making security more accessible. I think making security more accessible. I think we dress it up in a lot of arcane we dress it up in a lot of arcane we dress it up in a lot of arcane you know, key phrases and buzzwords um you know, key phrases and buzzwords um you know, key phrases and buzzwords um when really a lot of people could when really a lot of people could when really a lot of people could benefit the industry and contribute. So benefit the industry and contribute. So benefit the industry and contribute. So maybe simplifying how they can enter not maybe simplifying how they can enter not maybe simplifying how they can enter not requiring scripting knowledge could be a requiring scripting knowledge could be a requiring scripting knowledge could be a good thing. I also see it as potential good thing. I also see it as potential good thing. I also see it as potential foot gun. So I feel like I'm a little foot gun. So I feel like I'm a little foot gun. So I feel like I'm a little mixed. So now I hear you. I I'm with you mixed. So now I hear you. I I'm with you mixed. So now I hear you. I I'm with you 100% on the like human judgment like it 100% on the like human judgment like it 100% on the like human judgment like it cannot be overstated. I'm I assume that cannot be overstated. I'm I assume that cannot be overstated. I'm I assume that you're speaking to universities and you're speaking to universities and you're speaking to universities and early in career people often and you early in career people often and you early in career people often and you give your speeches and stuff and you give your speeches and stuff and you give your speeches and stuff and you talk to people like what should I learn? talk to people like what should I learn? talk to people like what should I learn? It's like you should learn how to have It's like you should learn how to have It's like you should learn how to have good taste. Yes. How do you learn good good taste. Yes. How do you learn good good taste. Yes. How do you learn good taste? Well, you just got to get in taste? Well, you just got to get in taste? Well, you just got to get in there and get your hands dirty and do there and get your hands dirty and do there and get your hands dirty and do the thing and start pulling wires and the thing and start pulling wires and the thing and start pulling wires and figure out the system. Certainly I don't figure out the system. Certainly I don't figure out the system. Certainly I don't I don't want to outsource things to to I don't want to outsource things to to I don't want to outsource things to to human judgment and toil human judgment and toil human judgment and toil like keeping a site up is toil. SRE is like keeping a site up is toil. SRE is like keeping a site up is toil. SRE is toil, but SREs that are really good at toil, but SREs that are really good at toil, but SREs that are really good at their job are good at their job because their job are good at their job because their job are good at their job because of their judgment. of their judgment. of their judgment. So there's going to be this constant So there's going to be this constant So there's going to be this constant tension between like I don't think the tension between like I don't think the tension between like I don't think the idea that you would replace an SRE with idea that you would replace an SRE with idea that you would replace an SRE with a markdown file makes me very nervous, a markdown file makes me very nervous, a markdown file makes me very nervous, but an SRE agent that could maybe kick but an SRE agent that could maybe kick but an SRE agent that could maybe kick the node and keep it running while I the node and keep it running while I the node and keep it running while I drive over there drive over there drive over there has value to me.

has value to me. has value to me. Yes, buying capacity and buying time I Yes, buying capacity and buying time I Yes, buying capacity and buying time I think is a great use case as well to think is a great use case as well to think is a great use case as well to your point. Just like how do we help the your point. Just like how do we help the your point. Just like how do we help the human engage better with the system? human engage better with the system? human engage better with the system? Mhm. Mhm. Mhm. Even just like the rubber duck problem Even just like the rubber duck problem Even just like the rubber duck problem solving, that can be a useful thing for solving, that can be a useful thing for solving, that can be a useful thing for the LLM. That does require to your point the LLM. That does require to your point the LLM. That does require to your point quite a bit of expertise already though. quite a bit of expertise already though. quite a bit of expertise already though. I love that you brought that up. I I was I love that you brought that up. I I was I love that you brought that up. I I was talking to someone recently and I gave a talking to someone recently and I gave a talking to someone recently and I gave a whole talk and I start brought up rubber whole talk and I start brought up rubber whole talk and I start brought up rubber duck debugging and like no one got it duck debugging and like no one got it duck debugging and like no one got it and like am I like am I unk suddenly? and like am I like am I unk suddenly? and like am I like am I unk suddenly? Like no one gives. This is not a Like no one gives. This is not a Like no one gives. This is not a generational thing like talking to the generational thing like talking to the generational thing like talking to the duck. duck. duck. Yeah, your face is saying the same Yeah, your face is saying the same Yeah, your face is saying the same thing. Like like those are important thing. Like like those are important thing. Like like those are important moments to moments to moments to brain you're talking to yourself in the brain you're talking to yourself in the brain you're talking to yourself in the mirror except now the mirror can talk mirror except now the mirror can talk mirror except now the mirror can talk back and that's really cool and I find back and that's really cool and I find back and that's really cool and I find that to be super helpful. Have you used that to be super helpful. Have you used that to be super helpful. Have you used LLMs in that context to like LLMs in that context to like LLMs in that context to like figure your thoughts out and talk to figure your thoughts out and talk to figure your thoughts out and talk to yourself? looks that make you really question, you looks that make you really question, you know, what you're throwing down. I think know, what you're throwing down. I think know, what you're throwing down. I think LLMs can be helpful though especially I LLMs can be helpful though especially I LLMs can be helpful though especially I see a lot of people struggle to get see a lot of people struggle to get see a lot of people struggle to get buy-in. This is kind of getting into, buy-in. This is kind of getting into, buy-in. This is kind of getting into, you know, corporate type stuff, but you know, corporate type stuff, but you know, corporate type stuff, but especially international companies where especially international companies where especially international companies where it's like hey, I want to get buy-in on it's like hey, I want to get buy-in on it's like hey, I want to get buy-in on this resilience initiative. How is this this resilience initiative. How is this this resilience initiative. How is this going to resonate across different going to resonate across different going to resonate across different cultural contexts? Ooh, that's a good cultural contexts? Ooh, that's a good cultural contexts? Ooh, that's a good one. Right? Like is chaos perceived one. Right? Like is chaos perceived one. Right? Like is chaos perceived negatively in certain nations versus negatively in certain nations versus negatively in certain nations versus others? I can tell you for instance when others? I can tell you for instance when others? I can tell you for instance when I talk about deception and using that as I talk about deception and using that as I talk about deception and using that as a technique for resilience engineering a technique for resilience engineering a technique for resilience engineering security engineering security engineering security engineering American security practitioners not American security practitioners not American security practitioners not universally tend to result in like well,

universally tend to result in like well, universally tend to result in like well, we don't want to be the bad guys. we don't want to be the bad guys. we don't want to be the bad guys. Hm. Now in the EU, they're like Hm. Now in the EU, they're like Hm. Now in the EU, they're like tell me more. Yes, please. Like we want tell me more. Yes, please. Like we want tell me more. Yes, please. Like we want the Sutter fudge here. So that's kind of the Sutter fudge here. So that's kind of the Sutter fudge here. So that's kind of fascinating, right? And I feel like fascinating, right? And I feel like fascinating, right? And I feel like that's an interesting that's an interesting that's an interesting like twist like twist like twist um that I found really useful with LLMs. um that I found really useful with LLMs. um that I found really useful with LLMs. I like that. That that I I didn't think I like that. That that I I didn't think I like that. That that I I didn't think about that. Yeah, you're right. It does about that. Yeah, you're right. It does about that. Yeah, you're right. It does seem broader than than we do and it can seem broader than than we do and it can seem broader than than we do and it can challenge your assumptions especially if challenge your assumptions especially if challenge your assumptions especially if you tell it challenge my assumptions as you tell it challenge my assumptions as you tell it challenge my assumptions as opposed to telling you that you're opposed to telling you that you're opposed to telling you that you're absolutely right. absolutely right. absolutely right. There's I you you probably work with There's I you you probably work with There's I you you probably work with like big companies like your banks and like big companies like your banks and like big companies like your banks and slower moving things health care. slower moving things health care. slower moving things health care. They're a little more conservative. I'm They're a little more conservative. I'm They're a little more conservative. I'm curious does is there an example where curious does is there an example where curious does is there an example where traditional compliance actively makes traditional compliance actively makes traditional compliance actively makes systems less secure where they think systems less secure where they think systems less secure where they think that they're checking boxes, but they're that they're checking boxes, but they're that they're checking boxes, but they're actually hurting themselves? actually hurting themselves? actually hurting themselves? Yes, actually frequent Yes, actually frequent Yes, actually frequent co-conspirator of mine Josiah co-conspirator of mine Josiah co-conspirator of mine Josiah Dykstra wrote a paper not with me. It's Dykstra wrote a paper not with me. It's Dykstra wrote a paper not with me. It's an excellent paper about that exact an excellent paper about that exact an excellent paper about that exact topic. I think specifically topic. I think specifically topic. I think specifically covers HIPAA and maybe one of the others covers HIPAA and maybe one of the others covers HIPAA and maybe one of the others that shows that it doesn't being more that shows that it doesn't being more that shows that it doesn't being more compliant doesn't actually result in compliant doesn't actually result in compliant doesn't actually result in better security outcomes.

better security outcomes. better security outcomes. I'm very much of the view and have tried I'm very much of the view and have tried I'm very much of the view and have tried to caution regulators as well as like to caution regulators as well as like to caution regulators as well as like well-intentioned regulation in this well-intentioned regulation in this well-intentioned regulation in this space very quickly calcifies and space very quickly calcifies and space very quickly calcifies and ossifies like it's ossifies like it's ossifies like it's what helped in year zero through maybe what helped in year zero through maybe what helped in year zero through maybe even even even year three may end up actually eroding year three may end up actually eroding year three may end up actually eroding resilience long term. Great example, resilience long term. Great example, resilience long term. Great example, I'll keep the person anonymous, very I'll keep the person anonymous, very I'll keep the person anonymous, very innovative CISO um innovative CISO um innovative CISO um had to explain I think over a few years had to explain I think over a few years had to explain I think over a few years to his auditors like actually it's a to his auditors like actually it's a to his auditors like actually it's a great thing that we don't allow SSH great thing that we don't allow SSH great thing that we don't allow SSH access anymore cuz that's what attackers access anymore cuz that's what attackers access anymore cuz that's what attackers love. They love when you leave the door love. They love when you leave the door love. They love when you leave the door open like that. But on the little open like that. But on the little open like that. But on the little compliance checklist for the auditors, compliance checklist for the auditors, compliance checklist for the auditors, they're like okay, but it says you're they're like okay, but it says you're they're like okay, but it says you're required to have SSH access. He's like required to have SSH access. He's like required to have SSH access. He's like okay, but the security outcome is now okay, but the security outcome is now okay, but the security outcome is now better. So that's where sometimes it can better. So that's where sometimes it can better. So that's where sometimes it can actually hold companies back Mhm. actually hold companies back Mhm. actually hold companies back Mhm. big companies who do want to innovate um big companies who do want to innovate um big companies who do want to innovate um but basically tying them to their but basically tying them to their but basically tying them to their investments and their spend to just investments and their spend to just investments and their spend to just checking those boxes which is a checking those boxes which is a checking those boxes which is a disservice to their overall mission.

disservice to their overall mission. disservice to their overall mission. Yeah, having I always want to assert Yeah, having I always want to assert Yeah, having I always want to assert assumptions. I feel like cuz I work at assumptions. I feel like cuz I work at assumptions. I feel like cuz I work at Microsoft in my day job that my Microsoft in my day job that my Microsoft in my day job that my ignorance is kind of my superpower cuz ignorance is kind of my superpower cuz ignorance is kind of my superpower cuz someone will throw me under a new someone will throw me under a new someone will throw me under a new situation and I'll see a check box like situation and I'll see a check box like situation and I'll see a check box like must include SSH. I'm like but why? Like must include SSH. I'm like but why? Like must include SSH. I'm like but why? Like and they don't no one knows. Like I and they don't no one knows. Like I and they don't no one knows. Like I don't know, 13 years ago someone wrote don't know, 13 years ago someone wrote don't know, 13 years ago someone wrote that checklist and now it's a thing. that checklist and now it's a thing. that checklist and now it's a thing. And then investors see it and compliance And then investors see it and compliance And then investors see it and compliance people see it and that check box is the people see it and that check box is the people see it and that check box is the thing that stands between you and some thing that stands between you and some thing that stands between you and some certificate or some badge certificate or some badge certificate or some badge and that's a problem. and that's a problem. and that's a problem. It's a huge problem. You actually bring It's a huge problem. You actually bring It's a huge problem. You actually bring up a kind of elegant point. If you look up a kind of elegant point. If you look up a kind of elegant point. If you look at what resilience means across all at what resilience means across all at what resilience means across all sorts of complex systems, but also the sorts of complex systems, but also the sorts of complex systems, but also the ones we're talking about here ones we're talking about here ones we're talking about here a lot of when a system is stuck in let's a lot of when a system is stuck in let's a lot of when a system is stuck in let's say it's not elegant like a a less say it's not elegant like a a less say it's not elegant like a a less resilient or like unresilient state or resilient or like unresilient state or resilient or like unresilient state or fragile state is because a lot of the fragile state is because a lot of the fragile state is because a lot of the processes and practices that they have processes and practices that they have processes and practices that they have in place to your point in place to your point in place to your point are from an equilibrium that no longer are from an equilibrium that no longer are from an equilibrium that no longer exists. Right? The status quo has moved exists. Right? The status quo has moved exists. Right? The status quo has moved on. The practices haven't. on. The practices haven't. on. The practices haven't. And so you're just continuing to erode And so you're just continuing to erode And so you're just continuing to erode resilience as you like stick to this old resilience as you like stick to this old resilience as you like stick to this old world and have not adapted to the new world and have not adapted to the new world and have not adapted to the new one and the new context. The same with I one and the new context. The same with I one and the new context. The same with I always hear CISO's being like well, once always hear CISO's being like well, once always hear CISO's being like well, once we patch the vulnerability or fix it, we patch the vulnerability or fix it, we patch the vulnerability or fix it, you know, then like it's fine. It's like you know, then like it's fine. It's like you know, then like it's fine. It's like well, if that actually resulted in an well, if that actually resulted in an well, if that actually resulted in an outage or a breach, it's not actually outage or a breach, it's not actually outage or a breach, it's not actually fine because you still haven't addressed fine because you still haven't addressed fine because you still haven't addressed the underlying impact. You've just the underlying impact. You've just the underlying impact. You've just patched over the one way attackers got patched over the one way attackers got patched over the one way attackers got in. Um they haven't adapted to that new in. Um they haven't adapted to that new in. Um they haven't adapted to that new paradigm and that new equilibrium um paradigm and that new equilibrium um paradigm and that new equilibrium um which is hard. It's updating your mental which is hard. It's updating your mental which is hard. It's updating your mental model of the system which is not easy.

model of the system which is not easy. model of the system which is not easy. That's why it is so important to have That's why it is so important to have That's why it is so important to have people who will be like well, why? Why people who will be like well, why? Why people who will be like well, why? Why is that? You know, just poke poke and is that? You know, just poke poke and is that? You know, just poke poke and prod. prod. prod. Um often CISOs and security teams make Um often CISOs and security teams make Um often CISOs and security teams make dashboards cuz you want to roll things dashboards cuz you want to roll things dashboards cuz you want to roll things up and the bigger the company, the up and the bigger the company, the up and the bigger the company, the bigger the dashboard and then the CISO bigger the dashboard and then the CISO bigger the dashboard and then the CISO has to really has to really has to really they don't they can't know the entire they don't they can't know the entire they don't they can't know the entire stack. The stack is now too deep. So, stack. The stack is now too deep. So, stack. The stack is now too deep. So, what is an example of a misleading what is an example of a misleading what is an example of a misleading security metric or dashboard that might security metric or dashboard that might security metric or dashboard that might cause someone to make a mistake? They're cause someone to make a mistake? They're cause someone to make a mistake? They're relying on a dashboard, but it's relying on a dashboard, but it's relying on a dashboard, but it's maybe a misleading metric. maybe a misleading metric. maybe a misleading metric. So many metrics. Certainly that security So many metrics. Certainly that security So many metrics. Certainly that security coverage one or risk coverage. Also the coverage one or risk coverage. Also the coverage one or risk coverage. Also the number of vulnerabilities number of vulnerabilities number of vulnerabilities discovered. Mhm. discovered. Mhm. discovered. Mhm. I'm trying to remember who it was who I'm trying to remember who it was who I'm trying to remember who it was who talked about this where actually when talked about this where actually when talked about this where actually when things started to get better, it meant things started to get better, it meant things started to get better, it meant that their application development teams that their application development teams that their application development teams were surfacing more security issues, were surfacing more security issues, were surfacing more security issues, which was a good thing. And it meant which was a good thing. And it meant which was a good thing. And it meant that there was more of that trust mutual that there was more of that trust mutual that there was more of that trust mutual trust between teams, but it looked like trust between teams, but it looked like trust between teams, but it looked like it was getting worse. Yeah, see that's a it was getting worse. Yeah, see that's a it was getting worse. Yeah, see that's a great example. That's the whole thing great example. That's the whole thing great example. That's the whole thing like, you know, oh my goodness all these like, you know, oh my goodness all these like, you know, oh my goodness all these bugs and all these security issues. This bugs and all these security issues. This bugs and all these security issues. This is good stuff. All of that is is good stuff. All of that is is good stuff. All of that is low-hanging fruit, but like low-hanging fruit, but like low-hanging fruit, but like they'll assume that something bad has they'll assume that something bad has they'll assume that something bad has happened or something has changed and happened or something has changed and happened or something has changed and they're going to then they're going to then they're going to then you know, correlation and causation are you know, correlation and causation are you know, correlation and causation are not not not the same.

not not not the same. not not not the same. Exactly. I think Exactly. I think Exactly. I think and also a lot of the security specific and also a lot of the security specific and also a lot of the security specific metrics don't tell the bigger picture. metrics don't tell the bigger picture. metrics don't tell the bigger picture. Includes I think about, you know, the Includes I think about, you know, the Includes I think about, you know, the poor platform engineering teams who are poor platform engineering teams who are poor platform engineering teams who are handed a list of a thousand handed a list of a thousand handed a list of a thousand vulnerabilities. Turns out a lot of them vulnerabilities. Turns out a lot of them vulnerabilities. Turns out a lot of them are in components that aren't even are in components that aren't even are in components that aren't even exposed to the public internet. Should exposed to the public internet. Should exposed to the public internet. Should they prioritize those? Probably not. And they prioritize those? Probably not. And they prioritize those? Probably not. And meanwhile, in meanwhile, in meanwhile, in actually there are multiple cases of actually there are multiple cases of actually there are multiple cases of this, so I'll keep them all anonymous, this, so I'll keep them all anonymous, this, so I'll keep them all anonymous, but it's praising actually often this but it's praising actually often this but it's praising actually often this happens. Security team will be on them happens. Security team will be on them happens. Security team will be on them and like fix all of these even if and like fix all of these even if and like fix all of these even if they're not publicly exposed. And then they're not publicly exposed. And then they're not publicly exposed. And then the security team actually maintains, the security team actually maintains, the security team actually maintains, you know, their creds into their, you you know, their creds into their, you you know, their creds into their, you know, like know, like know, like whatever admin system or security system whatever admin system or security system whatever admin system or security system that has its hooks into everything is that has its hooks into everything is that has its hooks into everything is like in a text file on their desktop. like in a text file on their desktop. like in a text file on their desktop. It's like, well, what do you think is It's like, well, what do you think is It's like, well, what do you think is actually the bigger issue here in terms actually the bigger issue here in terms actually the bigger issue here in terms of what attackers could leverage? So, of what attackers could leverage? So, of what attackers could leverage? So, there's a lot of that kind of attacker there's a lot of that kind of attacker there's a lot of that kind of attacker math attacker calculus that isn't baked math attacker calculus that isn't baked math attacker calculus that isn't baked in as well. in as well. in as well. And I think And I think And I think there's also even if we think about there's also even if we think about there's also even if we think about business context, business context, business context, the metrics that a lot of security teams the metrics that a lot of security teams the metrics that a lot of security teams track and even CISO's track aren't the track and even CISO's track aren't the track and even CISO's track aren't the ones that the board wants to understand ones that the board wants to understand ones that the board wants to understand Mhm. or other executives need to Mhm. or other executives need to Mhm. or other executives need to understand either. Yeah.

understand either. Yeah. understand either. Yeah. If I go to fastly.com and I click on If I go to fastly.com and I click on If I go to fastly.com and I click on products, you've got all the network products, you've got all the network products, you've got all the network services and all the things that services and all the things that services and all the things that Fastly's known for. There's a whole Fastly's known for. There's a whole Fastly's known for. There's a whole section on security and there's also, section on security and there's also, section on security and there's also, you know, you have services and folks you know, you have services and folks you know, you have services and folks that you can hire professional services that you can hire professional services that you can hire professional services and things like that. But what how and things like that. But what how and things like that. But what how should I think about what security is my should I think about what security is my should I think about what security is my responsibility and what is the responsibility and what is the responsibility and what is the responsibility of the vendor for whom I responsibility of the vendor for whom I responsibility of the vendor for whom I am paying a lot of money to make things am paying a lot of money to make things am paying a lot of money to make things secure? secure? secure? I think it depends on the vendor. I I think it depends on the vendor. I I think it depends on the vendor. I think in the case of let's say like think in the case of let's say like think in the case of let's say like some sort of SaaS application, um some sort of SaaS application, um some sort of SaaS application, um let's take it sales and marketing, so let's take it sales and marketing, so let's take it sales and marketing, so it's going to have some of your customer it's going to have some of your customer it's going to have some of your customer data, prospect data. data, prospect data. data, prospect data. Feels reasonable for the most part that Feels reasonable for the most part that Feels reasonable for the most part that like the encryption and things like that like the encryption and things like that like the encryption and things like that should be handled by the vendor for should be handled by the vendor for should be handled by the vendor for sure. sure. sure. There are cases I'll use like Fastly There are cases I'll use like Fastly There are cases I'll use like Fastly where we have where we have where we have the platform where you can basically the platform where you can basically the platform where you can basically write code, run code, etc. Um write code, run code, etc. Um write code, run code, etc. Um It's our responsibility and we have done It's our responsibility and we have done It's our responsibility and we have done this to layer in like memory safety by this to layer in like memory safety by this to layer in like memory safety by design. design. design. And same with like isolation models And same with like isolation models And same with like isolation models ensuring like safe multi-tenancy. It's ensuring like safe multi-tenancy. It's ensuring like safe multi-tenancy. It's very much our responsibility.

very much our responsibility. very much our responsibility. Making sure that like you don't write Making sure that like you don't write Making sure that like you don't write vulnerabilities into your own code. It's vulnerabilities into your own code. It's vulnerabilities into your own code. It's like, well, that's probably outside of like, well, that's probably outside of like, well, that's probably outside of our remit. Um though that's where our remit. Um though that's where our remit. Um though that's where sometimes pro serve can come in. Um sometimes pro serve can come in. Um sometimes pro serve can come in. Um things like, hey, you have spun up like things like, hey, you have spun up like things like, hey, you have spun up like a service on Fastly and that connects to a service on Fastly and that connects to a service on Fastly and that connects to a database that you have wide open a database that you have wide open a database that you have wide open without any ackle. It's sorry, access without any ackle. It's sorry, access without any ackle. It's sorry, access control list. Not really our control list. Not really our control list. Not really our responsibility cuz we don't touch that responsibility cuz we don't touch that responsibility cuz we don't touch that component, right? There are ways that we component, right? There are ways that we component, right? There are ways that we can help with middleware that runs on can help with middleware that runs on can help with middleware that runs on our platform. our platform. our platform. I do think though there's a fundamental I do think though there's a fundamental I do think though there's a fundamental principle principle principle though in the conversation a lot of though in the conversation a lot of though in the conversation a lot of people miss which is like you have to people miss which is like you have to people miss which is like you have to own your own dependencies. own your own dependencies. own your own dependencies. And so what you adopt, you do have to And so what you adopt, you do have to And so what you adopt, you do have to basically think about it like, well, we basically think about it like, well, we basically think about it like, well, we have to assume at some point something have to assume at some point something have to assume at some point something will go wrong with it whether that's a will go wrong with it whether that's a will go wrong with it whether that's a security issue or not. And I do see a security issue or not. And I do see a security issue or not. And I do see a lot of the like hot potato game happen lot of the like hot potato game happen lot of the like hot potato game happen out there. Yeah. That's exactly why I out there. Yeah. That's exactly why I out there. Yeah. That's exactly why I asked you that question because I think asked you that question because I think asked you that question because I think that that that people pay a lot of money for a platform people pay a lot of money for a platform people pay a lot of money for a platform a cloud platform because they want as a cloud platform because they want as a cloud platform because they want as they say a throat to choke. So, like who they say a throat to choke. So, like who they say a throat to choke. So, like who gets yelled at, right? You know, I do gets yelled at, right? You know, I do gets yelled at, right? You know, I do get Kelly on the phone. I want to know get Kelly on the phone. I want to know get Kelly on the phone. I want to know what's going on over there. You know, what's going on over there. You know, what's going on over there. You know, but then it's of course their thing.

but then it's of course their thing. but then it's of course their thing. Then they are bringing in, you know, Then they are bringing in, you know, Then they are bringing in, you know, unknown unknown unknown who knows unknown node packages from who knows unknown node packages from who knows unknown node packages from unknown provenance and then they haven't unknown provenance and then they haven't unknown provenance and then they haven't think to thought about their entire think to thought about their entire think to thought about their entire secure supply chain. But at the same secure supply chain. But at the same secure supply chain. But at the same time like I I like your point about the time like I I like your point about the time like I I like your point about the sales and marketing CRM or something sales and marketing CRM or something sales and marketing CRM or something like that. Is it their job as an app to like that. Is it their job as an app to like that. Is it their job as an app to be in charge of like AI bot management be in charge of like AI bot management be in charge of like AI bot management or DDoS or even API security? Like that or DDoS or even API security? Like that or DDoS or even API security? Like that would be an example where a cloud would be an example where a cloud would be an example where a cloud platform could secure those endpoints platform could secure those endpoints platform could secure those endpoints and and and and and and hide that. So, I like the separation of hide that. So, I like the separation of hide that. So, I like the separation of concerns concerns concerns there. But I I wonder if people who are there. But I I wonder if people who are there. But I I wonder if people who are putting together their own systems think putting together their own systems think putting together their own systems think about that. Like, we shouldn't be in about that. Like, we shouldn't be in about that. Like, we shouldn't be in charge of API security. Let Fastly charge of API security. Let Fastly charge of API security. Let Fastly secure our endpoints. And then they just secure our endpoints. And then they just secure our endpoints. And then they just have a nice clean bright line or is it have a nice clean bright line or is it have a nice clean bright line or is it always layered and they would have to do always layered and they would have to do always layered and they would have to do that basically you have two layers. that basically you have two layers. that basically you have two layers. I think it it really depends on the I think it it really depends on the I think it it really depends on the company and their level of resourcing. company and their level of resourcing. company and their level of resourcing. There are some companies that culturally There are some companies that culturally There are some companies that culturally want to own want to own want to own where things are built more of their where things are built more of their where things are built more of their things and so they'll leverage us more things and so they'll leverage us more things and so they'll leverage us more to be able to DIY. There's certainly to be able to DIY. There's certainly to be able to DIY. There's certainly others others others where it's like, well, let's just use where it's like, well, let's just use where it's like, well, let's just use what Fastly has, right? Especially when what Fastly has, right? Especially when what Fastly has, right? Especially when it comes to security. Putting in whether it comes to security. Putting in whether it comes to security. Putting in whether it's our laugh or like you said the AI it's our laugh or like you said the AI it's our laugh or like you said the AI bot kind of like insights we're able to bot kind of like insights we're able to bot kind of like insights we're able to surface like have that in front of our surface like have that in front of our surface like have that in front of our services, apps, sites, whatever it is.

services, apps, sites, whatever it is. services, apps, sites, whatever it is. It really depends on resourcing. There's It really depends on resourcing. There's It really depends on resourcing. There's also the element of also the element of also the element of I'm going to say in newer worlds where I I'm going to say in newer worlds where I I'm going to say in newer worlds where I have seen so many security leaders have seen so many security leaders have seen so many security leaders thrust into a conversation now where thrust into a conversation now where thrust into a conversation now where their CEO, their CMO, their board is their CEO, their CMO, their board is their CEO, their CMO, their board is like, hey, what AI bots are actually like, hey, what AI bots are actually like, hey, what AI bots are actually trying to scrape our stuff so we can trying to scrape our stuff so we can trying to scrape our stuff so we can monetize it. The CISO is like, I've monetize it. The CISO is like, I've monetize it. The CISO is like, I've never had to think about this before. never had to think about this before. never had to think about this before. Trying to DIY that is pretty hard and Trying to DIY that is pretty hard and Trying to DIY that is pretty hard and having that expertise is some of the having that expertise is some of the having that expertise is some of the most expensive expertise out there right most expensive expertise out there right most expensive expertise out there right now. Yeah. Using a tool probably makes now. Yeah. Using a tool probably makes now. Yeah. Using a tool probably makes sense in that case. Yeah, I think that's sense in that case. Yeah, I think that's sense in that case. Yeah, I think that's a great point. I mean, this is this is a great point. I mean, this is this is a great point. I mean, this is this is the thing. What do we do here at the the thing. What do we do here at the the thing. What do we do here at the company? We do insurance. Okay, then why company? We do insurance. Okay, then why company? We do insurance. Okay, then why are we doing AI bot management? Right. are we doing AI bot management? Right. are we doing AI bot management? Right. >> That's not that's not our job, you know. >> That's not that's not our job, you know. >> That's not that's not our job, you know. Exactly. Exactly. Exactly. >> I always think about the business and I >> I always think about the business and I >> I always think about the business and I I think sometimes when we are talking I think sometimes when we are talking I think sometimes when we are talking about all the things that we've been about all the things that we've been about all the things that we've been talking about on the show, we don't talk talking about on the show, we don't talk talking about on the show, we don't talk about like, why did we actually make about like, why did we actually make about like, why did we actually make this software? We made it to solve X this software? We made it to solve X this software? We made it to solve X business problem. Therefore, what business problem. Therefore, what business problem. Therefore, what responsibility is mine and what can be responsibility is mine and what can be responsibility is mine and what can be outsourced by someone who actually knows outsourced by someone who actually knows outsourced by someone who actually knows like the what they're doing whether it like the what they're doing whether it like the what they're doing whether it be Fastly or Azure or AWS. Let somebody be Fastly or Azure or AWS. Let somebody be Fastly or Azure or AWS. Let somebody who actually cares who actually cares who actually cares about that do that while I focus on the about that do that while I focus on the about that do that while I focus on the business problem cuz I don't want to do business problem cuz I don't want to do business problem cuz I don't want to do Honestly, I don't want to do the stuff Honestly, I don't want to do the stuff Honestly, I don't want to do the stuff Fastly does. That's why Fastly's good at Fastly does. That's why Fastly's good at Fastly does. That's why Fastly's good at it, you know what I mean? Yes, and it's it, you know what I mean? Yes, and it's it, you know what I mean? Yes, and it's laying out your own pop infrastructure laying out your own pop infrastructure laying out your own pop infrastructure especially in this day and age of RAM especially in this day and age of RAM especially in this day and age of RAM prices being what they are, especially prices being what they are, especially prices being what they are, especially if you were a small business, it's quite if you were a small business, it's quite if you were a small business, it's quite unlikely that you're going to be able to unlikely that you're going to be able to unlikely that you're going to be able to do that. Yeah. Nor should you cuz to do that. Yeah. Nor should you cuz to do that. Yeah. Nor should you cuz to your point it's not your core business.

your point it's not your core business. your point it's not your core business. I think it's I think it's I think it's I was thinking when you were talking I was thinking when you were talking I was thinking when you were talking about the example of the duplicate or about the example of the duplicate or about the example of the duplicate or secondary internet. secondary internet. secondary internet. Part of that is because your essentially Part of that is because your essentially Part of that is because your essentially critical function hosting this podcast critical function hosting this podcast critical function hosting this podcast is where you can record it. However, if is where you can record it. However, if is where you can record it. However, if you were to build your own microphone, you were to build your own microphone, you were to build your own microphone, I'd be a little bit like, I'd be a little bit like, I'd be a little bit like, is that actually is that actually is that actually like your core value add here, you know? like your core value add here, you know? like your core value add here, you know? No, that's a good example. Right? It's No, that's a good example. Right? It's No, that's a good example. Right? It's just understanding like what matters and just understanding like what matters and just understanding like what matters and what makes you unique as a business for what makes you unique as a business for what makes you unique as a business for sure. Yeah, absolutely. sure. Yeah, absolutely. sure. Yeah, absolutely. This is totally random and off-topic, This is totally random and off-topic, This is totally random and off-topic, but I we had a fishing thing happen at but I we had a fishing thing happen at but I we had a fishing thing happen at work yesterday where they they send us work yesterday where they they send us work yesterday where they they send us fishing emails, but it's from the red fishing emails, but it's from the red fishing emails, but it's from the red team. team. team. And I was so proud of myself. I was just And I was so proud of myself. I was just And I was so proud of myself. I was just like, like, like, I don't think that's real and I was I don't think that's real and I was I don't think that's real and I was like, report fishing and then it was like, report fishing and then it was like, report fishing and then it was like, congratulations. like, congratulations. like, congratulations. You are, you know, you're you're one of You are, you know, you're you're one of You are, you know, you're you're one of the better people. I don't know. There's the better people. I don't know. There's the better people. I don't know. There's some number of people at the company some number of people at the company some number of people at the company that did that does that. And I'm always that did that does that. And I'm always that did that does that. And I'm always impressed that there's whole teams out impressed that there's whole teams out impressed that there's whole teams out there trying to attack us internally there trying to attack us internally there trying to attack us internally that I've never even met. You know what that I've never even met. You know what that I've never even met. You know what I mean? The the red hats or the I guess I mean? The the red hats or the I guess I mean? The the red hats or the I guess they call them blue hats at at they call them blue hats at at they call them blue hats at at at Microsoft cuz our badges are blue.

at Microsoft cuz our badges are blue. at Microsoft cuz our badges are blue. So, I was just somehow I was just So, I was just somehow I was just So, I was just somehow I was just thinking about there's people in trying thinking about there's people in trying thinking about there's people in trying to create chaos internally at the to create chaos internally at the to create chaos internally at the company and they tried to catch me company and they tried to catch me company and they tried to catch me yesterday with a fish and I denied them. yesterday with a fish and I denied them. yesterday with a fish and I denied them. >> However, what I will say, that has gone >> However, what I will say, that has gone >> However, what I will say, that has gone wrong in the past and I've spoken wrong in the past and I've spoken wrong in the past and I've spoken publicly before it was cool to have this publicly before it was cool to have this publicly before it was cool to have this take. People were quite angry. I take. People were quite angry. I take. People were quite angry. I remember this is many years ago where I remember this is many years ago where I remember this is many years ago where I said, hey, it's maybe not a great thing said, hey, it's maybe not a great thing said, hey, it's maybe not a great thing especially during COVID this happened a especially during COVID this happened a especially during COVID this happened a lot to be like, here's your surprise lot to be like, here's your surprise lot to be like, here's your surprise bonus plan and that's the fishing bonus plan and that's the fishing bonus plan and that's the fishing like simulation email. Oh, no, that like simulation email. Oh, no, that like simulation email. Oh, no, that would be awful. would be awful. would be awful. Right, but that was happening. Right, but that was happening. Right, but that was happening. >> That's mean. That's just that's kind of >> That's mean. That's just that's kind of >> That's mean. That's just that's kind of punitive. I agree. punitive. I agree. punitive. I agree. >> Like, play here for more money. No, this >> Like, play here for more money. No, this >> Like, play here for more money. No, this was not that. This was more like, you was not that. This was more like, you was not that. This was more like, you have mail waiting for you in the mail have mail waiting for you in the mail have mail waiting for you in the mail room. room. room. And I'm like, we don't have a mail room, And I'm like, we don't have a mail room, And I'm like, we don't have a mail room, you know. There you go. Yeah. you know. There you go. Yeah. you know. There you go. Yeah. That's fair. You're right. I mean, this That's fair. You're right. I mean, this That's fair. You're right. I mean, this is the whole like sprinkling USB keys is the whole like sprinkling USB keys is the whole like sprinkling USB keys around the bank parking lot kind of way around the bank parking lot kind of way around the bank parking lot kind of way of doing things. It's like, Beyonce's of doing things. It's like, Beyonce's of doing things. It's like, Beyonce's new album, sprinkle sprinkle and then new album, sprinkle sprinkle and then new album, sprinkle sprinkle and then everyone plugs it in and then owns the everyone plugs it in and then owns the everyone plugs it in and then owns the entire bank.

entire bank. entire bank. Yeah. Something like that. I think there Yeah. Something like that. I think there Yeah. Something like that. I think there are a lot of experiments. I think it's are a lot of experiments. I think it's are a lot of experiments. I think it's always keeping in mind again the human always keeping in mind again the human always keeping in mind again the human element that you don't want to sow element that you don't want to sow element that you don't want to sow distrust. Um distrust. Um distrust. Um But again, you can also make the But again, you can also make the But again, you can also make the experiments collaborative, experiments collaborative, experiments collaborative, which is which that can get really fun which is which that can get really fun which is which that can get really fun because I've actually you mentioned because I've actually you mentioned because I've actually you mentioned SREs. When I talk to like real SREs. When I talk to like real SREs. When I talk to like real attackers, they're generally not scared attackers, they're generally not scared attackers, they're generally not scared of security engineering teams. They're of security engineering teams. They're of security engineering teams. They're scared of SREs cuz SREs will like obsess scared of SREs cuz SREs will like obsess scared of SREs cuz SREs will like obsess performance. There's that one backdoor, performance. There's that one backdoor, performance. There's that one backdoor, right? Was it X U tills where it was a right? Was it X U tills where it was a right? Was it X U tills where it was a guy who's like, oh, performance degraded guy who's like, oh, performance degraded guy who's like, oh, performance degraded by I think it was less than 1%. What is by I think it was less than 1%. What is by I think it was less than 1%. What is going on? We discovered the back door. going on? We discovered the back door. going on? We discovered the back door. Right. Right. Right. That was awesome actually. That was That was awesome actually. That was That was awesome actually. That was pretty cool. pretty cool. pretty cool. Right? So, I think security teams need Right? So, I think security teams need Right? So, I think security teams need to embrace like, hey, you may you will to embrace like, hey, you may you will to embrace like, hey, you may you will have good ideas, but like there're going have good ideas, but like there're going have good ideas, but like there're going to be other very clever people where if to be other very clever people where if to be other very clever people where if you say, okay, if you got really mad at you say, okay, if you got really mad at you say, okay, if you got really mad at the company, how would you attack us? the company, how would you attack us? the company, how would you attack us? They're probably going to have some They're probably going to have some They're probably going to have some interesting ideas that maybe can become interesting ideas that maybe can become interesting ideas that maybe can become experiments or clue you into some gaps experiments or clue you into some gaps experiments or clue you into some gaps maybe you have in your current security maybe you have in your current security maybe you have in your current security investments.

investments. investments. >> [snorts] >> [snorts] >> [snorts] >> Very cool. This is you've given me a lot >> Very cool. This is you've given me a lot >> Very cool. This is you've given me a lot to think about. to think about. to think about. I you know, I I I thought that chaos I you know, I I I thought that chaos I you know, I I I thought that chaos engineering and security chaos engineering and security chaos engineering and security chaos engineering and this kind of resilience engineering and this kind of resilience engineering and this kind of resilience was uh kind of a branding exercise, but was uh kind of a branding exercise, but was uh kind of a branding exercise, but it feels more concrete after having it feels more concrete after having it feels more concrete after having chatted with you. chatted with you. chatted with you. Yes, I mean, again, it was mostly a Yes, I mean, again, it was mostly a Yes, I mean, again, it was mostly a buzzword and it's part of playing the buzzword and it's part of playing the buzzword and it's part of playing the game that publishers have to play. I game that publishers have to play. I game that publishers have to play. I will say though for a very long time will say though for a very long time will say though for a very long time since I was since I was since I was a wee lad, as they say, uh I've been a wee lad, as they say, uh I've been a wee lad, as they say, uh I've been obsessed with chaos theory. obsessed with chaos theory. obsessed with chaos theory. And I do think theory, which is quite And I do think theory, which is quite And I do think theory, which is quite beautiful in the sense of like beautiful in the sense of like beautiful in the sense of like systems do have an order to them, but systems do have an order to them, but systems do have an order to them, but it's not necessarily predictable. It's it's not necessarily predictable. It's it's not necessarily predictable. It's more like mostly like a fractal or more like mostly like a fractal or more like mostly like a fractal or dragon curve in many cases, as any dragon curve in many cases, as any dragon curve in many cases, as any meteorologist knows well. So, we need to meteorologist knows well. So, we need to meteorologist knows well. So, we need to focus less on again, do we have control focus less on again, do we have control focus less on again, do we have control over it? Are we able to predict it? And over it? Are we able to predict it? And over it? Are we able to predict it? And more, you know, the quote I love is from more, you know, the quote I love is from more, you know, the quote I love is from Susan Elizabeth Howe, who's a geologist, Susan Elizabeth Howe, who's a geologist, Susan Elizabeth Howe, who's a geologist, who said, "A building doesn't care who said, "A building doesn't care who said, "A building doesn't care whether the earthquake was predicted or whether the earthquake was predicted or whether the earthquake was predicted or not. It either stays up or it doesn't."

not. It either stays up or it doesn't." not. It either stays up or it doesn't." That's good. That's very good. That's That's good. That's very good. That's That's good. That's very good. That's really good. That's very good. I feel really good. That's very good. I feel really good. That's very good. I feel like that's the essence of it. Yeah. like that's the essence of it. Yeah. like that's the essence of it. Yeah. I like that one. One of my favorites in I like that one. One of my favorites in I like that one. One of my favorites in the in a similar vein is a Babylon 5, the in a similar vein is a Babylon 5, the in a similar vein is a Babylon 5, "The avalanche has begun. It's too late "The avalanche has begun. It's too late "The avalanche has begun. It's too late for the pebbles to vote." for the pebbles to vote." for the pebbles to vote." That is also very good. Yes. Yes. That is also very good. Yes. Yes. That is also very good. Yes. Yes. >> pebble and I don't like this. This is >> pebble and I don't like this. This is >> pebble and I don't like this. This is not a good idea. This is happening. not a good idea. This is happening. not a good idea. This is happening. Sorry, this is happening. So, buckle up. Sorry, this is happening. So, buckle up. Sorry, this is happening. So, buckle up. >> Yeah. Exactly. >> Yeah. Exactly. >> Yeah. Exactly. >> you so much, Kelly Shortridge, for >> you so much, Kelly Shortridge, for >> you so much, Kelly Shortridge, for chatting with me today. Thank you for chatting with me today. Thank you for chatting with me today. Thank you for the great questions. Appreciate it. We the great questions. Appreciate it. We the great questions. Appreciate it. We have been chatting with Kelly have been chatting with Kelly have been chatting with Kelly Shortridge, the Chief Product Officer at Shortridge, the Chief Product Officer at Shortridge, the Chief Product Officer at Fastly. This has been another episode of Fastly. This has been another episode of Fastly. This has been another episode of Hanselminutes in association with the Hanselminutes in association with the Hanselminutes in association with the ACM Bytecast, and we'll see you again ACM Bytecast, and we'll see you again ACM Bytecast, and we'll see you again next week.