"Observabilitying" the Future of Software with Charity Majors

Because everyone is suddenly generating Because everyone is suddenly generating code faster than they can understand it. code faster than they can understand it. code faster than they can understand it. So, people are saying weird stuff like, So, people are saying weird stuff like, So, people are saying weird stuff like, "Oh, the code is free now and tokens are "Oh, the code is free now and tokens are "Oh, the code is free now and tokens are free." And they're not free. And what's free." And they're not free. And what's free." And they're not free. And what's more important is that I'm making code more important is that I'm making code more important is that I'm making code faster than I can personally absorb it. faster than I can personally absorb it. faster than I can personally absorb it. And some people are saying just let the And some people are saying just let the And some people are saying just let the code wash over you and I'm not doing code wash over you and I'm not doing code wash over you and I'm not doing that. You don't vibe into production. that. You don't vibe into production. that. You don't vibe into production. >> call. >> call. >> call. >> No, they're not on call because >> No, they're not on call because >> No, they're not on call because ultimately, yeah, if you're creating a ultimately, yeah, if you're creating a ultimately, yeah, if you're creating a bunch of code and trying to sleep bunch of code and trying to sleep bunch of code and trying to sleep through it and when production goes through it and when production goes through it and when production goes down, it's going to be a problem. down, it's going to be a problem. down, it's going to be a problem. >> Hey friends, you probably know that >> Hey friends, you probably know that >> Hey friends, you probably know that TextControl is a powerful library for TextControl is a powerful library for TextControl is a powerful library for document editing and PDF generation, but document editing and PDF generation, but document editing and PDF generation, but did you also know that they're a strong did you also know that they're a strong did you also know that they're a strong supporter of the developer community and supporter of the developer community and supporter of the developer community and it's part of their mission to build and it's part of their mission to build and it's part of their mission to build and support a strong community by being support a strong community by being support a strong community by being present, by listening to users and by present, by listening to users and by present, by listening to users and by sharing knowledge at conferences across sharing knowledge at conferences across sharing knowledge at conferences across Europe and the United States. If you're Europe and the United States. If you're Europe and the United States. If you're heading to a conference soon, heading to a conference soon, heading to a conference soon, maybe check if TextControl will be maybe check if TextControl will be maybe check if TextControl will be there. Stop by and say hi. You'll find there. Stop by and say hi. You'll find there. Stop by and say hi. You'll find their full conference calendar at their full conference calendar at their full conference calendar at textcontrol.com. textcontrol.com. textcontrol.com. That's t e x t control.com.

>> Hi, I'm Scott Hanselman and it's another >> Hi, I'm Scott Hanselman and it's another episode of Hanselminutes and it is the episode of Hanselminutes and it is the episode of Hanselminutes and it is the return of Charity Majors. She's the CEO return of Charity Majors. She's the CEO return of Charity Majors. She's the CEO and co-founder of Honeycomb and the and co-founder of Honeycomb and the and co-founder of Honeycomb and the world's most decorated engineer around world's most decorated engineer around world's most decorated engineer around observability and distributed systems. I observability and distributed systems. I observability and distributed systems. I know of no other person who when someone know of no other person who when someone know of no other person who when someone says observability, they say Charity says observability, they say Charity says observability, they say Charity Majors. How are you? Majors. How are you? Majors. How are you? >> I'm great. I'm the CTO. >> I'm great. I'm the CTO. >> I'm great. I'm the CTO. >> You're the CTO? Oh my goodness. >> You're the CTO? Oh my goodness. >> You're the CTO? Oh my goodness. >> was the CEO for the first few years, but >> was the CEO for the first few years, but >> was the CEO for the first few years, but I was a terrible CEO. I was a terrible CEO. I was a terrible CEO. >> Oh, no. Okay. >> Oh, no. Okay. >> Oh, no. Okay. >> and I swapped places a few years >> and I swapped places a few years >> and I swapped places a few years >> to update the about page on charity.wtf. >> to update the about page on charity.wtf. >> to update the about page on charity.wtf. >> Oh my god, seriously? >> Oh my god, seriously? >> Oh my god, seriously? >> Yeah, I got the fresh I thought I was >> Yeah, I got the fresh I thought I was >> Yeah, I got the fresh I thought I was going to the fresh source. going to the fresh source. going to the fresh source. >> No, that's a very legitimate assumption. >> No, that's a very legitimate assumption. >> No, that's a very legitimate assumption. I I I >> Yeah, yeah, yeah. >> Yeah, yeah, yeah. >> Yeah, yeah, yeah. >> Yeah. >> Yeah. >> Yeah. >> Go to your blog. I got to go update my >> Go to your blog. I got to go update my >> Go to your blog. I got to go update my about page as well and make sure it's about page as well and make sure it's about page as well and make sure it's not saying things about me that are not not saying things about me that are not not saying things about me that are not true. true. true. >> All right, I'll That's on my to-do list >> All right, I'll That's on my to-do list >> All right, I'll That's on my to-do list now. now. now. >> Excellent, excellent. Get one of your >> Excellent, excellent. Get one of your >> Excellent, excellent. Get one of your agents to work on that. agents to work on that. agents to work on that. >> Yeah, exactly. >> Yeah, exactly. >> Yeah, exactly. >> So, uh let me just dive right in, You've >> So, uh let me just dive right in, You've >> So, uh let me just dive right in, You've been talking about observability for a been talking about observability for a been talking about observability for a decade. Uh what do you think people decade. Uh what do you think people decade. Uh what do you think people finally understand now and what are finally understand now and what are finally understand now and what are people still like getting wrong?

people still like getting wrong? people still like getting wrong? >> Oh, boy. >> Oh, boy. >> Oh, boy. >> [laughter] >> [laughter] >> [laughter] [gasps] [gasps] [gasps] >> You know, it's been really >> You know, it's been really >> You know, it's been really fun in the last 2 months. It feels like fun in the last 2 months. It feels like fun in the last 2 months. It feels like people have realized people have realized people have realized that that that you don't need metrics, logs, traces, you don't need metrics, logs, traces, you don't need metrics, logs, traces, all of the different signals. Actually, all of the different signals. Actually, all of the different signals. Actually, you just need you just need you just need traces. You just need structured data traces. You just need structured data traces. You just need structured data that is connective, connectable. You that is connective, connectable. You that is connective, connectable. You could call that traces with spans. You could call that traces with spans. You could call that traces with spans. You could call that, you know, wide could call that, you know, wide could call that, you know, wide structured logs, whatever. What you need structured logs, whatever. What you need structured logs, whatever. What you need is is is there's actually so much more power in there's actually so much more power in there's actually so much more power in having your data united, not fragmented. having your data united, not fragmented. having your data united, not fragmented. And metrics and logs and you know, they And metrics and logs and you know, they And metrics and logs and you know, they will always be with us, but they're will always be with us, but they're will always be with us, but they're they're really they're the exhaust pipe they're really they're the exhaust pipe they're really they're the exhaust pipe of your infrastructure, right? It's for of your infrastructure, right? It's for of your infrastructure, right? It's for It's for the telemetry that you don't It's for the telemetry that you don't It's for the telemetry that you don't control. You can't change it. You just control. You can't change it. You just control. You can't change it. You just have to operate it, right? That's Those have to operate it, right? That's Those have to operate it, right? That's Those are your metrics and logs. But for the are your metrics and logs. But for the are your metrics and logs. But for the for for your software, the software you for for your software, the software you for for your software, the software you do control, do control, do control, your telemetry is a product problem. The your telemetry is a product problem. The your telemetry is a product problem. The trace is part of the product, right?

trace is part of the product, right? trace is part of the product, right? Being able to validate is part of the Being able to validate is part of the Being able to validate is part of the product. And it has been so fun the last product. And it has been so fun the last product. And it has been so fun the last couple months as you just see people couple months as you just see people couple months as you just see people like on X and on LinkedIn going, "Wait, like on X and on LinkedIn going, "Wait, like on X and on LinkedIn going, "Wait, wait, wait. Have you ever considered wait, wait. Have you ever considered wait, wait. Have you ever considered that maybe all we need is traces?" And that maybe all we need is traces?" And that maybe all we need is traces?" And I'm just like, "Duh, da, da, da." I'm just like, "Duh, da, da, da." I'm just like, "Duh, da, da, da." >> [laughter] >> [laughter] >> [laughter] >> This is amazing. >> This is amazing. >> This is amazing. >> This is one of my superpowers and one of >> This is one of my superpowers and one of >> This is one of my superpowers and one of the things that's worse about me is my the things that's worse about me is my the things that's worse about me is my ability to make really bad analogies, ability to make really bad analogies, ability to make really bad analogies, but I feel like observability and then but I feel like observability and then but I feel like observability and then putting it in the context of agents is putting it in the context of agents is putting it in the context of agents is like got here debugging at scale. Like like got here debugging at scale. Like like got here debugging at scale. Like before debuggers, before attaching before debuggers, before attaching before debuggers, before attaching debugger and stepping through debugger, debugger and stepping through debugger, debugger and stepping through debugger, we would just put like we would just put like we would just put like console.writeline, "Got here." console.writeline, "Got here." console.writeline, "Got here." Console.writeline, "Got here number Console.writeline, "Got here number Console.writeline, "Got here number two." And then you would just your whole two." And then you would just your whole two." And then you would just your whole life was just a series of bisects as you life was just a series of bisects as you life was just a series of bisects as you like it's between here and here. And I like it's between here and here. And I like it's between here and here. And I was recently doing some agent work on a was recently doing some agent work on a was recently doing some agent work on a Windows app and it was very visual and Windows app and it was very visual and Windows app and it was very visual and someone said, "Oh, you should use someone said, "Oh, you should use someone said, "Oh, you should use computer use." And I was like, "No, computer use." And I was like, "No, computer use." And I was like, "No, you're going to take screenshots of you're going to take screenshots of you're going to take screenshots of pixels and you're going to burn carbon pixels and you're going to burn carbon pixels and you're going to burn carbon and waste energy.

and waste energy. and waste energy. This feels like a debug.writeline This feels like a debug.writeline This feels like a debug.writeline situation." situation." situation." So I just observability the hell out of So I just observability the hell out of So I just observability the hell out of the entire system and then made an MCP the entire system and then made an MCP the entire system and then made an MCP server that pointed to it and then the server that pointed to it and then the server that pointed to it and then the agent looped on it and solved it within agent looped on it and solved it within agent looped on it and solved it within minutes. minutes. minutes. It was very much a got here debugging at It was very much a got here debugging at It was very much a got here debugging at scale problem. scale problem. scale problem. >> I respect the use of observability as a >> I respect the use of observability as a >> I respect the use of observability as a verb. verb. verb. That was well done. That was well done. That was well done. >> Thank you. I I think that's a thing we >> Thank you. I I think that's a thing we >> Thank you. I I think that's a thing we should make that a thing. should make that a thing. should make that a thing. >> I knew too. I observability the I >> I knew too. I observability the I >> I knew too. I observability the I observability the hell out of it. I will observability the hell out of it. I will observability the hell out of it. I will use that phrase. use that phrase. use that phrase. >> That is the new title of this podcast. I >> That is the new title of this podcast. I >> That is the new title of this podcast. I observability the hell out of it. observability the hell out of it. observability the hell out of it. >> Also by the way, I'm a sucker for a >> Also by the way, I'm a sucker for a >> Also by the way, I'm a sucker for a title that is a pun. Hansel minutes. title that is a pun. Hansel minutes. title that is a pun. Hansel minutes. >> Yeah, you like that? You know what? >> Yeah, you like that? You know what? >> Yeah, you like that? You know what? That's actually because I'm a really bad That's actually because I'm a really bad That's actually because I'm a really bad estimator. estimator. estimator. And 20 plus years ago someone was asking And 20 plus years ago someone was asking And 20 plus years ago someone was asking me how long something was going to take me how long something was going to take me how long something was going to take and I was telling it was going to take and I was telling it was going to take and I was telling it was going to take about 30 minutes and they went "Really? about 30 minutes and they went "Really? about 30 minutes and they went "Really? Are those minutes or Hansel minutes?" Are those minutes or Hansel minutes?" Are those minutes or Hansel minutes?" And they were right cuz it was about six And they were right cuz it was about six And they were right cuz it was about six hours. hours. hours. >> [laughter] >> [laughter] >> [laughter] >> That was well done. >> That was well done. >> That was well done. So that is what people are getting So that is what people are getting So that is what people are getting right, I think. right, I think. right, I think. >> Okay. >> Okay. >> Okay. At long last. At long last. At long last. >> Uh the the other part of that that I >> Uh the the other part of that that I >> Uh the the other part of that that I don't think they're they've really don't think they're they've really don't think they're they've really gotten gotten gotten yet is just wrapping their yet is just wrapping their yet is just wrapping their heads around the part where this is a heads around the part where this is a heads around the part where this is a product problem and where there's product problem and where there's product problem and where there's actually actually actually It's like one of the smallest It's like one of the smallest It's like one of the smallest investments you can make in your future investments you can make in your future investments you can make in your future with the biggest payoff. It's just like with the biggest payoff. It's just like with the biggest payoff. It's just like devoting just a little bit of attention devoting just a little bit of attention devoting just a little bit of attention to what you're putting in and what it's, to what you're putting in and what it's, to what you're putting in and what it's, you know, what it's what it's married you know, what it's what it's married you know, what it's what it's married with contextually, you know, because with contextually, you know, because with contextually, you know, because once you've separated the bits of data once you've separated the bits of data once you've separated the bits of data that all describe the same thing, you that all describe the same thing, you that all describe the same thing, you can never knit them back together.

can never knit them back together. can never knit them back together. But if they're knit together, you can But if they're knit together, you can But if they're knit together, you can tear them apart and put them in tear them apart and put them in tear them apart and put them in different dashboards and all these different dashboards and all these different dashboards and all these things is that your heart desires. things is that your heart desires. things is that your heart desires. >> And and and and and graphs and charts >> And and and and and graphs and charts >> And and and and and graphs and charts and visualizations and and visualizations and and visualizations and whatever makes you happy, but you've got whatever makes you happy, but you've got whatever makes you happy, but you've got to bring all the stuff together. And 30 to bring all the stuff together. And 30 to bring all the stuff together. And 30 years ago, when I was in banking, we years ago, when I was in banking, we years ago, when I was in banking, we were we were FTPing log files around and were we were FTPing log files around and were we were FTPing log files around and grepping our way to glory. And it was a grepping our way to glory. And it was a grepping our way to glory. And it was a nightmare. And observability and OTel nightmare. And observability and OTel nightmare. And observability and OTel would have solved all of that for me. would have solved all of that for me. would have solved all of that for me. >> And with OTel and auto instrumentation >> And with OTel and auto instrumentation >> And with OTel and auto instrumentation now, I think it is genuinely possible to now, I think it is genuinely possible to now, I think it is genuinely possible to build faster with telemetry than build faster with telemetry than build faster with telemetry than without, which is huge and I think without, which is huge and I think without, which is huge and I think under-appreciated. under-appreciated. under-appreciated. You know, I don't blame software You know, I don't blame software You know, I don't blame software engineers for engineers for engineers for I I I I have this theory that like the entire I have this theory that like the entire I have this theory that like the entire 20-year journey of DevOps was really all 20-year journey of DevOps was really all 20-year journey of DevOps was really all about us trying to create one feedback about us trying to create one feedback about us trying to create one feedback loop that included both people writing loop that included both people writing loop that included both people writing the code and the code in production, and the code and the code in production, and the code and the code in production, and it completely failed because it completely failed because it completely failed because it was just too hard. It was so Like it was just too hard. It was so Like it was just too hard. It was so Like imagine you're a software engineer, it's imagine you're a software engineer, it's imagine you're a software engineer, it's the year 2020, pre-AI. You are staring the year 2020, pre-AI. You are staring the year 2020, pre-AI. You are staring at your code and you're thinking about, at your code and you're thinking about, at your code and you're thinking about, okay, I need to instrument this. All okay, I need to instrument this. All okay, I need to instrument this. All right, I've got a bit of data. Does this right, I've got a bit of data. Does this right, I've got a bit of data. Does this go in a metric, a log, a trace, an go in a metric, a log, a trace, an go in a metric, a log, a trace, an exception, an error, a profiling, all of exception, an error, a profiling, all of exception, an error, a profiling, all of the above? If you decide it's a metric, the above? If you decide it's a metric, the above? If you decide it's a metric, does it Is it a counter? Is it a gauge?

does it Is it a counter? Is it a gauge? does it Is it a counter? Is it a gauge? Is it Just all these different like And Is it Just all these different like And Is it Just all these different like And then what's the data type? Is there then what's the data type? Is there then what's the data type? Is there going to be high card? Like just the the going to be high card? Like just the the going to be high card? Like just the the bench of deep knowledge that you bench of deep knowledge that you bench of deep knowledge that you >> true. That's so true. >> true. That's so true. >> true. That's so true. >> And then, if you get it in there, you >> And then, if you get it in there, you >> And then, if you get it in there, you get it deployed, you got the right data get it deployed, you got the right data get it deployed, you got the right data types, all these things. types, all these things. types, all these things. Okay, it's out there. Now, Okay, it's out there. Now, Okay, it's out there. Now, what are you going to do? Okay, now you what are you going to do? Okay, now you what are you going to do? Okay, now you go and you you have to like try and find go and you you have to like try and find go and you you have to like try and find it and craft the right dashboard and do it and craft the right dashboard and do it and craft the right dashboard and do the right thing. the right thing. the right thing. >> Oh my goodness. >> Oh my goodness. >> Oh my goodness. >> two, three time You you are the amount >> two, three time You you are the amount >> two, three time You you are the amount of labor that you and load and cognitive of labor that you and load and cognitive of labor that you and load and cognitive load You you're doubling if not five x load You you're doubling if not five x load You you're doubling if not five x in your life. in your life. in your life. >> And ironically, you're going to be >> And ironically, you're going to be >> And ironically, you're going to be putting it somewhere and squirreling it putting it somewhere and squirreling it putting it somewhere and squirreling it away like on Windows when I was working away like on Windows when I was working away like on Windows when I was working in Windows like 30 years ago, it was in Windows like 30 years ago, it was in Windows like 30 years ago, it was like perf counters. And we would throw like perf counters. And we would throw like perf counters. And we would throw all this data into perf counters, and we all this data into perf counters, and we all this data into perf counters, and we would never look at it again. would never look at it again. would never look at it again. >> Yeah. >> Yeah. >> Yeah. >> Until there was a crisis. >> Until there was a crisis. >> Until there was a crisis. >> Yeah. >> Yeah. >> Yeah. Yeah. And if you're not used if you're Yeah. And if you're not used if you're Yeah. And if you're not used if you're not fluent in it, looking at it it's not fluent in it, looking at it it's not fluent in it, looking at it it's just it's just been so hard. And one of just it's just been so hard. And one of just it's just been so hard. And one of the most exciting things about AI to me the most exciting things about AI to me the most exciting things about AI to me is the ability to close this loop and to is the ability to close this loop and to is the ability to close this loop and to and to be understanding as pretty much and to be understanding as pretty much and to be understanding as pretty much as you're writing it, you know?

as you're writing it, you know? as you're writing it, you know? >> Mhm. Yeah. Yeah. Yeah. Yeah. Okay. So, >> Mhm. Yeah. Yeah. Yeah. Yeah. Okay. So, >> Mhm. Yeah. Yeah. Yeah. Yeah. Okay. So, recently this the second of the second recently this the second of the second recently this the second of the second edition of observability engineering edition of observability engineering edition of observability engineering >> Yes. >> Yes. >> Yes. >> is happening. Is it out now or is it >> is happening. Is it out now or is it >> is happening. Is it out now or is it coming out in a week or so? coming out in a week or so? coming out in a week or so? >> It can be downloaded as of Wednesday. >> It can be downloaded as of Wednesday. >> It can be downloaded as of Wednesday. The hard the dead tree ones won't be out The hard the dead tree ones won't be out The hard the dead tree ones won't be out for another month, but you you can be for another month, but you you can be for another month, but you you can be downloaded now. downloaded now. downloaded now. >> Very exciting. Okay. So, second edition, >> Very exciting. Okay. So, second edition, >> Very exciting. Okay. So, second edition, but I I would argue that it's arriving but I I would argue that it's arriving but I I would argue that it's arriving at a kind of a weird moment because at a kind of a weird moment because at a kind of a weird moment because everyone is suddenly generating code everyone is suddenly generating code everyone is suddenly generating code faster than they can understand it. So, faster than they can understand it. So, faster than they can understand it. So, people are saying weird stuff like, "Oh, people are saying weird stuff like, "Oh, people are saying weird stuff like, "Oh, the code is free now, and tokens are the code is free now, and tokens are the code is free now, and tokens are free." And they're not free. And what's free." And they're not free. And what's free." And they're not free. And what's more important is that I'm making code more important is that I'm making code more important is that I'm making code faster than I can personally absorb it. faster than I can personally absorb it. faster than I can personally absorb it. And some people are saying just let the And some people are saying just let the And some people are saying just let the code wash over you, and I'm not code wash over you, and I'm not code wash over you, and I'm not you don't vibe into production. you don't vibe into production. you don't vibe into production. No, they're not on call because No, they're not on call because No, they're not on call because ultimately, yeah, if you're creating a ultimately, yeah, if you're creating a ultimately, yeah, if you're creating a bunch of code and trying to sleep bunch of code and trying to sleep bunch of code and trying to sleep through it and when production goes through it and when production goes through it and when production goes down, it's going to be a problem. So, down, it's going to be a problem. So, down, it's going to be a problem. So, what changed enough that the book had to what changed enough that the book had to what changed enough that the book had to change? change? change? >> Oh, that's such a great question. So, >> Oh, that's such a great question. So, >> Oh, that's such a great question. So, first of all, it's an O'Reilly thing. If first of all, it's an O'Reilly thing. If first of all, it's an O'Reilly thing. If the book was relatively successful, then the book was relatively successful, then the book was relatively successful, then two three years later they want a second two three years later they want a second two three years later they want a second edition cuz technology changes.

edition cuz technology changes. edition cuz technology changes. But, so the first book But, so the first book But, so the first book All right, the first edition it's about All right, the first edition it's about All right, the first edition it's about 250 pages, and it took us 3 and 1/2 250 pages, and it took us 3 and 1/2 250 pages, and it took us 3 and 1/2 years to write it because years to write it because years to write it because >> Wow. >> Wow. >> Wow. >> we were chasing a moving target. >> we were chasing a moving target. >> we were chasing a moving target. Everything Like, the definition of Everything Like, the definition of Everything Like, the definition of observability when we started writing observability when we started writing observability when we started writing changed two or three times over to when changed two or three times over to when changed two or three times over to when we finished, which was we finished, which was we finished, which was deeply frustrating. And And at the end, deeply frustrating. And And at the end, deeply frustrating. And And at the end, you never want to say that you're not you never want to say that you're not you never want to say that you're not proud of a book, but proud of a book, but proud of a book, but I there was a new point where we were I there was a new point where we were I there was a new point where we were just like, we feel great about it. It just like, we feel great about it. It just like, we feel great about it. It was just like, oh my god, I can't look was just like, oh my god, I can't look was just like, oh my god, I can't look at it any Please just get it I don't at it any Please just get it I don't at it any Please just get it I don't care what it says. I can't read it any care what it says. I can't read it any care what it says. I can't read it any Just get it away from That was the sort Just get it away from That was the sort Just get it away from That was the sort I'm grateful to the people who read it I'm grateful to the people who read it I'm grateful to the people who read it cuz I just cringe when I read it. So, I cuz I just cringe when I read it. So, I cuz I just cringe when I read it. So, I was really excited to write it a second was really excited to write it a second was really excited to write it a second time. I was like, yeah, it feels like time. I was like, yeah, it feels like time. I was like, yeah, it feels like the definition of observability is the definition of observability is the definition of observability is stabilized and observability is now the stabilized and observability is now the stabilized and observability is now the sort of first principles thing and the sort of first principles thing and the sort of first principles thing and the rest of the world has gone batshit crazy rest of the world has gone batshit crazy rest of the world has gone batshit crazy out of their minds. So, we started with it's over 600 pages and I'm a little it's over 600 pages and I'm a little sheepish sheepish sheepish >> from Oh, hang on. First edition 250, >> from Oh, hang on. First edition 250, >> from Oh, hang on. First edition 250, second edition 600.

second edition 600. second edition 600. >> Yeah. And And I cut like 200 pages. Like >> Yeah. And And I cut like 200 pages. Like >> Yeah. And And I cut like 200 pages. Like it it it >> This is a new book. >> This is a new book. >> This is a new book. >> It's It's It's an entirely rewritten. >> It's It's It's an entirely rewritten. >> It's It's It's an entirely rewritten. The whole thing has been rewritten. The whole thing has been rewritten. The whole thing has been rewritten. There's 27 new chapters. Only like four There's 27 new chapters. Only like four There's 27 new chapters. Only like four chapters even carried over any material. chapters even carried over any material. chapters even carried over any material. It's a whole new book. It's a whole new book. It's a whole new book. >> Oh my goodness. Okay, well. Which is >> Oh my goodness. Okay, well. Which is >> Oh my goodness. Okay, well. Which is Which is funny because I just said that Which is funny because I just said that Which is funny because I just said that people are generating content and code people are generating content and code people are generating content and code faster than I can understand it. faster than I can understand it. >> [laughter] >> So, here I am now. I'm going to have to >> So, here I am now. I'm going to have to >> So, here I am now. I'm going to have to absorb this book. Uh absorb this book. Uh absorb this book. Uh >> No, you No, you don't. So, let me give >> No, you No, you don't. So, let me give >> No, you No, you don't. So, let me give you a little bit just sort of a you a little bit just sort of a you a little bit just sort of a bird's-eye view. bird's-eye view. bird's-eye view. There's six sections. There's six sections. There's six sections. The first section is just sort of like The first section is just sort of like The first section is just sort of like So, I wrote section one and section six. So, I wrote section one and section six. So, I wrote section one and section six. Uh my co-authors wrote two and three and Uh my co-authors wrote two and three and Uh my co-authors wrote two and three and then we have just a wide range of then we have just a wide range of then we have just a wide range of incredible guest authors who contributed incredible guest authors who contributed incredible guest authors who contributed like sort of deep dive and use case like sort of deep dive and use case like sort of deep dive and use case chapters in four and five. chapters in four and five. chapters in four and five. So, So, So, the first part is sort of like the first part is sort of like the first part is sort of like to me one of the differences between to me one of the differences between to me one of the differences between monitoring and observability is monitoring and observability is monitoring and observability is monitoring was very pegged in, monitoring was very pegged in, monitoring was very pegged in, uh you know, here here are the grooves uh you know, here here are the grooves uh you know, here here are the grooves we've worn in our systems. This is how we've worn in our systems. This is how we've worn in our systems. This is how we know if it's up or down. To me, we know if it's up or down. To me, we know if it's up or down. To me, observability is reasoning from first observability is reasoning from first observability is reasoning from first principles about how to understand our principles about how to understand our principles about how to understand our code.

code. code. And so, the first chapter was kind of And so, the first chapter was kind of And so, the first chapter was kind of like acknowledging like what you said like acknowledging like what you said like acknowledging like what you said about this is a crazy moment. about this is a crazy moment. about this is a crazy moment. The possible futures seem way The possible futures seem way The possible futures seem way wider open than usual and it's a little wider open than usual and it's a little wider open than usual and it's a little scary, it's a little terrifying. scary, it's a little terrifying. scary, it's a little terrifying. But like at a moment like this, I think But like at a moment like this, I think But like at a moment like this, I think first principles matter more than ever. first principles matter more than ever. first principles matter more than ever. And so what we hope we can do in this And so what we hope we can do in this And so what we hope we can do in this book is equip you with the tools to book is equip you with the tools to book is equip you with the tools to navigate this scary time. navigate this scary time. navigate this scary time. >> When you said first principles matter >> When you said first principles matter >> When you said first principles matter more than ever, I was presenting at a more than ever, I was presenting at a more than ever, I was presenting at a very, very, very large company. We do very, very, very large company. We do very, very, very large company. We do these things at Microsoft called EBCs, these things at Microsoft called EBCs, these things at Microsoft called EBCs, executive briefings, where they fly all executive briefings, where they fly all executive briefings, where they fly all the C-suite people and everyone wears a the C-suite people and everyone wears a the C-suite people and everyone wears a suit and then I feel awkward wearing a suit and then I feel awkward wearing a suit and then I feel awkward wearing a t-shirt. And then they ask questions t-shirt. And then they ask questions t-shirt. And then they ask questions about what do you think about the future about what do you think about the future about what do you think about the future of engineering or whatever and I just of engineering or whatever and I just of engineering or whatever and I just kept saying kept saying kept saying first principles, first principles. This first principles, first principles. This first principles, first principles. This was literally yesterday. I was in was literally yesterday. I was in was literally yesterday. I was in Seattle. It's like you've got and I've Seattle. It's like you've got and I've Seattle. It's like you've got and I've always said this on the podcast, people always said this on the podcast, people always said this on the podcast, people are sick of me, you've got to learn how are sick of me, you've got to learn how are sick of me, you've got to learn how to drive stick shift. It doesn't mean to drive stick shift. It doesn't mean to drive stick shift. It doesn't mean you have to drive stick shift all the you have to drive stick shift all the you have to drive stick shift all the time, but your relationship with the car time, but your relationship with the car time, but your relationship with the car is fundamentally different if you drive is fundamentally different if you drive is fundamentally different if you drive stick. stick. stick. >> I love that. >> I love that. >> I love that. >> can't observe your system, you don't >> can't observe your system, you don't >> can't observe your system, you don't know what's going on. know what's going on. know what's going on. And AI is now making code so quickly, And AI is now making code so quickly, And AI is now making code so quickly, >> Yeah.

>> Yeah. >> Yeah. >> leaders hear that and they go, "Oh, we >> leaders hear that and they go, "Oh, we >> leaders hear that and they go, "Oh, we can ship more." can ship more." can ship more." But you you seem to be saying, "Now, we But you you seem to be saying, "Now, we But you you seem to be saying, "Now, we need better feedback loops." need better feedback loops." need better feedback loops." >> Yes. >> Yes. >> Yes. You know what creates a feedback loop? You know what creates a feedback loop? You know what creates a feedback loop? Observability. Observability. Observability. It's the Otherwise, you just have a It's the Otherwise, you just have a It's the Otherwise, you just have a bunch of things happening and a bunch of bunch of things happening and a bunch of bunch of things happening and a bunch of things causing things. things causing things. things causing things. >> Right. >> Right. >> Right. >> It's only if you observe the connection >> It's only if you observe the connection >> It's only if you observe the connection between the two that you have a feedback between the two that you have a feedback between the two that you have a feedback loop. loop. loop. >> So it's funny that you say that because >> So it's funny that you say that because >> So it's funny that you say that because I I was trying to explain like LLMs to a I I was trying to explain like LLMs to a I I was trying to explain like LLMs to a bunch of muggles and they weren't bunch of muggles and they weren't bunch of muggles and they weren't getting it and I talked about the a getting it and I talked about the a getting it and I talked about the a million monkeys with a million million monkeys with a million million monkeys with a million typewriters will eventually write typewriters will eventually write typewriters will eventually write Shakespeare, but only if those monkeys Shakespeare, but only if those monkeys Shakespeare, but only if those monkeys are you're actually observing the output are you're actually observing the output are you're actually observing the output and evaluating it against Shakespeare. and evaluating it against Shakespeare. and evaluating it against Shakespeare. Right? You can't just randomly pick one Right? You can't just randomly pick one Right? You can't just randomly pick one of those monkeys and say, "Look, of those monkeys and say, "Look, of those monkeys and say, "Look, Shakespeare." Shakespeare." Shakespeare." Right? The feedback loop and the the Right? The feedback loop and the the Right? The feedback loop and the the reinforcement learning effectively to reinforcement learning effectively to reinforcement learning effectively to extend the analogy is required. So that extend the analogy is required. So that extend the analogy is required. So that would imply that if someone is trying to would imply that if someone is trying to would imply that if someone is trying to go and implement AI agents and you know, go and implement AI agents and you know, go and implement AI agents and you know, agentic loops in software on brownfield agentic loops in software on brownfield agentic loops in software on brownfield systems that have no observability, systems that have no observability, systems that have no observability, they're probably screwed.

they're probably screwed. they're probably screwed. >> Oh, so deeply screwed. Deeply. >> Oh, so deeply screwed. Deeply. >> Oh, so deeply screwed. Deeply. >> Mhm. >> Mhm. >> Mhm. >> All right, so part one is sort of the >> All right, so part one is sort of the >> All right, so part one is sort of the context setting history first principles context setting history first principles context setting history first principles and then parts two and three are about and then parts two and three are about and then parts two and three are about for practitioners software engineers how for practitioners software engineers how for practitioners software engineers how to instrument your code and how to to instrument your code and how to to instrument your code and how to understand your telemetry. And we've got understand your telemetry. And we've got understand your telemetry. And we've got two tracks. There is doing it kind of I two tracks. There is doing it kind of I two tracks. There is doing it kind of I think of them as Coke Zero and Coke think of them as Coke Zero and Coke think of them as Coke Zero and Coke Classic doing it with Classic doing it with Classic doing it with AI and doing it you know, without AI. AI and doing it you know, without AI. AI and doing it you know, without AI. They're both they're they're both really They're both they're they're both really They're both they're they're both really solid. I had nothing to do with them but solid. I had nothing to do with them but solid. I had nothing to do with them but they're I think they're pretty good. And they're I think they're pretty good. And they're I think they're pretty good. And then we have part four and part five. then we have part four and part five. then we have part four and part five. One of them is deep dives into like use One of them is deep dives into like use One of them is deep dives into like use cases and the other one is into use cases and the other one is into use cases and the other one is into use cases and cases and cases and I don't know what you call it. I don't know what you call it. I don't know what you call it. They're system studies, right? Like They're system studies, right? Like They're system studies, right? Like a deep dive into agentic development and a deep dive into agentic development and a deep dive into agentic development and a deep dive into deep dive use cases, a deep dive into deep dive use cases, a deep dive into deep dive use cases, right? Into like the storage engines right? Into like the storage engines right? Into like the storage engines that power high cardinality telemetry that power high cardinality telemetry that power high cardinality telemetry systems. That's what they are. systems. That's what they are. systems. That's what they are. >> [snorts] >> [snorts] >> [snorts] >> And then >> And then >> And then part six is part six is part six is it ended up being a third of the book. it ended up being a third of the book. it ended up being a third of the book. >> [laughter] >> [laughter] >> [laughter] >> It's a >> It's a >> It's a it's it's it's All right, so if I can take to the tiny All right, so if I can take to the tiny All right, so if I can take to the tiny detour cuz this was this was supposed to detour cuz this was this was supposed to detour cuz this was this was supposed to be my little three I'm like at the end be my little three I'm like at the end be my little three I'm like at the end of the book I think I'll write a new of the book I think I'll write a new of the book I think I'll write a new section on for observability engineering section on for observability engineering section on for observability engineering teams on governance topics and I wrote teams on governance topics and I wrote teams on governance topics and I wrote it and then I ended up rewriting it it and then I ended up rewriting it it and then I ended up rewriting it twice and it turned into something twice and it turned into something twice and it turned into something completely different completely different completely different and it ends up being and it ends up being and it ends up being governance yes, but it starts with an governance yes, but it starts with an governance yes, but it starts with an open letter to CTOs and why all their open letter to CTOs and why all their open letter to CTOs and why all their hopes and dreams for AI are blocked hopes and dreams for AI are blocked hopes and dreams for AI are blocked behind their ability to

behind their ability to behind their ability to understand their systems and and it kind understand their systems and and it kind understand their systems and and it kind of starts at the top of the org chart of starts at the top of the org chart of starts at the top of the org chart and goes all the way down to like and goes all the way down to like and goes all the way down to like observability. Like what do VPs, what do observability. Like what do VPs, what do observability. Like what do VPs, what do distinguished engineers, what do what do distinguished engineers, what do what do distinguished engineers, what do what do people need to know? So like it starts people need to know? So like it starts people need to know? So like it starts with the open letter to CTOs and then with the open letter to CTOs and then with the open letter to CTOs and then there is two chapters of just systems there is two chapters of just systems there is two chapters of just systems thinking for software delivery all about thinking for software delivery all about thinking for software delivery all about feedback loops and feedback loops and feedback loops and and and how this works and you know and and how this works and you know and and how this works and you know and how and how and how >> That sounds like a book of its own >> That sounds like a book of its own >> That sounds like a book of its own honestly. honestly. honestly. >> It it really is >> It it really is >> It it really is >> you could spin that off into like a >> you could spin that off into like a >> you could spin that off into like a little Goldfinger mini plane and it little Goldfinger mini plane and it little Goldfinger mini plane and it could be its own pamphlet. could be its own pamphlet. could be its own pamphlet. >> Kind of should and we might do that but >> Kind of should and we might do that but >> Kind of should and we might do that but it it it Yeah, cuz what CTO is going to like Yeah, cuz what CTO is going to like Yeah, cuz what CTO is going to like start two-thirds of the way through the start two-thirds of the way through the start two-thirds of the way through the book and book and book and >> Yeah, yeah, don't bury the lead. >> Yeah, yeah, don't bury the lead. >> Yeah, yeah, don't bury the lead. >> Don't bury the >> Don't bury the >> Don't bury the So it it's it's system thinking and then So it it's it's system thinking and then So it it's it's system thinking and then there's some stuff about like there's some stuff about like there's some stuff about like how to drive change in an organization how to drive change in an organization how to drive change in an organization with influence not formal power, how to with influence not formal power, how to with influence not formal power, how to how how how how to partner effectively with how to partner effectively with how to partner effectively with My actual favorite chapter in the book My actual favorite chapter in the book My actual favorite chapter in the book comes almost at the very end and it's my comes almost at the very end and it's my comes almost at the very end and it's my favorite not cuz I think it's brilliant favorite not cuz I think it's brilliant favorite not cuz I think it's brilliant but because I don't think I've ever but because I don't think I've ever but because I don't think I've ever heard anyone write about this seen heard anyone write about this seen heard anyone write about this seen anyone write about this but it's when I anyone write about this but it's when I anyone write about this but it's when I think about how think about how think about how improbable improbable improbable every transformation is like most of every transformation is like most of every transformation is like most of them fail. The vast majority of it of them fail. The vast majority of it of them fail. The vast majority of it of technical transformations fail.

technical transformations fail. technical transformations fail. The ones that succeed The ones that succeed The ones that succeed What do you need to make them succeed? What do you need to make them succeed? What do you need to make them succeed? Well, as as a very senior engineer, Well, as as a very senior engineer, Well, as as a very senior engineer, you need trust and credibility inside you need trust and credibility inside you need trust and credibility inside your organization. People need to your organization. People need to your organization. People need to believe that when you say something, it believe that when you say something, it believe that when you say something, it is true. It will happen. It will be is true. It will happen. It will be is true. It will happen. It will be backed up. That cuts through backed up. That cuts through backed up. That cuts through bureaucracy like like a hot knife bureaucracy like like a hot knife bureaucracy like like a hot knife through butter. It's one of the only through butter. It's one of the only through butter. It's one of the only things that does that deep credibility. things that does that deep credibility. things that does that deep credibility. And when it comes to partnering with And when it comes to partnering with And when it comes to partnering with vendors vendors vendors like when when it comes to you like when when it comes to you like when when it comes to you partnering with people in your team, you partnering with people in your team, you partnering with people in your team, you need to understand you're all on the need to understand you're all on the need to understand you're all on the same team. Everybody has their needs but same team. Everybody has their needs but same team. Everybody has their needs but you're all in the pursuit of the same you're all in the pursuit of the same you're all in the pursuit of the same goal and you need to keep reminding goal and you need to keep reminding goal and you need to keep reminding people of this, right? But when it comes people of this, right? But when it comes people of this, right? But when it comes to partnering with vendors to partnering with vendors to partnering with vendors you need you need to build trust and you need you need to build trust and you need you need to build trust and reciprocity. You need to understand you reciprocity. You need to understand you reciprocity. You need to understand you are not on the same team. You both want are not on the same team. You both want are not on the same team. You both want different things, right? And that the different things, right? And that the different things, right? And that the trust of believing what they say can be trust of believing what they say can be trust of believing what they say can be built over time but you should not start built over time but you should not start built over time but you should not start out believing what they say because out believing what they say because out believing what they say because they're trying to make a sale. You're they're trying to make a sale. You're they're trying to make a sale. You're trying to do what's in best for your trying to do what's in best for your trying to do what's in best for your company. And these things are just in company. And these things are just in company. And these things are just in natural tension, right? And so like how natural tension, right? And so like how natural tension, right? And so like how But if you're going to partner with a But if you're going to partner with a But if you're going to partner with a vendor, you want influence over their vendor, you want influence over their vendor, you want influence over their road map. They want to be able to road map. They want to be able to road map. They want to be able to believe that you will invest in, you believe that you will invest in, you believe that you will invest in, you know, for us it's like are you actually know, for us it's like are you actually know, for us it's like are you actually going to instrument your systems or are going to instrument your systems or are going to instrument your systems or are you just going to like buy it and then you just going to like buy it and then you just going to like buy it and then it's going to sit there and you're going it's going to sit there and you're going it's going to sit there and you're going to be unhappy.

to be unhappy. to be unhappy. You know, so like setting the the trust You know, so like setting the the trust You know, so like setting the the trust and credibility versus trust and and credibility versus trust and and credibility versus trust and reciprocity and like building successful reciprocity and like building successful reciprocity and like building successful long-term partnerships. These are long-term partnerships. These are long-term partnerships. These are durable skills in the era of AI for very durable skills in the era of AI for very durable skills in the era of AI for very senior engineers. And I tell a little senior engineers. And I tell a little senior engineers. And I tell a little story about story about story about Mark Callaghan, the best engineer I ever Mark Callaghan, the best engineer I ever Mark Callaghan, the best engineer I ever worked with, and how while I was in worked with, and how while I was in worked with, and how while I was in Facebook, he's in databases, right? I Facebook, he's in databases, right? I Facebook, he's in databases, right? I watched him alter watched him alter watched him alter permanently alter the trajectory of permanently alter the trajectory of permanently alter the trajectory of multiple billion-dollar companies just multiple billion-dollar companies just multiple billion-dollar companies just with a few conversations. And I was just with a few conversations. And I was just with a few conversations. And I was just like mind-blown. like mind-blown. like mind-blown. >> So you changed my whole idea. >> So you changed my whole idea. >> So you changed my whole idea. >> I I I have to send you this paper. I did >> I I I have to send you this paper. I did >> I I I have to send you this paper. I did this paper that Mark Russinovich and I this paper that Mark Russinovich and I this paper that Mark Russinovich and I have been working on about how if we have been working on about how if we have been working on about how if we don't double down on investing in the don't double down on investing in the don't double down on investing in the early-in-career people early-in-career people early-in-career people >> Yes, I read it. >> Yes, I read it. >> Yes, I read it. >> Did you read it? >> Did you read it? >> Did you read it? >> Yeah, I did. >> Yeah, I did. >> Yeah, I did. >> So So when I'm trying to explain to a >> So So when I'm trying to explain to a >> So So when I'm trying to explain to a 25-year-old about a Mark Callaghan type 25-year-old about a Mark Callaghan type 25-year-old about a Mark Callaghan type person person person they seem to think that that's a coding they seem to think that that's a coding they seem to think that that's a coding problem and it's not. It is a human problem and it's not. It is a human problem and it's not. It is a human being problem and as much as the AI being problem and as much as the AI being problem and as much as the AI grifters are out there selling AI and grifters are out there selling AI and grifters are out there selling AI and I'm one of them I'm one of them I'm one of them I am selling it with a humanistic I am selling it with a humanistic I am selling it with a humanistic perspective that like if it's if this perspective that like if it's if this perspective that like if it's if this new power tool doesn't make humans talk new power tool doesn't make humans talk new power tool doesn't make humans talk more and better, then we're doing it more and better, then we're doing it more and better, then we're doing it wrong and we've got to like stop it. So wrong and we've got to like stop it. So wrong and we've got to like stop it. So you're absolutely right.

you're absolutely right. you're absolutely right. Uh someone told me actually I I I was at Uh someone told me actually I I I was at Uh someone told me actually I I I was at this company I was talking to yesterday, this company I was talking to yesterday, this company I was talking to yesterday, they said, "What programming language they said, "What programming language they said, "What programming language should we teach the young people?" And I should we teach the young people?" And I should we teach the young people?" And I was not trying to be snarky, but I just was not trying to be snarky, but I just was not trying to be snarky, but I just said, "English." said, "English." said, "English." Like really really like it's not the Like really really like it's not the Like really really like it's not the language that we deserve, it's just the language that we deserve, it's just the language that we deserve, it's just the one that we have. It's like JavaScript, one that we have. It's like JavaScript, one that we have. It's like JavaScript, it's just happened and here this Now we it's just happened and here this Now we it's just happened and here this Now we all speak English. But if you can't do all speak English. But if you can't do all speak English. But if you can't do it clearly and crisply and be persuasive it clearly and crisply and be persuasive it clearly and crisply and be persuasive and be credible and be thoughtful, then and be credible and be thoughtful, then and be credible and be thoughtful, then you're pretty much screwed. And you you're pretty much screwed. And you you're pretty much screwed. And you could call that prompt engineering, could call that prompt engineering, could call that prompt engineering, which I still don't think is a thing, or which I still don't think is a thing, or which I still don't think is a thing, or you could just call it being a good you could just call it being a good you could just call it being a good communicator. But that is the communicator. But that is the communicator. But that is the programming language of the future. It's programming language of the future. It's programming language of the future. It's clarity. clarity. clarity. >> Yes. >> Yes. >> Yes. Right. It's thinking on paper. Right. It's thinking on paper. Right. It's thinking on paper. Content creation and writing are not the Content creation and writing are not the Content creation and writing are not the same things. same things. same things. >> They're not. They're they're really not. >> They're not. They're they're really not. >> They're not. They're they're really not. And neither is yapping on TikTok, as And neither is yapping on TikTok, as And neither is yapping on TikTok, as much as I do love TikTok. Now, you you much as I do love TikTok. Now, you you much as I do love TikTok. Now, you you have written in the past that the unit have written in the past that the unit have written in the past that the unit of work is changing, right? Like it's of work is changing, right? Like it's of work is changing, right? Like it's conversations, it's workflows, it's conversations, it's workflows, it's conversations, it's workflows, it's retries, it's tool calls. Does that retries, it's tool calls. Does that retries, it's tool calls. Does that change the old trace model? Do we need change the old trace model? Do we need change the old trace model? Do we need to observe different things at a to observe different things at a to observe different things at a different levels? different levels? different levels? >> You know, I >> You know, I >> You know, I So, I think that for most of my career, So, I think that for most of my career, So, I think that for most of my career, the unit of work has been the the unit of work has been the the unit of work has been the transaction.

transaction. transaction. And and it just isn't now. It it's it And and it just isn't now. It it's it And and it just isn't now. It it's it the unit of work is is usually the the unit of work is is usually the the unit of work is is usually the trace, but the trace is just structured trace, but the trace is just structured trace, but the trace is just structured data. One of the things that I find data. One of the things that I find data. One of the things that I find really frustrating about people who have really frustrating about people who have really frustrating about people who have been in observability for a long time been in observability for a long time been in observability for a long time is that they get really wrapped around is that they get really wrapped around is that they get really wrapped around the axle about signal types often. And the axle about signal types often. And the axle about signal types often. And it's just data. They're just differently it's just data. They're just differently it's just data. They're just differently enveloped, wrapped bits of data, right? enveloped, wrapped bits of data, right? enveloped, wrapped bits of data, right? And the more fungible those data types And the more fungible those data types And the more fungible those data types are, the more powerful it they typically are, the more powerful it they typically are, the more powerful it they typically are. And so, I think that, you know, we are. And so, I think that, you know, we are. And so, I think that, you know, we just shipped something, I think it went just shipped something, I think it went just shipped something, I think it went live yesterday into GA called timelines, live yesterday into GA called timelines, live yesterday into GA called timelines, which is so cool because which is so cool because which is so cool because you know, you can imagine if you if you you know, you can imagine if you if you you know, you can imagine if you if you if you're if you're if you're if you're like Intercom, you you build if you're like Intercom, you you build if you're like Intercom, you you build like this chat product, right? Uh like this chat product, right? Uh like this chat product, right? Uh the interaction begins when someone asks the interaction begins when someone asks the interaction begins when someone asks a question. And then you might have like a question. And then you might have like a question. And then you might have like a supervisor agent that kicks off and a supervisor agent that kicks off and a supervisor agent that kicks off and spawns spawns spawns many different agents. And each of those many different agents. And each of those many different agents. And each of those agents does a bunch of API calls and RPC agents does a bunch of API calls and RPC agents does a bunch of API calls and RPC calls and storage calls and everything. calls and storage calls and everything. calls and storage calls and everything. And then, you know, at some later point And then, you know, at some later point And then, you know, at some later point in time, they converge, they bring back in time, they converge, they bring back in time, they converge, they bring back an answer, and they and they return it.

an answer, and they and they return it. an answer, and they and they return it. And then, the user says something else, And then, the user says something else, And then, the user says something else, and it kicks off another. And like and it kicks off another. And like and it kicks off another. And like knitting together all those bits, that's knitting together all those bits, that's knitting together all those bits, that's not a transaction. That might take place not a transaction. That might take place not a transaction. That might take place over hours, right? And so the primitives over hours, right? And so the primitives over hours, right? And so the primitives have changed. have changed. have changed. >> Yeah. >> Yeah. >> Yeah. >> And our need to like go up the stack has >> And our need to like go up the stack has >> And our need to like go up the stack has changed. changed. changed. >> I had a really interesting conversation >> I had a really interesting conversation >> I had a really interesting conversation yesterday and I think it'll be a show yesterday and I think it'll be a show yesterday and I think it'll be a show with Nathan Zobo who worked on Zed and with Nathan Zobo who worked on Zed and with Nathan Zobo who worked on Zed and he's now working on a thing called Delta he's now working on a thing called Delta he's now working on a thing called Delta DB. DB. DB. And Delta DB has a perspective on top of And Delta DB has a perspective on top of And Delta DB has a perspective on top of Git that Git is committing the before Git that Git is committing the before Git that Git is committing the before code and the after code and it's missing code and the after code and it's missing code and the after code and it's missing all the interesting bits. So one could all the interesting bits. So one could all the interesting bits. So one could argue that it is observability in the argue that it is observability in the argue that it is observability in the development of software process where development of software process where development of software process where they are committing the conversations so they are committing the conversations so they are committing the conversations so that if you look at a line of code that if you look at a line of code that if you look at a line of code and let's say that that got refactored and let's say that that got refactored and let's say that that got refactored and moved somewhere. You could imagine and moved somewhere. You could imagine and moved somewhere. You could imagine putting your cursor on some code and putting your cursor on some code and putting your cursor on some code and then being jumped to another another then being jumped to another another then being jumped to another another temporal location which is a physical temporal location which is a physical temporal location which is a physical location or the conversation where that location or the conversation where that location or the conversation where that code was conceived of. code was conceived of. code was conceived of. So drawing a direct line from So drawing a direct line from So drawing a direct line from conversation into code into moving code.

conversation into code into moving code. conversation into code into moving code. So I feel like as much as people are So I feel like as much as people are So I feel like as much as people are lamenting that like lamenting that like lamenting that like the the the the crafts the the Japanese carpentry of the crafts the the Japanese carpentry of the crafts the the Japanese carpentry of bespoke assembly language like you know bespoke assembly language like you know bespoke assembly language like you know 6502 assembler like that those days may 6502 assembler like that those days may 6502 assembler like that those days may be over. The the days of us slapping the be over. The the days of us slapping the be over. The the days of us slapping the keyboard may be over but they're still keyboard may be over but they're still keyboard may be over but they're still really really really interesting really really really interesting really really really interesting philosophical work in the practice of philosophical work in the practice of philosophical work in the practice of software engineering. software engineering. software engineering. >> Oh god yes. This is still an engineering >> Oh god yes. This is still an engineering >> Oh god yes. This is still an engineering problem. We need more problem. We need more problem. We need more more rigor not less. more rigor not less. more rigor not less. >> One of my spicy takes and it's I don't >> One of my spicy takes and it's I don't >> One of my spicy takes and it's I don't know if it's good or not cuz I don't know if it's good or not cuz I don't know if it's good or not cuz I don't know how many people have this degree know how many people have this degree know how many people have this degree but I don't have a computer science but I don't have a computer science but I don't have a computer science degree. degree. degree. >> [snorts] >> [snorts] >> [snorts] >> I have a software engineering degree. >> I have a software engineering degree. >> I have a software engineering degree. >> Oh I'm just a music major dropout. >> Oh I'm just a music major dropout. >> Oh I'm just a music major dropout. >> Well music music is also a valid major >> Well music music is also a valid major >> Well music music is also a valid major if you want to become the CTO of a major if you want to become the CTO of a major if you want to become the CTO of a major company. But the reason I'm pointing company. But the reason I'm pointing company. But the reason I'm pointing that out is that I don't have a good that out is that I don't have a good that out is that I don't have a good background in computer science. Like I background in computer science. Like I background in computer science. Like I did compiler class and I did one and I did compiler class and I did one and I did compiler class and I did one and I was done. was done. was done. So I'm not a researcher. I don't write So I'm not a researcher. I don't write So I'm not a researcher. I don't write papers and research but I do know how to papers and research but I do know how to papers and research but I do know how to ship and when I started in the early 90s ship and when I started in the early 90s ship and when I started in the early 90s test driven development the gang of test driven development the gang of test driven development the gang of four, and like the Agile Manifesto was four, and like the Agile Manifesto was four, and like the Agile Manifesto was just getting started. And we called them just getting started. And we called them just getting started. And we called them build servers, and now it's DevOps. So, build servers, and now it's DevOps. So, build servers, and now it's DevOps. So, they they my my university was like, they they my my university was like, they they my my university was like, "Software engineering is a valid "Software engineering is a valid "Software engineering is a valid practice, and shipping is a thing that practice, and shipping is a thing that practice, and shipping is a thing that is different than computer science."

is different than computer science." is different than computer science." And I think that people need to just And I think that people need to just And I think that people need to just always, if there's a question, first always, if there's a question, first always, if there's a question, first principles. principles. principles. >> Yeah. >> Yeah. >> Yeah. >> Do we have a good birth a good build >> Do we have a good birth a good build >> Do we have a good birth a good build server? Uh talking to these giant server? Uh talking to these giant server? Uh talking to these giant companies at these executive briefing companies at these executive briefing companies at these executive briefing centers, and then finding out that their centers, and then finding out that their centers, and then finding out that their DevOps is a hot mess. DevOps is a hot mess. DevOps is a hot mess. >> Yeah. >> Yeah. >> Yeah. >> No amount of AI is going to save you. >> No amount of AI is going to save you. >> No amount of AI is going to save you. >> No. No, in fact, that is a trap. >> No. No, in fact, that is a trap. >> No. No, in fact, that is a trap. >> [laughter] >> [laughter] >> [laughter] >> It really is. And they're going to be >> It really is. And they're going to be >> It really is. And they're going to be absolutely screwed, and then they're absolutely screwed, and then they're absolutely screwed, and then they're going to be sad when their teams going to be sad when their teams going to be sad when their teams collapse. collapse. collapse. >> Yeah. >> Yeah. >> Yeah. >> Yeah. >> Yeah. >> Yeah. You have also been associated with test You have also been associated with test You have also been associated with test in production. in production. in production. >> Oh. >> Oh. >> Oh. >> Which some people have said is kind of >> Which some people have said is kind of >> Which some people have said is kind of reckless. Yeah, see, look, test in prod reckless. Yeah, see, look, test in prod reckless. Yeah, see, look, test in prod or live a lie. or live a lie. or live a lie. >> Test in prod. >> Test in prod. >> Test in prod. >> So, some people call that reckless, but >> So, some people call that reckless, but >> So, some people call that reckless, but these are the same people who might be these are the same people who might be these are the same people who might be using AI to just kind of yeet code using AI to just kind of yeet code using AI to just kind of yeet code directly into production. directly into production. directly into production. >> [snorts] >> [snorts] >> [snorts] >> So, >> So, >> So, >> as long as you have >> as long as you have >> as long as you have >> Yeah. What does it look like? >> Yeah. What does it look like? >> Yeah. What does it look like? >> production. It's only a The only >> production. It's only a The only >> production. It's only a The only question is whether or not you question is whether or not you question is whether or not you acknowledge that that's what you're acknowledge that that's what you're acknowledge that that's what you're doing, and build tooling to do it doing, and build tooling to do it doing, and build tooling to do it safely. safely. safely. There is no There is no replacement for There is no There is no replacement for There is no There is no replacement for reality. No matter how much you think reality. No matter how much you think reality. No matter how much you think you have you have you have tested or examined or validated, tested or examined or validated, tested or examined or validated, every single moment of intersection of every single moment of intersection of every single moment of intersection of software, infrastructure, deploy software, infrastructure, deploy software, infrastructure, deploy process, time, data, users, every single process, time, data, users, every single process, time, data, users, every single one of those is unique.

one of those is unique. one of those is unique. >> Okay. >> Okay. >> Okay. >> is no test for reality. There is There >> is no test for reality. There is There >> is no test for reality. There is There There is only practice runs, right? There is only practice runs, right? There is only practice runs, right? You're always going to be You're always You're always going to be You're always You're always going to be You're always going to be [laughter] tested by going to be [laughter] tested by going to be [laughter] tested by >> It's Schrödinger's production. >> It's Schrödinger's production. >> It's Schrödinger's production. >> It really is. >> It really is. >> It really is. >> You just have to open it and find out >> You just have to open it and find out >> You just have to open it and find out the cat's okay. the cat's okay. the cat's okay. >> And And none of this is an argument to >> And And none of this is an argument to >> And And none of this is an argument to not test, not test, not test, >> Mhm. >> Mhm. >> Mhm. >> but I feel like >> but I feel like >> but I feel like we all have limited attention cycles, we we all have limited attention cycles, we we all have limited attention cycles, we all have limited engineering cycles, all have limited engineering cycles, all have limited engineering cycles, even with, you know, even with AI. And I even with, you know, even with AI. And I even with, you know, even with AI. And I feel like most people get so wrapped feel like most people get so wrapped feel like most people get so wrapped around the axle of making sure it's around the axle of making sure it's around the axle of making sure it's perfect, that they're under investing in perfect, that they're under investing in perfect, that they're under investing in the other side of it, which is how am I the other side of it, which is how am I the other side of it, which is how am I going to find it, how am I going to going to find it, how am I going to going to find it, how am I going to trace it, how am I going to react to it, trace it, how am I going to react to it, trace it, how am I going to react to it, how am I going to fix it? how am I going to fix it? how am I going to fix it? >> Mhm. Yeah, that's a really great point. >> Mhm. Yeah, that's a really great point. >> Mhm. Yeah, that's a really great point. So, every deploy then is just a So, every deploy then is just a So, every deploy then is just a production experiment, whether you production experiment, whether you production experiment, whether you choose to admit it or not. choose to admit it or not. choose to admit it or not. >> And And the thing is that you don't have >> And And the thing is that you don't have >> And And the thing is that you don't have to do this dangerously. We have so many to do this dangerously. We have so many to do this dangerously. We have so many tools now for tools now for tools now for uh canaries or progressive delivery or uh canaries or progressive delivery or uh canaries or progressive delivery or feature flags or you're deploying the feature flags or you're deploying the feature flags or you're deploying the code and then turning it on for one code and then turning it on for one code and then turning it on for one person and then turn cranking up the person and then turn cranking up the person and then turn cranking up the ratio. Like nobody's saying you ratio. Like nobody's saying you ratio. Like nobody's saying you >> So few people do that stuff. So few >> So few people do that stuff. So few >> So few people do that stuff. So few people do that stuff.

people do that stuff. people do that stuff. >> Well, you know, there are lots of >> Well, you know, there are lots of >> Well, you know, there are lots of contexts where that would be contexts where that would be contexts where that would be over-engineering, but if it's not, you over-engineering, but if it's not, you over-engineering, but if it's not, you should do that. should do that. should do that. >> Yeah. Yeah. Feature flags are like >> Yeah. Yeah. Feature flags are like >> Yeah. Yeah. Feature flags are like it's so funny that like what's the most it's so funny that like what's the most it's so funny that like what's the most sophisticated, most powerful thing that sophisticated, most powerful thing that sophisticated, most powerful thing that I can do to really enable production and I can do to really enable production and I can do to really enable production and like experimentation? It's like a bool. like experimentation? It's like a bool. like experimentation? It's like a bool. You could put a bool. You could put a You could put a bool. You could put a You could put a bool. You could put a bool in there bool in there bool in there and just maybe like turn that feature on and just maybe like turn that feature on and just maybe like turn that feature on or off. And if it goes bad or off. And if it goes bad or off. And if it goes bad >> We react, baby. >> We react, baby. >> We react, baby. >> You know, like yeah, just throwing it >> You know, like yeah, just throwing it >> You know, like yeah, just throwing it out there, just spit spitballing, maybe out there, just spit spitballing, maybe out there, just spit spitballing, maybe a debug.writeline, a little printf and a a debug.writeline, a little printf and a a debug.writeline, a little printf and a boolean. boolean. boolean. >> Yeah. It's so funny though because >> Yeah. It's so funny though because >> Yeah. It's so funny though because >> it's it is >> it's it is >> it's it is >> a bool. >> a bool. >> a bool. >> the core of what we're doing. >> the core of what we're doing. >> the core of what we're doing. It's It's It's Yeah. Yeah. That's so funny. Yeah. Yeah. That's so funny. Yeah. Yeah. That's so funny. >> decision. Like the combination of >> decision. Like the combination of >> decision. Like the combination of feature flags and the kind of feature flags and the kind of feature flags and the kind of the kind of high cardinality the kind of high cardinality the kind of high cardinality observability data that Honeycomb does, observability data that Honeycomb does, observability data that Honeycomb does, like it's they're greater than the sum like it's they're greater than the sum like it's they're greater than the sum of their parts. I know a lot of people of their parts. I know a lot of people of their parts. I know a lot of people have used feature flags when all they have used feature flags when all they have used feature flags when all they have are like metrics and aggregates and have are like metrics and aggregates and have are like metrics and aggregates and log lines and and feature flags are log lines and and feature flags are log lines and and feature flags are still useful, but they're not like still useful, but they're not like still useful, but they're not like superpowers, but like it is a superpower superpowers, but like it is a superpower superpowers, but like it is a superpower when you can break down by one in a when you can break down by one in a when you can break down by one in a million apps and enable just that app million apps and enable just that app million apps and enable just that app and then compare it against the baseline and then compare it against the baseline and then compare it against the baseline of all the other apps and trace every of all the other apps and trace every of all the other apps and trace every request. Like it's just Yeah, you can request. Like it's just Yeah, you can request. Like it's just Yeah, you can test in prod then. It's just like why test in prod then. It's just like why test in prod then. It's just like why wouldn't you test in prod? It's the wouldn't you test in prod? It's the wouldn't you test in prod? It's the easiest way to test.

easiest way to test. easiest way to test. >> Yeah. >> Yeah. >> Yeah. All right. In the In the In the All right. In the In the In the All right. In the In the In the remaining couple of minutes, let me get remaining couple of minutes, let me get remaining couple of minutes, let me get a little bit rapid fire, but also I want a little bit rapid fire, but also I want a little bit rapid fire, but also I want to explore some of the spaces where you to explore some of the spaces where you to explore some of the spaces where you and I I think deeply agree because you and I I think deeply agree because you and I I think deeply agree because you have argued that not hiring juniors is a have argued that not hiring juniors is a have argued that not hiring juniors is a problem. If a company says we only hire problem. If a company says we only hire problem. If a company says we only hire seniors now because AI does the easy seniors now because AI does the easy seniors now because AI does the easy stuff. What do you say to that? stuff. What do you say to that? stuff. What do you say to that? >> all people away. >> all people away. >> all people away. >> Okay. >> Okay. >> Okay. You don't like that. Totally You don't like that. Totally You don't like that. Totally unacceptable. unacceptable. unacceptable. >> I mean, I am a person and I would like >> I mean, I am a person and I would like >> I mean, I am a person and I would like to have a job. So yeah, I don't think to have a job. So yeah, I don't think to have a job. So yeah, I don't think that's very smart. that's very smart. that's very smart. >> Good. Okay, so if an A Is an AI more >> Good. Okay, so if an A Is an AI more >> Good. Okay, so if an A Is an AI more like a junior engineer or an intern with like a junior engineer or an intern with like a junior engineer or an intern with bad judgment or should we not bad judgment or should we not bad judgment or should we not anthropomorphize it at all? What do we anthropomorphize it at all? What do we anthropomorphize it at all? What do we How do we think about an AI? How do we think about an AI? How do we think about an AI? >> I anthropomorphizing it because it's >> I anthropomorphizing it because it's >> I anthropomorphizing it because it's just just just I love the definition of consciousness I love the definition of consciousness I love the definition of consciousness that is there is something it is like to that is there is something it is like to that is there is something it is like to be that person. be that person. be that person. It's It's about like the way that we It's It's about like the way that we It's It's about like the way that we develop taste and judgment is through develop taste and judgment is through develop taste and judgment is through repeated exposure and experience and repeated exposure and experience and repeated exposure and experience and that is just it is categorically that is just it is categorically that is just it is categorically different from what we get from agents. different from what we get from agents. different from what we get from agents. >> Okay, so my next That's awesome because >> Okay, so my next That's awesome because >> Okay, so my next That's awesome because my next question was going to be how do my next question was going to be how do my next question was going to be how do we teach judgment when the visible we teach judgment when the visible we teach judgment when the visible artifact, in this case code, is so easy artifact, in this case code, is so easy artifact, in this case code, is so easy to produce? You got to read more code.

to produce? You got to read more code. to produce? You got to read more code. >> We create the conditions where people >> We create the conditions where people >> We create the conditions where people can fail and experiment and try things can fail and experiment and try things can fail and experiment and try things safely. safely. safely. I [clears throat] have a lot of faith in I [clears throat] have a lot of faith in I [clears throat] have a lot of faith in humans. There are a lot of problems with humans. There are a lot of problems with humans. There are a lot of problems with AI. Don't get me wrong. I am not one to AI. Don't get me wrong. I am not one to AI. Don't get me wrong. I am not one to wish any of them away and I suspect that wish any of them away and I suspect that wish any of them away and I suspect that a lot of the problems of how do junior a lot of the problems of how do junior a lot of the problems of how do junior developers learn these things is going developers learn these things is going developers learn these things is going to come from them, not from us. to come from them, not from us. to come from them, not from us. I believe they're learnable and I I believe they're learnable and I I believe they're learnable and I believe this is solvable, but I just I believe this is solvable, but I just I believe this is solvable, but I just I fundamentally believe that you decide to fundamentally believe that you decide to fundamentally believe that you decide to take a risk on people, you support them, take a risk on people, you support them, take a risk on people, you support them, and it pays off. and it pays off. and it pays off. >> Mhm. >> Mhm. >> Mhm. >> [snorts] >> [snorts] >> [snorts] >> How important I think it's hugely >> How important I think it's hugely >> How important I think it's hugely important. How important do you think important. How important do you think important. How important do you think like empathy is and how did being on like empathy is and how did being on like empathy is and how did being on call early teach you about empathy? call early teach you about empathy? call early teach you about empathy? >> I'm just kidding. I mean >> I'm just kidding. I mean >> I'm just kidding. I mean >> [laughter] >> [laughter] >> [laughter] >> It's >> It's >> It's How important is empathy is kind of like How important is empathy is kind of like How important is empathy is kind of like saying do you really need oxygen? saying do you really need oxygen? saying do you really need oxygen? >> Let me rephrase. How did being on call, >> Let me rephrase. How did being on call, >> Let me rephrase. How did being on call, being on prod, teach you about empathy? being on prod, teach you about empathy? being on prod, teach you about empathy? That's the real question. That's the real question. That's the real question. >> That is the thing that I love about >> That is the thing that I love about >> That is the thing that I love about being on call is how much it aligns you being on call is how much it aligns you being on call is how much it aligns you with the pain that your customers are with the pain that your customers are with the pain that your customers are going into. We were talking about going into. We were talking about going into. We were talking about feedback loops earlier.

feedback loops earlier. feedback loops earlier. And I have told many people that the way And I have told many people that the way And I have told many people that the way that they get their developers to care that they get their developers to care that they get their developers to care is by putting them on call because when is by putting them on call because when is by putting them on call because when you align when the pain is separated and you align when the pain is separated and you align when the pain is separated and you don't you're not a good developer, you don't you're not a good developer, you don't you're not a good developer, but when your pain is aligned with that but when your pain is aligned with that but when your pain is aligned with that of your customers, it it pain is of your customers, it it pain is of your customers, it it pain is nature's teacher. nature's teacher. nature's teacher. >> Yeah, it really is. Like it has to hurt >> Yeah, it really is. Like it has to hurt >> Yeah, it really is. Like it has to hurt a little bit and I was talking about a little bit and I was talking about a little bit and I was talking about this if you work out, your muscles need this if you work out, your muscles need this if you work out, your muscles need to hurt a little bit. If you are to hurt a little bit. If you are to hurt a little bit. If you are studying, your brain needs to hurt a studying, your brain needs to hurt a studying, your brain needs to hurt a little bit. Uh I'm a little bit sad that little bit. Uh I'm a little bit sad that little bit. Uh I'm a little bit sad that like long-form learning is now becoming like long-form learning is now becoming like long-form learning is now becoming Tik Toks, is now becoming Tik Toks, is now becoming Tik Toks, is now becoming snackable. snackable. snackable. >> People add friction. >> People add friction. >> People add friction. Every interaction with a person has Every interaction with a person has Every interaction with a person has friction and that's friction and that's friction and that's why it's valuable. why it's valuable. why it's valuable. >> Yeah, yeah, and that builds it it it >> Yeah, yeah, and that builds it it it >> Yeah, yeah, and that builds it it it that you put them into the forge and you that you put them into the forge and you that you put them into the forge and you turn them into something special, turn them into something special, turn them into something special, absolutely. absolutely. absolutely. >> Yeah. >> Yeah. >> Yeah. >> That idea of being in prod, but also >> That idea of being in prod, but also >> That idea of being in prod, but also letting the early in career person like letting the early in career person like letting the early in career person like I use the analogy of hold the scalpel. I use the analogy of hold the scalpel. I use the analogy of hold the scalpel. Like we're about to do heart surgery. Like we're about to do heart surgery. Like we're about to do heart surgery. Hey, have you ever you ever held a human Hey, have you ever you ever held a human Hey, have you ever you ever held a human heart? Here, you hold the scalpel. We're heart? Here, you hold the scalpel. We're heart? Here, you hold the scalpel. We're going to go into prod right now and going to go into prod right now and going to go into prod right now and let's do that.

let's do that. let's do that. >> Yes. >> Yes. >> Yes. >> Do you have like how do you treat early >> Do you have like how do you treat early >> Do you have like how do you treat early in career people in career people in career people at your company? Like do you have at your company? Like do you have at your company? Like do you have interns? Do you call them interns? What interns? Do you call them interns? What interns? Do you call them interns? What are they? are they? are they? >> had a few interns and we have a few >> had a few interns and we have a few >> had a few interns and we have a few early career folks and early career folks and early career folks and we try to make sure and we try to make sure and we try to make sure and never have just one. You don't want never have just one. You don't want never have just one. You don't want there to be one person that feels like there to be one person that feels like there to be one person that feels like they're you know, you need to hire in they're you know, you need to hire in they're you know, you need to hire in pairs. We're not a super big company. pairs. We're not a super big company. pairs. We're not a super big company. >> Sure. >> Sure. >> Sure. >> And >> And >> And watching them learn has been so watching them learn has been so watching them learn has been so interesting because everyone uses AI in interesting because everyone uses AI in interesting because everyone uses AI in a different way. But like a different way. But like a different way. But like >> [clears throat] >> [clears throat] >> [clears throat] >> most of them have just this running >> most of them have just this running >> most of them have just this running dialogue all day long with they have a dialogue all day long with they have a dialogue all day long with they have a 24/7 tutor personally devoted to them, 24/7 tutor personally devoted to them, 24/7 tutor personally devoted to them, you know, and and it helps them figure you know, and and it helps them figure you know, and and it helps them figure out how to ask interesting questions and out how to ask interesting questions and out how to ask interesting questions and interrupt their more senior counterpart interrupt their more senior counterpart interrupt their more senior counterpart and and and >> Yeah. >> Yeah. >> Yeah. >> they just they're >> they just they're >> they just they're people are people are great. people are people are great. people are people are great. >> Yeah, I'm I'm really looking forward to >> Yeah, I'm I'm really looking forward to >> Yeah, I'm I'm really looking forward to spending the last kind of thousand days spending the last kind of thousand days spending the last kind of thousand days of my career spending as much time as I of my career spending as much time as I of my career spending as much time as I can with early in career people and then can with early in career people and then can with early in career people and then going and teaching again at community going and teaching again at community going and teaching again at community college and college and college and >> just talking to Jay Genglebach from >> just talking to Jay Genglebach from >> just talking to Jay Genglebach from Vercel and he was saying, you know, Vercel and he was saying, you know, Vercel and he was saying, you know, he was expressing the same thing. He's he was expressing the same thing. He's he was expressing the same thing. He's like, man, I see so many more bugs that like, man, I see so many more bugs that like, man, I see so many more bugs that I can fix. I see so many more problems I can fix. I see so many more problems I can fix. I see so many more problems that I can I can't build all the things that I can I can't build all the things that I can I can't build all the things that I see and and I can't ask ages to that I see and and I can't ask ages to that I see and and I can't ask ages to go build those things or solve those go build those things or solve those go build those things or solve those things cuz they're too they're too big, things cuz they're too they're too big, things cuz they're too they're too big, they're too they're too they're too they're the right size for a more junior they're the right size for a more junior they're the right size for a more junior or mid-level engineer and I could have or mid-level engineer and I could have or mid-level engineer and I could have so much more impact if I just had so much more impact if I just had so much more impact if I just had people people people to do this and I was just like, I think to do this and I was just like, I think to do this and I was just like, I think that's the best pitch for junior

that's the best pitch for junior that's the best pitch for junior engineers that I've ever heard. I engineers that I've ever heard. I engineers that I've ever heard. I honestly do. honestly do. honestly do. >> They've got to have exposure. Well, the >> They've got to have exposure. Well, the >> They've got to have exposure. Well, the second edition of observability second edition of observability second edition of observability engineering is available as PDF, dead engineering is available as PDF, dead engineering is available as PDF, dead trees coming soon. It is fantastic trees coming soon. It is fantastic trees coming soon. It is fantastic apparently because it's 600 plus pages, apparently because it's 600 plus pages, apparently because it's 600 plus pages, you said. you said. you said. >> I feel like I shouldn't have said that. >> I feel like I shouldn't have said that. >> I feel like I shouldn't have said that. Now nobody's going to want to read Now nobody's going to want to read Now nobody's going to want to read [laughter] it. [laughter] it. [laughter] it. >> No, it's a whole new book. Hey, when I >> No, it's a whole new book. Hey, when I >> No, it's a whole new book. Hey, when I hear 600 pages, I hear value. hear 600 pages, I hear value. hear 600 pages, I hear value. >> No. >> No. >> No. >> And I think this is that is exciting to >> And I think this is that is exciting to >> And I think this is that is exciting to me. That is the hook. me. That is the hook. me. That is the hook. >> It is the War and Peace of technical >> It is the War and Peace of technical >> It is the War and Peace of technical books. books. books. >> There it is, the War and Peace of >> There it is, the War and Peace of >> There it is, the War and Peace of technical books technical books technical books and we're going to observability the and we're going to observability the and we're going to observability the heck out of it. So, heck out of it. So, heck out of it. So, >> [laughter] >> [laughter] >> [laughter] >> thank you so much for chatting with me >> thank you so much for chatting with me >> thank you so much for chatting with me today, Charity Majors. today, Charity Majors. today, Charity Majors. This has been another episode of This has been another episode of This has been another episode of Hanselminutes and we'll see you again Hanselminutes and we'll see you again Hanselminutes and we'll see you again next week.