The Power of Open Telemetry with Dr. Sally Wahba

hey friends we have some news from our hey friends we have some news from our sponsor text control they just released sponsor text control they just released sponsor text control they just released version 32 of their document processing version 32 of their document processing version 32 of their document processing Library which includes new core Library which includes new core Library which includes new core functionality such as document footnotes functionality such as document footnotes functionality such as document footnotes SVG export and more you can integrate SVG export and more you can integrate SVG export and more you can integrate document editing signing collaboration document editing signing collaboration document editing signing collaboration and PDF processing into your asp.net and PDF processing into your asp.net and PDF processing into your asp.net core and angular applications with TX core and angular applications with TX core and angular applications with TX text control these powerful libraries text control these powerful libraries text control these powerful libraries will let your developer teams focus on will let your developer teams focus on will let your developer teams focus on their core competencies while text their core competencies while text their core competencies while text control handles your digital document control handles your digital document control handles your digital document processing check out all the new featur processing check out all the new featur processing check out all the new featur feat and see the Technologies live in feat and see the Technologies live in feat and see the Technologies live in action by visiting the demo at demos. action by visiting the demo at demos. action by visiting the demo at demos. text control.com that's demos. text text control.com that's demos. text text control.com that's demos. text [Music] [Music] [Music] control.com hi I'm Scott Hansel and this control.com hi I'm Scott Hansel and this control.com hi I'm Scott Hansel and this is another episode of Hansel minutes is another episode of Hansel minutes is another episode of Hansel minutes today I'm talking with Dr Sally BBA she today I'm talking with Dr Sally BBA she today I'm talking with Dr Sally BBA she is a principal engineer at Splunk it's a is a principal engineer at Splunk it's a is a principal engineer at Splunk it's a PhD from Clemson how are you I'm doing PhD from Clemson how are you I'm doing PhD from Clemson how are you I'm doing well how about you I am very well and well how about you I am very well and well how about you I am very well and I'm excited to be chatting with you I'm excited to be chatting with you I'm excited to be chatting with you today I um you're just such a prolific today I um you're just such a prolific today I um you're just such a prolific writer and researcher and you've done so writer and researcher and you've done so writer and researcher and you've done so much in the space I I just love talking much in the space I I just love talking much in the space I I just love talking with people with as much experience as with people with as much experience as with people with as much experience as you have you've worked at net apppp you you have you've worked at net apppp you you have you've worked at net apppp you now work at Splunk but you also have a now work at Splunk but you also have a now work at Splunk but you also have a lot of interest in large distributed lot of interest in large distributed lot of interest in large distributed systems and particularly in the spaces systems and particularly in the spaces systems and particularly in the spaces of testing and debugging and I've been of testing and debugging and I've been of testing and debugging and I've been trying to understand open Telemetry in trying to understand open Telemetry in trying to understand open Telemetry in these large distributed systems and I

these large distributed systems and I these large distributed systems and I know that's a space that you have worked know that's a space that you have worked know that's a space that you have worked in um yeah so first thanks for having me in um yeah so first thanks for having me in um yeah so first thanks for having me it's it's pretty exciting to to be on it's it's pretty exciting to to be on it's it's pretty exciting to to be on your podcast I've been a fan for a while your podcast I've been a fan for a while your podcast I've been a fan for a while yeah so my journey with observability yeah so my journey with observability yeah so my journey with observability and open Telemetry started only two and open Telemetry started only two and open Telemetry started only two years ago when I joined Splunk before years ago when I joined Splunk before years ago when I joined Splunk before Splunk I was at net App working on uh Splunk I was at net App working on uh Splunk I was at net App working on uh data storage so we had our own operating data storage so we had our own operating data storage so we had our own operating system so observability was not really a system so observability was not really a system so observability was not really a thing so it's mostly monitoring and uh thing so it's mostly monitoring and uh thing so it's mostly monitoring and uh the traditional debugging things and the traditional debugging things and the traditional debugging things and looking at cores and it was very very looking at cores and it was very very looking at cores and it was very very different but my last two years at net different but my last two years at net different but my last two years at net app I started working more on kubernetes app I started working more on kubernetes app I started working more on kubernetes and that's when things have really and that's when things have really and that's when things have really changed and I started uh looking at like changed and I started uh looking at like changed and I started uh looking at like wow the debugging these things is hard wow the debugging these things is hard wow the debugging these things is hard correlating things is hard figuring out correlating things is hard figuring out correlating things is hard figuring out where the problem is is also hard and uh where the problem is is also hard and uh where the problem is is also hard and uh that kind of started planting a seed in that kind of started planting a seed in that kind of started planting a seed in my head I like well I want I want to get my head I like well I want I want to get my head I like well I want I want to get more experience in large distri system more experience in large distri system more experience in large distri system and also getting more into observability and also getting more into observability and also getting more into observability and so that's how I ended up Splunk and and so that's how I ended up Splunk and and so that's how I ended up Splunk and also the operating system and uh also the operating system and uh also the operating system and uh knowledge came in handy as well but you knowledge came in handy as well but you knowledge came in handy as well but you had also proposed a course on had also proposed a course on had also proposed a course on distributed computing and written a distributed computing and written a distributed computing and written a paper on that in as far back as 2007 so paper on that in as far back as 2007 so paper on that in as far back as 2007 so you thought about these these systems you thought about these these systems you thought about these these systems for quite a while yes so I used to work for quite a while yes so I used to work for quite a while yes so I used to work in uh sensor networks and uh the nature in uh sensor networks and uh the nature in uh sensor networks and uh the nature of it was very uh distributed but at um of it was very uh distributed but at um of it was very uh distributed but at um it was different so in terms of having it was different so in terms of having it was different so in terms of having these large beefy machines or VMS we had

these large beefy machines or VMS we had these large beefy machines or VMS we had these tiny uh sensors that are very these tiny uh sensors that are very these tiny uh sensors that are very resource constrained and they are very resource constrained and they are very resource constrained and they are very very distributed because you can just very distributed because you can just very distributed because you can just throw them in a field a whole bunch of throw them in a field a whole bunch of throw them in a field a whole bunch of them and they communicate together they them and they communicate together they them and they communicate together they built a mesh basically it's kind of like built a mesh basically it's kind of like built a mesh basically it's kind of like the underlying technology behind uh the underlying technology behind uh the underlying technology behind uh Internet of Things uh that's basically Internet of Things uh that's basically Internet of Things uh that's basically what it was and then when I was working what it was and then when I was working what it was and then when I was working on it uh we thought well this stuff on it uh we thought well this stuff on it uh we thought well this stuff isn't really taught at the time when I isn't really taught at the time when I isn't really taught at the time when I was in CED school you don't have a whole was in CED school you don't have a whole was in CED school you don't have a whole lot of classes available and so that's lot of classes available and so that's lot of classes available and so that's when we uh propose a class to teach when we uh propose a class to teach when we uh propose a class to teach these techniques and these Concepts is these techniques and these Concepts is these techniques and these Concepts is there a lot of parallels between there a lot of parallels between there a lot of parallels between distributed nodes where that node might distributed nodes where that node might distributed nodes where that node might be a kubernetes node or it might be a be a kubernetes node or it might be a be a kubernetes node or it might be a tiny device that just happens to have a tiny device that just happens to have a tiny device that just happens to have a lot of latency because it's far away in lot of latency because it's far away in lot of latency because it's far away in a farmer field somewhere that's a great a farmer field somewhere that's a great a farmer field somewhere that's a great uh one to think about so these sensors uh one to think about so these sensors uh one to think about so these sensors when I was working on them they're they when I was working on them they're they when I was working on them they're they can come and go and so if you think can come and go and so if you think can come and go and so if you think about kubernetes pots for example uh if about kubernetes pots for example uh if about kubernetes pots for example uh if you're scaling you can have some you you're scaling you can have some you you're scaling you can have some you think about joining the network and not think about joining the network and not think about joining the network and not so you can have more pods fewer pods so you can have more pods fewer pods so you can have more pods fewer pods some of them are ephemeral some of them some of them are ephemeral some of them some of them are ephemeral some of them might disappear depending on your might disappear depending on your might disappear depending on your autoscaler for example these were autoscaler for example these were autoscaler for example these were similar the also each one is its own similar the also each one is its own similar the also each one is its own thing with it's doing its own thing not thing with it's doing its own thing not thing with it's doing its own thing not knowing about the other neighbors either knowing about the other neighbors either knowing about the other neighbors either so that was also another similarity the so that was also another similarity the so that was also another similarity the focus on open Telemetry and the concept

focus on open Telemetry and the concept focus on open Telemetry and the concept of just observability in software of just observability in software of just observability in software engineering is understanding the state engineering is understanding the state engineering is understanding the state of a system by observing its external of a system by observing its external of a system by observing its external outputs I grew up in a time when we were outputs I grew up in a time when we were outputs I grew up in a time when we were always try to observe internal state by always try to observe internal state by always try to observe internal state by attaching a debugger and I'm noticing attaching a debugger and I'm noticing attaching a debugger and I'm noticing amongst my peers that there are those amongst my peers that there are those amongst my peers that there are those that do basically print F debugging and that do basically print F debugging and that do basically print F debugging and then there are those who attach a then there are those who attach a then there are those who attach a debugger and I I'm kind of wondering in debugger and I I'm kind of wondering in debugger and I I'm kind of wondering in my mind when that started and i' I I've my mind when that started and i' I I've my mind when that started and i' I I've noticed that a lot of kubernetes and noticed that a lot of kubernetes and noticed that a lot of kubernetes and distributed people don't really do a lot distributed people don't really do a lot distributed people don't really do a lot of attach a debugger type work they of attach a debugger type work they of attach a debugger type work they really love their logs they love their really love their logs they love their really love their logs they love their observability I not really have a observability I not really have a observability I not really have a question there but I've just I noticed question there but I've just I noticed question there but I've just I noticed that about us yeah I mean I've had my that about us yeah I mean I've had my that about us yeah I mean I've had my fair share of both and uh the attaching fair share of both and uh the attaching fair share of both and uh the attaching at the buger will take you so far uh at the buger will take you so far uh at the buger will take you so far uh especially if if it's something if especially if if it's something if especially if if it's something if you're talking about microservices at you're talking about microservices at you're talking about microservices at some point you're not going to be like some point you're not going to be like some point you're not going to be like your request is not even going to be your request is not even going to be your request is not even going to be handled by your pod and so there is no handled by your pod and so there is no handled by your pod and so there is no debugger to attached to because it's now debugger to attached to because it's now debugger to attached to because it's now out of your your process right so you out of your your process right so you out of your your process right so you start looking at like well maybe we can start looking at like well maybe we can start looking at like well maybe we can have logs and maybe we can correlate have logs and maybe we can correlate have logs and maybe we can correlate them and we can kind of weave a story of them and we can kind of weave a story of them and we can kind of weave a story of of my request for example so I had my of my request for example so I had my of my request for example so I had my fair share of attaching debuggers but uh fair share of attaching debuggers but uh fair share of attaching debuggers but uh I still do as I'm working on like I still do as I'm working on like I still do as I'm working on like smaller pieces but yeah there are like smaller pieces but yeah there are like smaller pieces but yeah there are like places and tools for each thing well you places and tools for each thing well you places and tools for each thing well you you mentioned in your journey to you mentioned in your journey to you mentioned in your journey to observability that you started working observability that you started working observability that you started working on a monolith and you were using both

on a monolith and you were using both on a monolith and you were using both debugging techniques but also monitoring debugging techniques but also monitoring debugging techniques but also monitoring at some point that's kind of when you at some point that's kind of when you at some point that's kind of when you came to the religion of like wait a came to the religion of like wait a came to the religion of like wait a second Telemetry is really important and second Telemetry is really important and second Telemetry is really important and instrumenting your applications it's not instrumenting your applications it's not instrumenting your applications it's not a small investment but it's crucial yeah a small investment but it's crucial yeah a small investment but it's crucial yeah it's definitely a crucial and there is it's definitely a crucial and there is it's definitely a crucial and there is this misconception that monitoring and this misconception that monitoring and this misconception that monitoring and Telemetry is observability and it's not Telemetry is observability and it's not Telemetry is observability and it's not so if you think about Telemetry and so if you think about Telemetry and so if you think about Telemetry and monitoring get more reactive and so you monitoring get more reactive and so you monitoring get more reactive and so you already understand so much of the system already understand so much of the system already understand so much of the system that you can figure things out after the that you can figure things out after the that you can figure things out after the fact or you can get involved after an fact or you can get involved after an fact or you can get involved after an issue happens but the goal of issue happens but the goal of issue happens but the goal of observability is to be able to reason observability is to be able to reason observability is to be able to reason about your system and like you mentioned about your system and like you mentioned about your system and like you mentioned looking at its internal State and you looking at its internal State and you looking at its internal State and you want to be able to ask questions that at want to be able to ask questions that at want to be able to ask questions that at the time you're instrumenting your code the time you're instrumenting your code the time you're instrumenting your code you don't even think about that becomes you don't even think about that becomes you don't even think about that becomes a little tricky of what questions may I a little tricky of what questions may I a little tricky of what questions may I think about in the future and do I have think about in the future and do I have think about in the future and do I have instrumentation to help me do that uh instrumentation to help me do that uh instrumentation to help me do that uh and that is lacking with monitoring and and that is lacking with monitoring and and that is lacking with monitoring and Telemetry but it is something that Telemetry but it is something that Telemetry but it is something that observability would afford you that is a observability would afford you that is a observability would afford you that is a really great way of putting that and my really great way of putting that and my really great way of putting that and my my brain is trying to process what you my brain is trying to process what you my brain is trying to process what you just said that's so interesting because just said that's so interesting because just said that's so interesting because you're right if something goes wrong and you're right if something goes wrong and you're right if something goes wrong and you can answer why it went wrong without you can answer why it went wrong without you can answer why it went wrong without having to do additional instrumentation having to do additional instrumentation having to do additional instrumentation then you've succeeded is what I'm then you've succeeded is what I'm then you've succeeded is what I'm hearing you say yeah like I I try to hearing you say yeah like I I try to hearing you say yeah like I I try to think about monitoring versus think about monitoring versus think about monitoring versus observability of one is reactive and one

observability of one is reactive and one observability of one is reactive and one is proactive more so monitoring the site is proactive more so monitoring the site is proactive more so monitoring the site has gone down or which one is proactive has gone down or which one is proactive has gone down or which one is proactive which want explain that again so which want explain that again so which want explain that again so monitoring you're detecting the problem monitoring you're detecting the problem monitoring you're detecting the problem when it happen so the problem has when it happen so the problem has when it happen so the problem has already happened okay but observability already happened okay but observability already happened okay but observability you're putting in all the you're putting in all the you're putting in all the instrumentation yes it will definitely instrumentation yes it will definitely instrumentation yes it will definitely help you in figuring out like reactive help you in figuring out like reactive help you in figuring out like reactive what issue happened but you can also what issue happened but you can also what issue happened but you can also you're kind of investing in your future you're kind of investing in your future you're kind of investing in your future self of being able to answer questions self of being able to answer questions self of being able to answer questions that you may not even know existed at that you may not even know existed at that you may not even know existed at the time okay I'm understanding now okay the time okay I'm understanding now okay the time okay I'm understanding now okay so monitoring you know you can collect so monitoring you know you can collect so monitoring you know you can collect you can analyze data you can look at you can analyze data you can look at you can analyze data you can look at these predefined set of metrics and logs these predefined set of metrics and logs these predefined set of metrics and logs yes react to something going wrong the yes react to something going wrong the yes react to something going wrong the dis is at 100% whatever yep but with dis is at 100% whatever yep but with dis is at 100% whatever yep but with observability it's also that you're observability it's also that you're observability it's also that you're understanding you're truly grocking understanding you're truly grocking understanding you're truly grocking what's happening in that system's what's happening in that system's what's happening in that system's internal State yeah and then can look internal State yeah and then can look internal State yeah and then can look you don't have to do any additional you don't have to do any additional you don't have to do any additional testing no additional coding yes that's testing no additional coding yes that's testing no additional coding yes that's the goal that is the okay if if you've the goal that is the okay if if you've the goal that is the okay if if you've succeeded that is the trick right yes succeeded that is the trick right yes succeeded that is the trick right yes that is the goal yes okay how often does that is the goal yes okay how often does that is the goal yes okay how often does that happen though because you've called that happen though because you've called that happen though because you've called out before what a huge investment that out before what a huge investment that out before what a huge investment that is is this like back in the day remember is is this like back in the day remember is is this like back in the day remember test driven development where they were test driven development where they were test driven development where they were like test all the time write tests all like test all the time write tests all like test all the time write tests all the time like you can't have enough the time like you can't have enough the time like you can't have enough tests is the same same thing happening tests is the same same thing happening tests is the same same thing happening in observability should I just in observability should I just in observability should I just instrument everything every line I like instrument everything every line I like instrument everything every line I like when what's too much Telemetry and when what's too much Telemetry and when what's too much Telemetry and what's the right amount of instrument

what's the right amount of instrument what's the right amount of instrument yeah that's one of those things that is yeah that's one of those things that is yeah that's one of those things that is uh an art more than a science I think uh an art more than a science I think uh an art more than a science I think with same thing with unit test like you with same thing with unit test like you with same thing with unit test like you can at some point you have to put a line can at some point you have to put a line can at some point you have to put a line and say we're done here and we need to and say we're done here and we need to and say we're done here and we need to ship uh and then you can always like ship uh and then you can always like ship uh and then you can always like iterate and that's one thing for example iterate and that's one thing for example iterate and that's one thing for example with net app when we worked in an with net app when we worked in an with net app when we worked in an operating system if you think about it operating system if you think about it operating system if you think about it how often do you upgrade your operating how often do you upgrade your operating how often do you upgrade your operating system at home not every day MH but if system at home not every day MH but if system at home not every day MH but if you're hosting sass then yeah you might you're hosting sass then yeah you might you're hosting sass then yeah you might be pushing your code to production be pushing your code to production be pushing your code to production multiple times a day or multiple times a multiple times a day or multiple times a multiple times a day or multiple times a week these are kind of different things week these are kind of different things week these are kind of different things to think about it makes me think about to think about it makes me think about to think about it makes me think about when a boss of mine many many years ago when a boss of mine many many years ago when a boss of mine many many years ago learned about code coverage and they learned about code coverage and they learned about code coverage and they just picked a number and they were like just picked a number and they were like just picked a number and they were like 95% C and I was like yeah you know and 95% C and I was like yeah you know and 95% C and I was like yeah you know and and that was just an arbitrary number and that was just an arbitrary number and that was just an arbitrary number that they wanted and then we we had a that they wanted and then we we had a that they wanted and then we we had a build server and they said you have to build server and they said you have to build server and they said you have to have this much code coverage but we have this much code coverage but we have this much code coverage but we weren't thinking about the hot path we weren't thinking about the hot path we weren't thinking about the hot path we weren't thinking about the business weren't thinking about the business weren't thinking about the business problem it was just a number that they problem it was just a number that they problem it was just a number that they made up yes it sounds like that would be made up yes it sounds like that would be made up yes it sounds like that would be an unhealthy way to think about an unhealthy way to think about an unhealthy way to think about Telemetry uh yes like you definitely Telemetry uh yes like you definitely Telemetry uh yes like you definitely want to help your future self but at the want to help your future self but at the want to help your future self but at the same time there is cost right so how same time there is cost right so how same time there is cost right so how much is too much is also like am I if much is too much is also like am I if much is too much is also like am I if I'm going to instrument every single I'm going to instrument every single I'm going to instrument every single thing then then I'm going to be thing then then I'm going to be thing then then I'm going to be inundated with data then it's going to inundated with data then it's going to inundated with data then it's going to be counter intuitive really and as I be counter intuitive really and as I be counter intuitive really and as I start querying my data then maybe all start querying my data then maybe all start querying my data then maybe all the traces I'm sending that's that's too the traces I'm sending that's that's too the traces I'm sending that's that's too much it's one of those things of are you much it's one of those things of are you much it's one of those things of are you creating a hay stack where you can't

creating a hay stack where you can't creating a hay stack where you can't find the needle or H how big is your hay find the needle or H how big is your hay find the needle or H how big is your hay stack that is a really good analogy I stack that is a really good analogy I stack that is a really good analogy I love that that's exactly what it is it's love that that's exactly what it is it's love that that's exactly what it is it's like you know I have no hay at all yeah like you know I have no hay at all yeah like you know I have no hay at all yeah where am I going to hide my needles where am I going to hide my needles where am I going to hide my needles yes that is great okay so then I want to yes that is great okay so then I want to yes that is great okay so then I want to understand where open Telemetry comes in understand where open Telemetry comes in understand where open Telemetry comes in because in the past when I worked on because in the past when I worked on because in the past when I worked on large distributed systems we were all large distributed systems we were all large distributed systems we were all about the log files and every process it about the log files and every process it about the log files and every process it was a big distributed banking system had was a big distributed banking system had was a big distributed banking system had a had a separate uh log file and it was a had a separate uh log file and it was a had a separate uh log file and it was like process like process like process name- sometim stamp. log and they were name- sometim stamp. log and they were name- sometim stamp. log and they were spread all over creation and we thought spread all over creation and we thought spread all over creation and we thought we were so clever and then when we were so clever and then when we were so clever and then when everything went bad you jump into the everything went bad you jump into the everything went bad you jump into the log files and try to come up with some log files and try to come up with some log files and try to come up with some kind of a correlation ID yes so if you kind of a correlation ID yes so if you kind of a correlation ID yes so if you think about their different think about their different think about their different observability vendors in industry and so observability vendors in industry and so observability vendors in industry and so if you want to instrument your code you if you want to instrument your code you if you want to instrument your code you can use their proprietary libraries can use their proprietary libraries can use their proprietary libraries which like we said we already which like we said we already which like we said we already established that's an investment to established that's an investment to established that's an investment to instrument your code well your CTO or instrument your code well your CTO or instrument your code well your CTO or CEO or whatever said well we're not CEO or whatever said well we're not CEO or whatever said well we're not going to use that vendor anymore we're going to use that vendor anymore we're going to use that vendor anymore we're going to use another vendor so now you going to use another vendor so now you going to use another vendor so now you as an engineer have to go back and as an engineer have to go back and as an engineer have to go back and instrument your code with this new B instrument your code with this new B instrument your code with this new B spoke library of the new vendor that's a spoke library of the new vendor that's a spoke library of the new vendor that's a lot you're you're wasting resources uh lot you're you're wasting resources uh lot you're you're wasting resources uh you're wasting time you're wasting you're wasting time you're wasting you're wasting time you're wasting engineering effort where you can be engineering effort where you can be engineering effort where you can be doing something else and so open doing something else and so open doing something else and so open Telemetry comes in where it says we're Telemetry comes in where it says we're Telemetry comes in where it says we're going to establish a standard you're going to establish a standard you're going to establish a standard you're going to instrument your code once using

going to instrument your code once using going to instrument your code once using open Telemetry and then if you want to open Telemetry and then if you want to open Telemetry and then if you want to use another vendor then you don't have use another vendor then you don't have use another vendor then you don't have to reinstrumentation and and modify things but that initial and and modify things but that initial investment is a one-time thing um which investment is a one-time thing um which investment is a one-time thing um which is is great MH and The Importance of is is great MH and The Importance of is is great MH and The Importance of Being vendor neutral is important Being vendor neutral is important Being vendor neutral is important certainly within open source but also certainly within open source but also certainly within open source but also any observability framework you're not any observability framework you're not any observability framework you're not going to want to switch but then I going to want to switch but then I going to want to switch but then I presume that a vendor could then build presume that a vendor could then build presume that a vendor could then build on top of it with their own herbs and on top of it with their own herbs and on top of it with their own herbs and spices absolutely yes yes and that's spices absolutely yes yes and that's spices absolutely yes yes and that's basically what would make one vendor not basically what would make one vendor not basically what would make one vendor not necessarily better but more use cases to necessarily better but more use cases to necessarily better but more use cases to yeah to different customers yes okay so yeah to different customers yes okay so yeah to different customers yes okay so then open Telemetry was an incubator then open Telemetry was an incubator then open Telemetry was an incubator within the cloud native Computing within the cloud native Computing within the cloud native Computing foundation and really kind of like foundation and really kind of like foundation and really kind of like popped once all of these distributed popped once all of these distributed popped once all of these distributed systems particularly kubernetes of of systems particularly kubernetes of of systems particularly kubernetes of of note started happening people were like note started happening people were like note started happening people were like wow this was just too much and the wow this was just too much and the wow this was just too much and the amount of data was huge like my our log amount of data was huge like my our log amount of data was huge like my our log files were you know megabytes but the files were you know megabytes but the files were you know megabytes but the data and then metrics and then traces data and then metrics and then traces data and then metrics and then traces yes if you think about it especially yes if you think about it especially yes if you think about it especially with like with microservices and with like with microservices and with like with microservices and kubernetes when I was at netup for kubernetes when I was at netup for kubernetes when I was at netup for example and there's a problem I can example and there's a problem I can example and there's a problem I can maybe SSH into a machine and see what maybe SSH into a machine and see what maybe SSH into a machine and see what the problem is or I have a cord that I the problem is or I have a cord that I the problem is or I have a cord that I can look at uh and I can look at the can look at uh and I can look at the can look at uh and I can look at the different things and with kubernetes different things and with kubernetes different things and with kubernetes that's not necessarily the case so that's not necessarily the case so that's not necessarily the case so unless you're harvesting the logs and unless you're harvesting the logs and unless you're harvesting the logs and pushing them somewhere your pod is gone

pushing them somewhere your pod is gone pushing them somewhere your pod is gone and everything with it is gone uh and so and everything with it is gone uh and so and everything with it is gone uh and so so observability tries to help you kind so observability tries to help you kind so observability tries to help you kind of Reason about all these different of Reason about all these different of Reason about all these different things yeah that's a good point and it things yeah that's a good point and it things yeah that's a good point and it also calls out that a lot of these also calls out that a lot of these also calls out that a lot of these compute units of work are ephemeral in compute units of work are ephemeral in compute units of work are ephemeral in nature yes not just a node but a a nature yes not just a node but a a nature yes not just a node but a a function some kind of a function as a function some kind of a function as a function some kind of a function as a service situation totally yes I agree service situation totally yes I agree service situation totally yes I agree okay so then when I was doing you know okay so then when I was doing you know okay so then when I was doing you know test driven development or doing cicd we test driven development or doing cicd we test driven development or doing cicd we used to call them build servers there used to call them build servers there used to call them build servers there was always the knee-jerk reaction to go was always the knee-jerk reaction to go was always the knee-jerk reaction to go in whatever all the things test driven in whatever all the things test driven in whatever all the things test driven development all the things or build development all the things or build development all the things or build server all the things is it important to server all the things is it important to server all the things is it important to have a culture around observability or have a culture around observability or have a culture around observability or should I just go in and imp Implement should I just go in and imp Implement should I just go in and imp Implement open Telemetry and then just run run run open Telemetry and then just run run run open Telemetry and then just run run run and get everyone to do all the things and get everyone to do all the things and get everyone to do all the things it's not just run run run like if you it's not just run run run like if you it's not just run run run like if you think about unit tests like unit test think about unit tests like unit test think about unit tests like unit test has a a great value right so it's not has a a great value right so it's not has a a great value right so it's not just that gives you a little bit of just that gives you a little bit of just that gives you a little bit of confidence of hey I have some code confidence of hey I have some code confidence of hey I have some code coverage can I have a level of coverage can I have a level of coverage can I have a level of confidence of my code is doing what it's confidence of my code is doing what it's confidence of my code is doing what it's supposed to do but also it helps you supposed to do but also it helps you supposed to do but also it helps you reason about your code so as you're reason about your code so as you're reason about your code so as you're writing unit test like H am I using the writing unit test like H am I using the writing unit test like H am I using the right abstractions because I have if I right abstractions because I have if I right abstractions because I have if I have my code tightly coupled then unit have my code tightly coupled then unit have my code tightly coupled then unit testing it is not going to be easy and testing it is not going to be easy and testing it is not going to be easy and so so that helps you to take a step back so so that helps you to take a step back so so that helps you to take a step back and think about your design decisions and think about your design decisions and think about your design decisions and I feel OB observability is the same and I feel OB observability is the same and I feel OB observability is the same way but again there's the balance of you way but again there's the balance of you way but again there's the balance of you need to ship at a certain point because need to ship at a certain point because need to ship at a certain point because businesses are there to make money and businesses are there to make money and businesses are there to make money and they need to make products available and they need to make products available and they need to make products available and customers have requirements there's

customers have requirements there's customers have requirements there's definitely that balance but you want to definitely that balance but you want to definitely that balance but you want to create the culture of observability create the culture of observability create the culture of observability helps you have a better product overall helps you have a better product overall helps you have a better product overall because even if like outages happen bugs because even if like outages happen bugs because even if like outages happen bugs exist because we don't write perfect exist because we don't write perfect exist because we don't write perfect code uh so there are a lot of things code uh so there are a lot of things code uh so there are a lot of things that are out of our control as well um that are out of our control as well um that are out of our control as well um especially when you have managed especially when you have managed especially when you have managed services and and hosted Services your services and and hosted Services your services and and hosted Services your goal is to be able to solve problems goal is to be able to solve problems goal is to be able to solve problems quickly so observability helps you with quickly so observability helps you with quickly so observability helps you with that so having a culture around hey that so having a culture around hey that so having a culture around hey we're we're not just going to ship but we're we're not just going to ship but we're we're not just going to ship but we're also want to be able to have a we're also want to be able to have a we're also want to be able to have a solid product that is reliable that we solid product that is reliable that we solid product that is reliable that we can recover quickly when issues happen can recover quickly when issues happen can recover quickly when issues happen because they will happen so you want to because they will happen so you want to because they will happen so you want to build that culture it's not just about build that culture it's not just about build that culture it's not just about let's just crank out code and push to let's just crank out code and push to let's just crank out code and push to production mhm so then it sounds like production mhm so then it sounds like production mhm so then it sounds like having the right data and collecting the having the right data and collecting the having the right data and collecting the right data is important collecting the right data is important collecting the right data is important collecting the right data but also uh it's not just the right data but also uh it's not just the right data but also uh it's not just the data not just even like within data not just even like within data not just even like within engineering you also want your engineering you also want your engineering you also want your management chain to be aware because management chain to be aware because management chain to be aware because that is time that you're investing not that is time that you're investing not that is time that you're investing not writing the actual code that's going to writing the actual code that's going to writing the actual code that's going to give customer the whatever Behavior they give customer the whatever Behavior they give customer the whatever Behavior they want right the idea of getting this into want right the idea of getting this into want right the idea of getting this into the brain of the team and having the the brain of the team and having the the brain of the team and having the team's culture such that it is natural team's culture such that it is natural team's culture such that it is natural to start thinking about observability to start thinking about observability to start thinking about observability then of course plugging in something then of course plugging in something then of course plugging in something like open Telemetry also makes me wonder like open Telemetry also makes me wonder like open Telemetry also makes me wonder about privacy and how uh you could about privacy and how uh you could about privacy and how uh you could accidentally leak I don't know credit

accidentally leak I don't know credit accidentally leak I don't know credit cards or social security numbers or cards or social security numbers or cards or social security numbers or passwords and things like that sure yeah passwords and things like that sure yeah passwords and things like that sure yeah yeah like there you can go through data yeah like there you can go through data yeah like there you can go through data cleansing phases and like one thing when cleansing phases and like one thing when cleansing phases and like one thing when we're instrumenting for examp example we're instrumenting for examp example we're instrumenting for examp example we're very mindful of what data we can we're very mindful of what data we can we're very mindful of what data we can put like you're you're not going to put put like you're you're not going to put put like you're you're not going to put any personal identifying information in any personal identifying information in any personal identifying information in that it's more for your system debugging that it's more for your system debugging that it's more for your system debugging so even when we were doing monitoring so even when we were doing monitoring so even when we were doing monitoring for net app for example we were also not for net app for example we were also not for net app for example we were also not having anything that is customer data having anything that is customer data having anything that is customer data being stored anywhere you're definitely being stored anywhere you're definitely being stored anywhere you're definitely being mindful of that and if there's being mindful of that and if there's being mindful of that and if there's anything that might be the case we also anything that might be the case we also anything that might be the case we also catch that and like code reviews for catch that and like code reviews for catch that and like code reviews for example that's also one other thing to example that's also one other thing to example that's also one other thing to make sure you have a culture of of good make sure you have a culture of of good make sure you have a culture of of good code reviews as well that's like code reviews as well that's like code reviews as well that's like unrelated observability just general unrelated observability just general unrelated observability just general development as well I'm trying to think development as well I'm trying to think development as well I'm trying to think about how you would sell this to about how you would sell this to about how you would sell this to management though because I feel like management though because I feel like management though because I feel like we're already trying to convince them we're already trying to convince them we're already trying to convince them that we need to do more testing we're that we need to do more testing we're that we need to do more testing we're already trying to convince them we need already trying to convince them we need already trying to convince them we need more devops all of them are are going to more devops all of them are are going to more devops all of them are are going to look at something like observability as look at something like observability as look at something like observability as yet another thing that doesn't help my yet another thing that doesn't help my yet another thing that doesn't help my business these are you know business these are you know business these are you know cross-cutting concerns yes that's an cross-cutting concerns yes that's an cross-cutting concerns yes that's an excellent question so it's easier to excellent question so it's easier to excellent question so it's easier to show disasters as a way to say if we had show disasters as a way to say if we had show disasters as a way to say if we had observability that would have helped us observability that would have helped us observability that would have helped us to recover earlier or faster or uh we to recover earlier or faster or uh we to recover earlier or faster or uh we wouldn't have run into this issue to wouldn't have run into this issue to wouldn't have run into this issue to begin with because we would have caught begin with because we would have caught begin with because we would have caught it before it even reached a customer it before it even reached a customer it before it even reached a customer these are things that you can argue

these are things that you can argue these are things that you can argue about it definitely helps to have people about it definitely helps to have people about it definitely helps to have people higher up who believe in that as well higher up who believe in that as well higher up who believe in that as well and that's something that's really good and that's something that's really good and that's something that's really good at at splan is we are in observability at at splan is we are in observability at at splan is we are in observability so we do food our own products that's a so we do food our own products that's a so we do food our own products that's a great point I presume that if anything great point I presume that if anything great point I presume that if anything is really observable it would be Splunk is really observable it would be Splunk is really observable it would be Splunk products it would be deeply observable products it would be deeply observable products it would be deeply observable because yeah so then that probably because yeah so then that probably because yeah so then that probably changes culture that changes team changes culture that changes team changes culture that changes team culture uh and if you're on call the culture uh and if you're on call the culture uh and if you're on call the culture of being on call how does how do culture of being on call how does how do culture of being on call how does how do that change your view about the right that change your view about the right that change your view about the right way to build software wow yes at netup way to build software wow yes at netup way to build software wow yes at netup like if you're a software engineer you like if you're a software engineer you like if you're a software engineer you were very very sheltered from production were very very sheltered from production were very very sheltered from production because we don't really have production because we don't really have production because we don't really have production because we give our customers an because we give our customers an because we give our customers an operating system and they have their own operating system and they have their own operating system and they have their own production so we didn't have a culture production so we didn't have a culture production so we didn't have a culture of un call at all yes I would I would of un call at all yes I would I would of un call at all yes I would I would deal with customer issues but these are deal with customer issues but these are deal with customer issues but these are more a regular hey support contacted me more a regular hey support contacted me more a regular hey support contacted me and said there's an issue that I need to and said there's an issue that I need to and said there's an issue that I need to look at and so when I mov to Splunk now look at and so when I mov to Splunk now look at and so when I mov to Splunk now we have Dev on production we push to we have Dev on production we push to we have Dev on production we push to production we support our service we production we support our service we production we support our service we have our uptime and all all our slos are have our uptime and all all our slos are have our uptime and all all our slos are met and so that made me think about how met and so that made me think about how met and so that made me think about how can I not just build something that is can I not just build something that is can I not just build something that is more resilient but how do I help myself more resilient but how do I help myself more resilient but how do I help myself when I wake up at 2:00 in the morning when I wake up at 2:00 in the morning when I wake up at 2:00 in the morning because I got paged how can I first because I got paged how can I first because I got paged how can I first minimize that and because I really like minimize that and because I really like minimize that and because I really like my uninterrupted sleep and how do I go my uninterrupted sleep and how do I go my uninterrupted sleep and how do I go back to bed quickly so to do that I want back to bed quickly so to do that I want back to bed quickly so to do that I want to make sure that again I help my future

to make sure that again I help my future to make sure that again I help my future s so it's more out of selfishness that s so it's more out of selfishness that s so it's more out of selfishness that wait I need to make sure that I'm wait I need to make sure that I'm wait I need to make sure that I'm building something solid and I need to building something solid and I need to building something solid and I need to invest in all the different ecosystem invest in all the different ecosystem invest in all the different ecosystem that is going to help me solve any issue that is going to help me solve any issue that is going to help me solve any issue quickly I think that thinking about your quickly I think that thinking about your quickly I think that thinking about your future self is like a really great future self is like a really great future self is like a really great quality that any software engineer could quality that any software engineer could quality that any software engineer could have right like who are the comments for have right like who are the comments for have right like who are the comments for oh that's for future me yes right who's oh that's for future me yes right who's oh that's for future me yes right who's the test for future me who's the the test for future me who's the the test for future me who's the observability for future me totally yes observability for future me totally yes observability for future me totally yes and one of the things that like before and one of the things that like before and one of the things that like before plun I was very detached from like for plun I was very detached from like for plun I was very detached from like for example the support we I didn't see what example the support we I didn't see what example the support we I didn't see what this was doing in production but now this was doing in production but now this was doing in production but now that I'm on call I see exactly what's that I'm on call I see exactly what's that I'm on call I see exactly what's going going going on people hate being on call though but on people hate being on call though but on people hate being on call though but what you're saying is that being on call what you're saying is that being on call what you're saying is that being on call makes you a better engineer it did I makes you a better engineer it did I makes you a better engineer it did I have so much empathy now to my my fellow have so much empathy now to my my fellow have so much empathy now to my my fellow I love that you said that I was going to I love that you said that I was going to I love that you said that I was going to say empathy because you know I've talked say empathy because you know I've talked say empathy because you know I've talked about this before on the show and about this before on the show and about this before on the show and sometimes people think I'm sometimes people think I'm sometimes people think I'm squishy empathy is just completely squishy empathy is just completely squishy empathy is just completely underrated within software engineering underrated within software engineering underrated within software engineering because support person is going to because support person is going to because support person is going to appreciate the work that you put into appreciate the work that you put into appreciate the work that you put into instrumenting your code yes absolutely instrumenting your code yes absolutely instrumenting your code yes absolutely yes do you believe in like Dev owning yes do you believe in like Dev owning yes do you believe in like Dev owning service like that the dev should have a service like that the dev should have a service like that the dev should have a we always talk about full stack we always talk about full stack we always talk about full stack developers and I don't think honestly developers and I don't think honestly developers and I don't think honestly full stack development is a thing full stack development is a thing full stack development is a thing because the stack is too deep there too because the stack is too deep there too because the stack is too deep there too much yeah that said though I love taking much yeah that said though I love taking much yeah that said though I love taking devs and having them live on a project

devs and having them live on a project devs and having them live on a project like you're going to carry the pager for like you're going to carry the pager for like you're going to carry the pager for a month and you're going to work on that a month and you're going to work on that a month and you're going to work on that and now you're going to do Ops and now and now you're going to do Ops and now and now you're going to do Ops and now you're going to do Sr because it builds you're going to do Sr because it builds you're going to do Sr because it builds empathy as they roam around the business empathy as they roam around the business empathy as they roam around the business being exposed to production is I think being exposed to production is I think being exposed to production is I think is is very important you still want to is is very important you still want to is is very important you still want to have expertise and something though I have expertise and something though I have expertise and something though I will run my service I will own my will run my service I will own my will run my service I will own my service but I'm not going to be as good service but I'm not going to be as good service but I'm not going to be as good as an Sr who's running some kka cluster as an Sr who's running some kka cluster as an Sr who's running some kka cluster right of course like you said the stack right of course like you said the stack right of course like you said the stack is too deep it's not just the stack of is too deep it's not just the stack of is too deep it's not just the stack of software it's also all the different software it's also all the different software it's also all the different pieces that make a product successful pieces that make a product successful pieces that make a product successful okay I I usually use the analogy of okay I I usually use the analogy of okay I I usually use the analogy of being of a Swiss army knife yep cuz it's being of a Swiss army knife yep cuz it's being of a Swiss army knife yep cuz it's like it's just a lousy tool it's a tiny like it's just a lousy tool it's a tiny like it's just a lousy tool it's a tiny pair of scissors it's not a very good pair of scissors it's not a very good pair of scissors it's not a very good cork screw yeah that's me as an engineer cork screw yeah that's me as an engineer cork screw yeah that's me as an engineer right like I I have a lot of little right like I I have a lot of little right like I I have a lot of little skills that I'm not very good at but you skills that I'm not very good at but you skills that I'm not very good at but you have to be a t-shaped developer you have have to be a t-shaped developer you have have to be a t-shaped developer you have to have deep in something yes yeah I to have deep in something yes yeah I to have deep in something yes yeah I agree absolutely so then open Telemetry agree absolutely so then open Telemetry agree absolutely so then open Telemetry as a project and observability as a as a project and observability as a as a project and observability as a concept has you know kind of lit the concept has you know kind of lit the concept has you know kind of lit the distributed engineering world on fire it distributed engineering world on fire it distributed engineering world on fire it must be easy then to hire observability must be easy then to hire observability must be easy then to hire observability experts and Engineers right no not at experts and Engineers right no not at experts and Engineers right no not at all like if you think about it iian all like if you think about it iian all like if you think about it iian spunk two years ago and one of the spunk two years ago and one of the spunk two years ago and one of the things when I was interviewing I said things when I was interviewing I said things when I was interviewing I said hey I did so monitoring just by being at hey I did so monitoring just by being at hey I did so monitoring just by being at NP but I don't know much about NP but I don't know much about NP but I don't know much about absorbability and here I am with years absorbability and here I am with years absorbability and here I am with years and years of experience and so they were and years of experience and so they were and years of experience and so they were like yeah you learn it there aren't many like yeah you learn it there aren't many like yeah you learn it there aren't many people around like if you think about it people around like if you think about it people around like if you think about it observability is a new thing open

observability is a new thing open observability is a new thing open Telemetry is new that stuff isn't really Telemetry is new that stuff isn't really Telemetry is new that stuff isn't really taught at school much uh or boot camp if taught at school much uh or boot camp if taught at school much uh or boot camp if you're yourself working as a hobby is you're yourself working as a hobby is you're yourself working as a hobby is not something that many people do side not something that many people do side not something that many people do side quests in either right that's a really quests in either right that's a really quests in either right that's a really great point you know I I know that my great point you know I I know that my great point you know I I know that my blog is not a kubernetes system but it blog is not a kubernetes system but it blog is not a kubernetes system but it does run in in the cloud and it has for does run in in the cloud and it has for does run in in the cloud and it has for years and it does run in a container as years and it does run in a container as years and it does run in a container as does the podcast and it's small but as does the podcast and it's small but as does the podcast and it's small but as the single employee of Hanselman kids little league make a website and kids little league make a website and try to keep it up for 24 hours and you try to keep it up for 24 hours and you try to keep it up for 24 hours and you will learn so much but you're right will learn so much but you're right will learn so much but you're right people don't have those side quests people don't have those side quests people don't have those side quests enough I like that I like keeping my enough I like that I like keeping my enough I like that I like keeping my site I've actually stepped out of movie site I've actually stepped out of movie site I've actually stepped out of movie theaters where I've got the alert of theaters where I've got the alert of theaters where I've got the alert of like from pingdom of like oh the site's like from pingdom of like oh the site's like from pingdom of like oh the site's down how do I get my blog back up from down how do I get my blog back up from down how do I get my blog back up from my phone or you know I'm sshing into my phone or you know I'm sshing into my phone or you know I'm sshing into production oh W that's production oh W that's production oh W that's awesome I just it gives you it gives you awesome I just it gives you it gives you awesome I just it gives you it gives you empathy it gives you perspective it empathy it gives you perspective it empathy it gives you perspective it gives you a lot yeah I would be a little gives you a lot yeah I would be a little gives you a lot yeah I would be a little sad if someone had a 10 or 20 year sad if someone had a 10 or 20 year sad if someone had a 10 or 20 year career but they never got to touch career but they never got to touch career but they never got to touch production yes I agree but then at the production yes I agree but then at the production yes I agree but then at the same time it's not everyone can do same time it's not everyone can do same time it's not everyone can do everything and yeah I it's a challenge I everything and yeah I it's a challenge I everything and yeah I it's a challenge I don't want to gatekeep you know yeah don't want to gatekeep you know yeah don't want to gatekeep you know yeah yeah I appreciate what you're saying yeah I appreciate what you're saying yeah I appreciate what you're saying like the thing that's challenging of like the thing that's challenging of like the thing that's challenging of course about having a podcast is that if course about having a podcast is that if course about having a podcast is that if you make a declarative statement any one you make a declarative statement any one you make a declarative statement any one of them taken out of context can become of them taken out of context can become of them taken out of context can become a little a little wonderful uh thing a little a little wonderful uh thing a little a little wonderful uh thing that someone tweets and then you're in

that someone tweets and then you're in that someone tweets and then you're in trouble yes you're that a person of the trouble yes you're that a person of the trouble yes you're that a person of the day yeah exactly but I guess I'm saying day yeah exactly but I guess I'm saying day yeah exactly but I guess I'm saying that like one of the things that I've that like one of the things that I've that like one of the things that I've asked advocated for and I'm interested asked advocated for and I'm interested asked advocated for and I'm interested in your opinion is know what level you in your opinion is know what level you in your opinion is know what level you operate at and then occasionally go one operate at and then occasionally go one operate at and then occasionally go one level below just to remind yourself what level below just to remind yourself what level below just to remind yourself what that feels like know change a tire yes that feels like know change a tire yes that feels like know change a tire yes and go okay I don't like changing tires and go okay I don't like changing tires and go okay I don't like changing tires I'm gonna hire somebody to change a tire I'm gonna hire somebody to change a tire I'm gonna hire somebody to change a tire absolutely yes yes but at the same time absolutely yes yes but at the same time absolutely yes yes but at the same time if you get stranded at the side of the if you get stranded at the side of the if you get stranded at the side of the road you know how to change a tire that road you know how to change a tire that road you know how to change a tire that exact so so then the question is for me exact so so then the question is for me exact so so then the question is for me how much do I need to know about how much do I need to know about how much do I need to know about observability and open Telemetry so that observability and open Telemetry so that observability and open Telemetry so that it is there to help me but I may not it is there to help me but I may not it is there to help me but I may not become an expert who can you know become an expert who can you know become an expert who can you know implement it into an organization myself implement it into an organization myself implement it into an organization myself yeah you can always start small right yeah you can always start small right yeah you can always start small right like you don't have to do everything and like you don't have to do everything and like you don't have to do everything and well that's a great Point could I I well that's a great Point could I I well that's a great Point could I I could go and make my podcast my tiny could go and make my podcast my tiny could go and make my podcast my tiny little thing my little five-page podcast little thing my little five-page podcast little thing my little five-page podcast observable yes absolutely and then maybe observable yes absolutely and then maybe observable yes absolutely and then maybe would that be a reasonable way for folks would that be a reasonable way for folks would that be a reasonable way for folks that are listening to then go and share that are listening to then go and share that are listening to then go and share their Journey with maybe their company their Journey with maybe their company their Journey with maybe their company and go to their boss and say hey you and go to their boss and say hey you and go to their boss and say hey you know I've been toying with observability know I've been toying with observability know I've been toying with observability and I think we should implement it here and I think we should implement it here and I think we should implement it here at the company yeah absolutely and there at the company yeah absolutely and there at the company yeah absolutely and there like there are really good documentation like there are really good documentation like there are really good documentation out there for open Telemetry and how to out there for open Telemetry and how to out there for open Telemetry and how to instrument your code and there's also if instrument your code and there's also if instrument your code and there's also if you want something small there is auto you want something small there is auto you want something small there is auto instrumentation which will take you so instrumentation which will take you so instrumentation which will take you so far but you can do a zero config Auto far but you can do a zero config Auto far but you can do a zero config Auto instrumentation for the languages that instrumentation for the languages that instrumentation for the languages that we use and that can be a good start to we use and that can be a good start to we use and that can be a good start to just kind of get your feet wet really just kind of get your feet wet really just kind of get your feet wet really and that you think that that that's a and that you think that that that's a and that you think that that that's a good thing like I know I could generate good thing like I know I could generate good thing like I know I could generate unit tests as well but I always wonder unit tests as well but I always wonder unit tests as well but I always wonder to myself how far autogeneration could to myself how far autogeneration could to myself how far autogeneration could take me yeah I mean it would take you so

take me yeah I mean it would take you so take me yeah I mean it would take you so far but then at the same time you still far but then at the same time you still far but then at the same time you still have your own logic that you want to have your own logic that you want to have your own logic that you want to instrument but it's definitely like you instrument but it's definitely like you instrument but it's definitely like you said how do I start that would be a said how do I start that would be a said how do I start that would be a great place to start yeah that's really great place to start yeah that's really great place to start yeah that's really good advice I I really appreciate you good advice I I really appreciate you good advice I I really appreciate you taking the time to get me up to speed taking the time to get me up to speed taking the time to get me up to speed and uh help me who I'm quite a bit and uh help me who I'm quite a bit and uh help me who I'm quite a bit earlier than you are in my journey to earlier than you are in my journey to earlier than you are in my journey to observability but I appreciate the work observability but I appreciate the work observability but I appreciate the work that you do and for spending time with that you do and for spending time with that you do and for spending time with me today yeah thank you for having me me today yeah thank you for having me me today yeah thank you for having me it's been fun chatting with you today it's been fun chatting with you today it's been fun chatting with you today all right we've been chatting with Dr s all right we've been chatting with Dr s all right we've been chatting with Dr s waha and you can check her out on social waha and you can check her out on social waha and you can check her out on social media and you can also feel free to go media and you can also feel free to go media and you can also feel free to go learn about open Telemetry at open learn about open Telemetry at open learn about open Telemetry at open Telemetry doio and uh if you feel like Telemetry doio and uh if you feel like Telemetry doio and uh if you feel like it look at the Splunk cloudplatform at it look at the Splunk cloudplatform at it look at the Splunk cloudplatform at splunk.com this has been another episode splunk.com this has been another episode splunk.com this has been another episode of Hansel minut and we'll see you again of Hansel minut and we'll see you again of Hansel minut and we'll see you again next next next [Music]