From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize

Well, thank thank you all. Um, let me Well, thank thank you all. Um, let me get set up here. So, not just the the get set up here. So, not just the the get set up here. So, not just the the founder of Arise, but but I tend to founder of Arise, but but I tend to founder of Arise, but but I tend to build an incredible amount of stuff. Um, build an incredible amount of stuff. Um, build an incredible amount of stuff. Um, let's see if we get this going here. let's see if we get this going here. let's see if we get this going here. Oh, sorry. One more second. Um, so not Oh, sorry. One more second. Um, so not Oh, sorry. One more second. Um, so not just a founder here, but but also a just a founder here, but but also a just a founder here, but but also a builder and I do my best to um uh to to builder and I do my best to um uh to to builder and I do my best to um uh to to to build agents assistance. Um, we have to build agents assistance. Um, we have to build agents assistance. Um, we have an agent in product. an agent in product. an agent in product. We have an agent in product called Alex We have an agent in product called Alex We have an agent in product called Alex and uh and a lot of I think a lot of my and uh and a lot of I think a lot of my and uh and a lot of I think a lot of my experience has come from actually um experience has come from actually um experience has come from actually um trying to make the stuff work and work trying to make the stuff work and work trying to make the stuff work and work well. Our first version of our our own well. Our first version of our our own well. Our first version of our our own agent frankly sucked. Uh it was many agent frankly sucked. Uh it was many agent frankly sucked. Uh it was many years ago uh probably two years ago when years ago uh probably two years ago when years ago uh probably two years ago when the first in the space to do it. Um and the first in the space to do it. Um and the first in the space to do it. Um and a lot of what we have built uh has come a lot of what we have built uh has come a lot of what we have built uh has come out of that our own experience in out of that our own experience in out of that our own experience in building building this agent and and building building this agent and and building building this agent and and signal is kind of our our next signal is kind of our our next signal is kind of our our next generation of this which is trying to generation of this which is trying to generation of this which is trying to automate a bunch of things which we do automate a bunch of things which we do automate a bunch of things which we do every day uh and build it into products every day uh and build it into products every day uh and build it into products that that people can use. Um so I'm that that people can use. Um so I'm that that people can use. Um so I'm going to try to I'm going to go through going to try to I'm going to go through going to try to I'm going to go through this this materials here. I'll try to go this this materials here. I'll try to go this this materials here. I'll try to go fast and try to show you a lot of fast and try to show you a lot of fast and try to show you a lot of product too. I'm a product person. Um, product too. I'm a product person. Um, product too. I'm a product person. Um, so so so if you've built a startup before, um, if you've built a startup before, um, if you've built a startup before, um, you you've experienced this, your you you've experienced this, your you you've experienced this, your platform's down, it's it's late at night platform's down, it's it's late at night platform's down, it's it's late at night and and you and you want to go fix it.

and and you and you want to go fix it. and and you and you want to go fix it. Um, and and really the the it takes a Um, and and really the the it takes a Um, and and really the the it takes a lot of energy to go do that. And we're lot of energy to go do that. And we're lot of energy to go do that. And we're going to talk about like the automation going to talk about like the automation going to talk about like the automation we've built a little bit and and what we've built a little bit and and what we've built a little bit and and what what the future looks like and and I what the future looks like and and I what the future looks like and and I truly believe um that the future of the truly believe um that the future of the truly believe um that the future of the the observability space is is actually the observability space is is actually the observability space is is actually changing massively right now. Why is changing massively right now. Why is changing massively right now. Why is that? Well, observability used to be for that? Well, observability used to be for that? Well, observability used to be for humans. Used to be a UI you click, a humans. Used to be a UI you click, a humans. Used to be a UI you click, a graph you click, something you look at. graph you click, something you look at. graph you click, something you look at. Um and and today it's I would argue it's Um and and today it's I would argue it's Um and and today it's I would argue it's a lot of 2.0 which is like this a lot of 2.0 which is like this a lot of 2.0 which is like this combination of coding agent. Those of combination of coding agent. Those of combination of coding agent. Those of you who built skills skills for um you who built skills skills for um you who built skills skills for um Pyroscope, Google Cloud or or whatnot, Pyroscope, Google Cloud or or whatnot, Pyroscope, Google Cloud or or whatnot, that these these skills help you with that these these skills help you with that these these skills help you with your your human debugging these systems. your your human debugging these systems. your your human debugging these systems. Um and and really telemetry is like this Um and and really telemetry is like this Um and and really telemetry is like this smoke uh thrown off of your system that smoke uh thrown off of your system that smoke uh thrown off of your system that can allow these agents to go make fixes. can allow these agents to go make fixes. can allow these agents to go make fixes. It tells you what path in the code it It tells you what path in the code it It tells you what path in the code it took. Without that, you're guessing and took. Without that, you're guessing and took. Without that, you're guessing and there's a million paths it could have there's a million paths it could have there's a million paths it could have taken. the the data thrown off by your taken. the the data thrown off by your taken. the the data thrown off by your system allows um allows you to to go go system allows um allows you to to go go system allows um allows you to to go go use agents to go debug your software.

use agents to go debug your software. use agents to go debug your software. Evals add another layer to this. Um but Evals add another layer to this. Um but Evals add another layer to this. Um but really what we're at here is is how do I really what we're at here is is how do I really what we're at here is is how do I build systems that autonomously fix build systems that autonomously fix build systems that autonomously fix themselves? Really that that is what themselves? Really that that is what themselves? Really that that is what we're after. Both both AI agents, I put we're after. Both both AI agents, I put we're after. Both both AI agents, I put AI into my my my system. How do I have AI into my my my system. How do I have AI into my my my system. How do I have this thing just improve itself? and and this thing just improve itself? and and this thing just improve itself? and and today we're kind of in the 2.0 which is today we're kind of in the 2.0 which is today we're kind of in the 2.0 which is a human making fixes and reviewing a human making fixes and reviewing a human making fixes and reviewing things. Um but there's a future we're things. Um but there's a future we're things. Um but there's a future we're all driving towards and throwing off all driving towards and throwing off all driving towards and throwing off traces, throwing off logs, throwing off traces, throwing off logs, throwing off traces, throwing off logs, throwing off way more than you normally would and way more than you normally would and way more than you normally would and having agents run at this for a having agents run at this for a having agents run at this for a continuous loop is where we're going. continuous loop is where we're going. continuous loop is where we're going. You can build at agent speed, but today You can build at agent speed, but today You can build at agent speed, but today you can't improve your systems really at you can't improve your systems really at you can't improve your systems really at this agent speed. So those of us feel this agent speed. So those of us feel this agent speed. So those of us feel this this kind of governor happening this this kind of governor happening this this kind of governor happening within our our our our products. um and within our our our our products. um and within our our our our products. um and and the bottleneck is actually not the and the bottleneck is actually not the and the bottleneck is actually not the fix anymore. So those of us who've used fix anymore. So those of us who've used fix anymore. So those of us who've used these systems and and use used um coding these systems and and use used um coding these systems and and use used um coding agents with with skills, the the the agents with with skills, the the the agents with with skills, the the the bottlenecks, a lot of the the confidence bottlenecks, a lot of the the confidence bottlenecks, a lot of the the confidence in in do I have it right? You know, a in in do I have it right? You know, a in in do I have it right? You know, a lot of this is is about is this fix the lot of this is is about is this fix the lot of this is is about is this fix the right one to push? Um and and so these right one to push? Um and and so these right one to push? Um and and so these are kind of the challenges here. And are kind of the challenges here. And are kind of the challenges here. And then how do you how do you build this then how do you how do you build this then how do you how do you build this loop in a way that just moves faster? Um loop in a way that just moves faster? Um loop in a way that just moves faster? Um and and a little bit of the way we we've and and a little bit of the way we we've and and a little bit of the way we we've kind of come to do it and we do it in kind of come to do it and we do it in kind of come to do it and we do it in our system is we've kind of inverted our system is we've kind of inverted our system is we've kind of inverted this this loop which is like a human you this this loop which is like a human you this this loop which is like a human you know looks at things and an agent uh know looks at things and an agent uh know looks at things and an agent uh fixes it to a person now can wake up

fixes it to a person now can wake up fixes it to a person now can wake up with with an idea of the issues based with with an idea of the issues based with with an idea of the issues based upon the errors occurred in their upon the errors occurred in their upon the errors occurred in their system. So so the agent is actually you system. So so the agent is actually you system. So so the agent is actually you know maybe it's not a a fix itself but know maybe it's not a a fix itself but know maybe it's not a a fix itself but it's putting up an issue. It's looking it's putting up an issue. It's looking it's putting up an issue. It's looking at the data before a human even looks at at the data before a human even looks at at the data before a human even looks at it. Um and and what you move from there it. Um and and what you move from there it. Um and and what you move from there is is kind of humans grabbing tickets to is is kind of humans grabbing tickets to is is kind of humans grabbing tickets to to having some amount of evidence um to having some amount of evidence um to having some amount of evidence um some deep evidence relative to whatever some deep evidence relative to whatever some deep evidence relative to whatever you're looking at already sitting in you're looking at already sitting in you're looking at already sitting in front of you by the time you actually front of you by the time you actually front of you by the time you actually even look at it. even look at it. even look at it. And and human review is kind of one And and human review is kind of one And and human review is kind of one thing but but a lot of times maybe thing but but a lot of times maybe thing but but a lot of times maybe you're driving this little investigation you're driving this little investigation you're driving this little investigation a bit from where it started. So that's a bit from where it started. So that's a bit from where it started. So that's the reality of where we are today is the reality of where we are today is the reality of where we are today is there's still maybe it's not human there's still maybe it's not human there's still maybe it's not human reviewing but human driving the the step reviewing but human driving the the step reviewing but human driving the the step two and three. Um but but this is kind two and three. Um but but this is kind two and three. Um but but this is kind of what what we view the loop as. And of what what we view the loop as. And of what what we view the loop as. And really what it is is there's you know really what it is is there's you know really what it is is there's you know there's an event that occurs that you're there's an event that occurs that you're there's an event that occurs that you're kind of kicking things off on or you're kind of kicking things off on or you're kind of kicking things off on or you're looking at periodically. Um and then looking at periodically. Um and then looking at periodically. Um and then there's some context around that which there's some context around that which there's some context around that which is really driven by skills. Um, I guess is really driven by skills. Um, I guess is really driven by skills. Um, I guess a question for all of you. Who's created a question for all of you. Who's created a question for all of you. Who's created skills in this room? Who's created a skills in this room? Who's created a skills in this room? Who's created a skill that that that interfaces to an skill that that that interfaces to an skill that that that interfaces to an observability platform?

observability platform? observability platform? Okay, handful. Okay, cool. Awesome. Um, Okay, handful. Okay, cool. Awesome. Um, Okay, handful. Okay, cool. Awesome. Um, so the magic of of of skills that that so the magic of of of skills that that so the magic of of of skills that that that connect to observability platforms that connect to observability platforms that connect to observability platforms um is it can gather the context. The um is it can gather the context. The um is it can gather the context. The agent can decide what it needs, what it agent can decide what it needs, what it agent can decide what it needs, what it needs to look at um to to start to needs to look at um to to start to needs to look at um to to start to troubleshoot what you have there. Um, troubleshoot what you have there. Um, troubleshoot what you have there. Um, and then there's idea of triggers which and then there's idea of triggers which and then there's idea of triggers which are like periodic and and um and uh and are like periodic and and um and uh and are like periodic and and um and uh and event based. And so the future event based. And so the future event based. And so the future observability actually looks a lot more observability actually looks a lot more observability actually looks a lot more like this than it does clicking around like this than it does clicking around like this than it does clicking around graphana UI. graphana UI. graphana UI. So first off evidence well normally So first off evidence well normally So first off evidence well normally these like or are or what do you start these like or are or what do you start these like or are or what do you start with what do you look at? Uh traces are with what do you look at? Uh traces are with what do you look at? Uh traces are are pretty nice logs as well. uh but but are pretty nice logs as well. uh but but are pretty nice logs as well. uh but but you know most of the AI systems these you know most of the AI systems these you know most of the AI systems these days have like traces at the core of of days have like traces at the core of of days have like traces at the core of of the agent framework. So, so you kind of the agent framework. So, so you kind of the agent framework. So, so you kind of start with with looking at traces and start with with looking at traces and start with with looking at traces and this is this could be periodic you know this is this could be periodic you know this is this could be periodic you know every five minutes this could be based every five minutes this could be based every five minutes this could be based upon an event an error and normally upon an event an error and normally upon an event an error and normally there's some combinations of these which there's some combinations of these which there's some combinations of these which is um you know some some like uh context is um you know some some like uh context is um you know some some like uh context and log you know context and skills used and log you know context and skills used and log you know context and skills used to put together logs maybe there's the to put together logs maybe there's the to put together logs maybe there's the repo uh you want kind of a combination repo uh you want kind of a combination repo uh you want kind of a combination of all this together um to understand of all this together um to understand of all this together um to understand what to go fix the repo tells you the what to go fix the repo tells you the what to go fix the repo tells you the code path that you know the you know all code path that you know the you know all code path that you know the you know all tells you everything that's there, the tells you everything that's there, the tells you everything that's there, the the production logs or traces that the the production logs or traces that the the production logs or traces that the agent pulls down. Um, normally our agent pulls down. Um, normally our agent pulls down. Um, normally our skills actually pull pull little temp skills actually pull pull little temp skills actually pull pull little temp files down into the the repo. Um, so files down into the the repo. Um, so files down into the the repo. Um, so that you kind of have this this this that you kind of have this this this that you kind of have this this this idea of what actually happened, what the

idea of what actually happened, what the idea of what actually happened, what the code is there enough and and all that code is there enough and and all that code is there enough and and all that together to put up a fix. Um, so it's together to put up a fix. Um, so it's together to put up a fix. Um, so it's this combination of the right data and this combination of the right data and this combination of the right data and file format in the repo along with your file format in the repo along with your file format in the repo along with your code in the repo. That's kind of the code in the repo. That's kind of the code in the repo. That's kind of the magic of this skills which are magic of this skills which are magic of this skills which are composable for the agent to go actually composable for the agent to go actually composable for the agent to go actually put a fix. Um, and a lot of this, some put a fix. Um, and a lot of this, some put a fix. Um, and a lot of this, some of you, a lot of you probably do this of you, a lot of you probably do this of you, a lot of you probably do this locally today. You you run this locally. locally today. You you run this locally. locally today. You you run this locally. You have an agent that you you kick up, You have an agent that you you kick up, You have an agent that you you kick up, maybe you're spinning up, but it's on maybe you're spinning up, but it's on maybe you're spinning up, but it's on your laptop. And I think we all feel your laptop. And I think we all feel your laptop. And I think we all feel this this this move from from this this this this move from from this this this this move from from this laptop um to to maybe to to basically laptop um to to maybe to to basically laptop um to to maybe to to basically sandboxes. Um sandboxes. Um sandboxes. Um and and really the sandbox is this this and and really the sandbox is this this and and really the sandbox is this this running environment where um based upon running environment where um based upon running environment where um based upon an event or a periodic you know a an event or a periodic you know a an event or a periodic you know a periodic event you can kick this thing periodic event you can kick this thing periodic event you can kick this thing off and it does the same thing you were off and it does the same thing you were off and it does the same thing you were doing locally get it working locally doing locally get it working locally doing locally get it working locally first locally on your laptop and then first locally on your laptop and then first locally on your laptop and then event based based upon the observability event based based upon the observability event based based upon the observability platforms like ourselves. Um you can platforms like ourselves. Um you can platforms like ourselves. Um you can trigger these on a schedule or or kind trigger these on a schedule or or kind trigger these on a schedule or or kind of you know every every error that comes of you know every every error that comes of you know every every error that comes up. Um, and generally, you know, you up. Um, and generally, you know, you up. Um, and generally, you know, you generally it's kind of put putting the generally it's kind of put putting the generally it's kind of put putting the loop together to do this and and um, and loop together to do this and and um, and loop together to do this and and um, and I want to kind of give you one example.

I want to kind of give you one example. I want to kind of give you one example. So, this is Alex, our agent. This is a a So, this is Alex, our agent. This is a a So, this is Alex, our agent. This is a a real example. It's a very simple one. real example. It's a very simple one. real example. It's a very simple one. And then I'm going to show you what it And then I'm going to show you what it And then I'm going to show you what it looks like in product. Um, this what we looks like in product. Um, this what we looks like in product. Um, this what we use every day. Um, but this is just an use every day. Um, but this is just an use every day. Um, but this is just an example where um, we had a stream example where um, we had a stream example where um, we had a stream canceled event. So um so Alex is is canceled event. So um so Alex is is canceled event. So um so Alex is is basically um Alex is is basically our basically um Alex is is basically our basically um Alex is is basically our our inproduct assistant. Um to-do update our inproduct assistant. Um to-do update our inproduct assistant. Um to-do update is is a uh is a is a way of of managing is is a uh is a is a way of of managing is is a uh is a is a way of of managing kind of its its task list. Um and it was kind of its its task list. Um and it was kind of its its task list. Um and it was trying to you know I'll walk you through trying to you know I'll walk you through trying to you know I'll walk you through the the error in a second but basically the the error in a second but basically the the error in a second but basically um it's calling a bunch of these two um it's calling a bunch of these two um it's calling a bunch of these two to-do updates and kind of um errors out to-do updates and kind of um errors out to-do updates and kind of um errors out and and so for us it's it's you know how and and so for us it's it's you know how and and so for us it's it's you know how do I put the data together um to debug do I put the data together um to debug do I put the data together um to debug this? how do we do it automatically and this? how do we do it automatically and this? how do we do it automatically and and signal is just something that's and signal is just something that's and signal is just something that's running in the background for us that's running in the background for us that's running in the background for us that's putting up like issues relative to these putting up like issues relative to these putting up like issues relative to these things. Um this was uh a kind of one or things. Um this was uh a kind of one or things. Um this was uh a kind of one or two line fix that it comes up with. two line fix that it comes up with. two line fix that it comes up with. These are these are ideal but a lot of These are these are ideal but a lot of These are these are ideal but a lot of times the fixes are bigger. Um and and times the fixes are bigger. Um and and times the fixes are bigger. Um and and the bigger it is the more likely a the bigger it is the more likely a the bigger it is the more likely a human's involved in kind of like human's involved in kind of like human's involved in kind of like spearheading it over the line. But again spearheading it over the line. But again spearheading it over the line. But again it's about that that cold start. Can I it's about that that cold start. Can I it's about that that cold start. Can I start with like all this information on start with like all this information on start with like all this information on the issue and guide it the rest of the the issue and guide it the rest of the the issue and guide it the rest of the way is kind of where we are right now.

way is kind of where we are right now. way is kind of where we are right now. Um, and for us, your job kind of moves Um, and for us, your job kind of moves Um, and for us, your job kind of moves from responder to reviewer. Um, and and from responder to reviewer. Um, and and from responder to reviewer. Um, and and and the view is like traces and evals and the view is like traces and evals and the view is like traces and evals don't go away in any way, shape, or don't go away in any way, shape, or don't go away in any way, shape, or form. They're just they're they're a key form. They're just they're they're a key form. They're just they're they're a key part of the loop. Now, you're going to part of the loop. Now, you're going to part of the loop. Now, you're going to trace 10 times more. You're going to log trace 10 times more. You're going to log trace 10 times more. You're going to log 10 times more because that helps you 10 times more because that helps you 10 times more because that helps you know what path your software took. know what path your software took. know what path your software took. Before, you wouldn't do that because Before, you wouldn't do that because Before, you wouldn't do that because because humans can't dig through all the because humans can't dig through all the because humans can't dig through all the logs. It's just noise. But by logging logs. It's just noise. But by logging logs. It's just noise. But by logging and tracing more of your like is it and tracing more of your like is it and tracing more of your like is it every inch of your software? Maybe in every inch of your software? Maybe in every inch of your software? Maybe in some places. Um by logging and tracing some places. Um by logging and tracing some places. Um by logging and tracing orders and orders of magnitude more than orders and orders of magnitude more than orders and orders of magnitude more than we do today, we can actually create we do today, we can actually create we do today, we can actually create these continuous loops that know what these continuous loops that know what these continuous loops that know what path was taking your software and and path was taking your software and and path was taking your software and and and actually have it fix itself. So this and actually have it fix itself. So this and actually have it fix itself. So this is kind of my vision for where I think is kind of my vision for where I think is kind of my vision for where I think things are going. um in a way and and things are going. um in a way and and things are going. um in a way and and for us I'll show you signal in a second for us I'll show you signal in a second for us I'll show you signal in a second and you'll see all these I mean I feel and you'll see all these I mean I feel and you'll see all these I mean I feel like there's there's this think of this like there's there's this think of this like there's there's this think of this as an an SR you know something that as an an SR you know something that as an an SR you know something that helps you debug maybe SR for for AI um helps you debug maybe SR for for AI um helps you debug maybe SR for for AI um but I feel like there's a lot of black but I feel like there's a lot of black but I feel like there's a lot of black boxes out there like oh there's a SR boxes out there like oh there's a SR boxes out there like oh there's a SR agent that does this or S agent that agent that does this or S agent that agent that does this or S agent that does this all we're really trying to do does this all we're really trying to do does this all we're really trying to do ourselves is take your local debugging ourselves is take your local debugging ourselves is take your local debugging experience with cloud code cursor and experience with cloud code cursor and experience with cloud code cursor and run it periodically so pick your sandbox run it periodically so pick your sandbox run it periodically so pick your sandbox pick your harness, pick your skills, pick your harness, pick your skills, pick your harness, pick your skills, we'll pre-bank a bunch of things with we'll pre-bank a bunch of things with we'll pre-bank a bunch of things with you. So, we're just trying to again take you. So, we're just trying to again take you. So, we're just trying to again take the things we were doing locally and the things we were doing locally and the things we were doing locally and actually run them um uh you know, run actually run them um uh you know, run actually run them um uh you know, run them in a system. So, we believe in you

them in a system. So, we believe in you them in a system. So, we believe in you know an open approach um to this and um know an open approach um to this and um know an open approach um to this and um I I'll give you a demo of what this I I'll give you a demo of what this I I'll give you a demo of what this looks like um from a product looks like um from a product looks like um from a product perspective. So, so this is um this is a a financial so this is um this is a a financial trading agent. Um given what you saw in trading agent. Um given what you saw in trading agent. Um given what you saw in the previous the previous the previous uh presentation, I would not recommend uh presentation, I would not recommend uh presentation, I would not recommend doing a financial trading agent. Um they doing a financial trading agent. Um they doing a financial trading agent. Um they they they're unlikely to make you money. they they're unlikely to make you money. they they're unlikely to make you money. Uh at least not not yet. Um maybe Uh at least not not yet. Um maybe Uh at least not not yet. Um maybe there's some people uh doing it good. there's some people uh doing it good. there's some people uh doing it good. But long story short is this one's, you But long story short is this one's, you But long story short is this one's, you know, uh people asking questions about know, uh people asking questions about know, uh people asking questions about stock trading right now and it's giving stock trading right now and it's giving stock trading right now and it's giving giving answers. Um there's a lot of ways giving answers. Um there's a lot of ways giving answers. Um there's a lot of ways this this can fail. And so this this this this can fail. And so this this this this can fail. And so this this gives you this is a rise. It's a gives you this is a rise. It's a gives you this is a rise. It's a platform. So first off from let me platform. So first off from let me platform. So first off from let me describe the products we have. Uh this describe the products we have. Uh this describe the products we have. Uh this is AX which is our our SAS platform. Um is AX which is our our SAS platform. Um is AX which is our our SAS platform. Um we also have Phoenix which is open we also have Phoenix which is open we also have Phoenix which is open source if you just want to start source if you just want to start source if you just want to start tomorrow. Um Signal right now is is just tomorrow. Um Signal right now is is just tomorrow. Um Signal right now is is just available in in our AX SAS platform. Um available in in our AX SAS platform. Um available in in our AX SAS platform. Um which also could be deployed VPC but but which also could be deployed VPC but but which also could be deployed VPC but but this so so give you an idea of our this so so give you an idea of our this so so give you an idea of our product lines. Uh if you want to try out product lines. Uh if you want to try out product lines. Uh if you want to try out Signal, it's it's an AX. Um Signal, it's it's an AX. Um Signal, it's it's an AX. Um and what it looks like is something like and what it looks like is something like and what it looks like is something like this, which is it's just periodically this, which is it's just periodically this, which is it's just periodically running and and kind of coming up with running and and kind of coming up with running and and kind of coming up with like issues and you can hook it up to like issues and you can hook it up to like issues and you can hook it up to your GitHub repo. It can create an issue your GitHub repo. It can create an issue your GitHub repo. It can create an issue in your repo. You can create an in your repo. You can create an in your repo. You can create an evaluator from this. Maybe u maybe evaluator from this. Maybe u maybe evaluator from this. Maybe u maybe there's a a specific problem by which there's a a specific problem by which there's a a specific problem by which you want to catch again. You can add you want to catch again. You can add you want to catch again. You can add these to a data set. So if you want to

these to a data set. So if you want to these to a data set. So if you want to add these and and it has evidence add these and and it has evidence add these and and it has evidence associated with this like traces um in associated with this like traces um in associated with this like traces um in this case um in this one it has skills this case um in this one it has skills this case um in this one it has skills like for Google cloud and some other like for Google cloud and some other like for Google cloud and some other logging systems. So we can frontend a logging systems. So we can frontend a logging systems. So we can frontend a bunch of places the data we're we're bunch of places the data we're we're bunch of places the data we're we're pretty good at building I think these pretty good at building I think these pretty good at building I think these skills to debug issues again uh but you skills to debug issues again uh but you skills to debug issues again uh but you can add your own skills. So these can add your own skills. So these can add your own skills. So these examples here you know traces running examples here you know traces running examples here you know traces running out without a guardrail. Um there's out without a guardrail. Um there's out without a guardrail. Um there's there's um uh you know safety safety there's um uh you know safety safety there's um uh you know safety safety issues and intent issues and a lot of issues and intent issues and a lot of issues and intent issues and a lot of this too is like you know how does this this too is like you know how does this this too is like you know how does this work? How do I you know it feels a work? How do I you know it feels a work? How do I you know it feels a little too black box to me. Well all little too black box to me. Well all little too black box to me. Well all this is open and open box um in the this is open and open box um in the this is open and open box um in the sense that um I can set up you know I sense that um I can set up you know I sense that um I can set up you know I can set up the harness that I want it to can set up the harness that I want it to can set up the harness that I want it to run on. This one's cloud code. I can run on. This one's cloud code. I can run on. This one's cloud code. I can pick my sandboxes and sandbox systems. pick my sandboxes and sandbox systems. pick my sandboxes and sandbox systems. Um, I can use cloud managed agents if I Um, I can use cloud managed agents if I Um, I can use cloud managed agents if I want. I can use Arise sandbox. Uh, why want. I can use Arise sandbox. Uh, why want. I can use Arise sandbox. Uh, why would I want to use Arise sandboxes would I want to use Arise sandboxes would I want to use Arise sandboxes versus cloud managed agents? Well, a lot versus cloud managed agents? Well, a lot versus cloud managed agents? Well, a lot of our customers um don't want to of our customers um don't want to of our customers um don't want to connect their production systems to connect their production systems to connect their production systems to Tanthropic. You know, you you you want Tanthropic. You know, you you you want Tanthropic. You know, you you you want your s you want these these sandboxes to your s you want these these sandboxes to your s you want these these sandboxes to debug your database or connect to it. So debug your database or connect to it. So debug your database or connect to it. So we install in the VPC of a lot of you we install in the VPC of a lot of you we install in the VPC of a lot of you know bigname companies out there um from know bigname companies out there um from know bigname companies out there um from from Uber to um to bookings to you name from Uber to um to bookings to you name from Uber to um to bookings to you name it and and these people don't want to it and and these people don't want to it and and these people don't want to send their connections out but they'll send their connections out but they'll send their connections out but they'll they'll use a s you know many many they'll use a s you know many many they'll use a s you know many many companies um are very comfortable companies um are very comfortable companies um are very comfortable installing a V into a VPC and actually

installing a V into a VPC and actually installing a V into a VPC and actually connecting it up. So you can use Arise connecting it up. So you can use Arise connecting it up. So you can use Arise sandboxes or you can use Daytona or any sandboxes or you can use Daytona or any sandboxes or you can use Daytona or any of any of these that you're comfortable of any of these that you're comfortable of any of these that you're comfortable with um that you built relationships with um that you built relationships with um that you built relationships with. Um and and then from a a platform with. Um and and then from a a platform with. Um and and then from a a platform perspective, you know, we support perspective, you know, we support perspective, you know, we support um running we support, you know, um running we support, you know, um running we support, you know, tracking the different agents that tracking the different agents that tracking the different agents that you're running. So you have this swarm you're running. So you have this swarm you're running. So you have this swarm of agents. Maybe you've kicked off um of agents. Maybe you've kicked off um of agents. Maybe you've kicked off um maybe you're kicking off a signal which maybe you're kicking off a signal which maybe you're kicking off a signal which which is our agent. It's running which is our agent. It's running which is our agent. It's running periodically. Maybe you're kicking off periodically. Maybe you're kicking off periodically. Maybe you're kicking off your you know you've named another agent your you know you've named another agent your you know you've named another agent um in the system. And these all support, um in the system. And these all support, um in the system. And these all support, you know, viewing the session that that you know, viewing the session that that you know, viewing the session that that ran, downloading the transcript, and you ran, downloading the transcript, and you ran, downloading the transcript, and you can resume a clouded session locally, can resume a clouded session locally, can resume a clouded session locally, too. So, the idea is that this thing's too. So, the idea is that this thing's too. So, the idea is that this thing's constantly running. You're picking the constantly running. You're picking the constantly running. You're picking the harness, the sandbox. You're deciding harness, the sandbox. You're deciding harness, the sandbox. You're deciding the prompt if you want, hey, don't be the prompt if you want, hey, don't be the prompt if you want, hey, don't be aggressive or look for, you know, look aggressive or look for, you know, look aggressive or look for, you know, look for security issues. So, you're deciding for security issues. So, you're deciding for security issues. So, you're deciding the prompts that drive this and you're the prompts that drive this and you're the prompts that drive this and you're also deciding the skills that go along also deciding the skills that go along also deciding the skills that go along with this. So um in with this. So um in with this. So um in a preset here um I can add you know add a preset here um I can add you know add a preset here um I can add you know add different skills. I can add my own different skills. I can add my own different skills. I can add my own skills. I can link repos. I can I have skills. I can link repos. I can I have skills. I can link repos. I can I have pre-baked skills too. Um so the ideas pre-baked skills too. Um so the ideas pre-baked skills too. Um so the ideas observability platforms are really observability platforms are really observability platforms are really starting to get are becoming tied to the starting to get are becoming tied to the starting to get are becoming tied to the continuous loop to the the fix not just continuous loop to the the fix not just continuous loop to the the fix not just the the signal. Um, and and you you want the the signal. Um, and and you you want the the signal. Um, and and you you want to take your local experience you have to take your local experience you have to take your local experience you have debugging the stuff locally. You want to

debugging the stuff locally. You want to debugging the stuff locally. You want to take the evals that are running and and take the evals that are running and and take the evals that are running and and actually have these all work in actually have these all work in actually have these all work in something that puts up a fix or at least something that puts up a fix or at least something that puts up a fix or at least gets you a cold start and then I can gets you a cold start and then I can gets you a cold start and then I can take it over locally if I want to take it over locally if I want to take it over locally if I want to continue debugging uh from from here. continue debugging uh from from here. continue debugging uh from from here. Um, so this gives you um a rough idea of Um, so this gives you um a rough idea of Um, so this gives you um a rough idea of of kind of of of signal to PR um what of kind of of of signal to PR um what of kind of of of signal to PR um what we're doing. Um, I did want to offer, we're doing. Um, I did want to offer, we're doing. Um, I did want to offer, you know, questions if people people you know, questions if people people you know, questions if people people have any questions on what we're doing have any questions on what we're doing have any questions on what we're doing or how we see uh the industry evolving. or how we see uh the industry evolving. or how we see uh the industry evolving. Happy to happy to answer. Thank you. question of like why can't we just question of like why can't we just connect cloud code to your data and have connect cloud code to your data and have connect cloud code to your data and have cloud code do all these things. I think cloud code do all these things. I think cloud code do all these things. I think there's like a version of that question there's like a version of that question there's like a version of that question that can probably be asked for these that can probably be asked for these that can probably be asked for these autofixes, right? Like why not have autofixes, right? Like why not have autofixes, right? Like why not have cloud code read the traces and push the cloud code read the traces and push the cloud code read the traces and push the PR itself.

PR itself. PR itself. >> I'm curious how you would respond to >> I'm curious how you would respond to >> I'm curious how you would respond to that question. that question. that question. >> Yeah. Uh so so why wouldn't have cloud >> Yeah. Uh so so why wouldn't have cloud >> Yeah. Uh so so why wouldn't have cloud code kind of hook to your your data and code kind of hook to your your data and code kind of hook to your your data and just just do it? Um the answer is like just just do it? Um the answer is like just just do it? Um the answer is like you should um like like the vision and you should um like like the vision and you should um like like the vision and what we do actually at at Arise is we what we do actually at at Arise is we what we do actually at at Arise is we have uh a lot of skills. I think first have uh a lot of skills. I think first have uh a lot of skills. I think first off to to make that really work well you off to to make that really work well you off to to make that really work well you have to do a bit of well-designed skills have to do a bit of well-designed skills have to do a bit of well-designed skills in the data space. The skill like the in the data space. The skill like the in the data space. The skill like the the important things of designing these the important things of designing these the important things of designing these skills are are around really around skills are are around really around skills are are around really around getting data you know finding the right getting data you know finding the right getting data you know finding the right data first. So I want to find a group of data first. So I want to find a group of data first. So I want to find a group of traces relative to a session or traces relative to a session or traces relative to a session or something. Getting that data into the something. Getting that data into the something. Getting that data into the repo in a file format. These harnesses repo in a file format. These harnesses repo in a file format. These harnesses are magical with files. So you get the are magical with files. So you get the are magical with files. So you get the file what happened. In some cases we file what happened. In some cases we file what happened. In some cases we have 10meg files like sitting in the have 10meg files like sitting in the have 10meg files like sitting in the repo. Um so it's designing the skill to repo. Um so it's designing the skill to repo. Um so it's designing the skill to be really really well done with the um be really really well done with the um be really really well done with the um with this data and and then giving with this data and and then giving with this data and and then giving claude enough skills to be composable to claude enough skills to be composable to claude enough skills to be composable to find issues. So the answer is absolutely find issues. So the answer is absolutely find issues. So the answer is absolutely yes. like we have Pyroscope skills that yes. like we have Pyroscope skills that yes. like we have Pyroscope skills that will find memory issues. We have facets will find memory issues. We have facets will find memory issues. We have facets in Pyroscope that the skill knows how to in Pyroscope that the skill knows how to in Pyroscope that the skill knows how to use. I can cohort by customer to see if use. I can cohort by customer to see if use. I can cohort by customer to see if a customer is causing an issue. Um but a customer is causing an issue. Um but a customer is causing an issue. Um but but you've got to kind of design the but you've got to kind of design the but you've got to kind of design the skill surface area in a way that Claude skill surface area in a way that Claude skill surface area in a way that Claude can really really work well and and and can really really work well and and and can really really work well and and and it's not just like point Claude at the it's not just like point Claude at the it's not just like point Claude at the data.

data. data. >> I see. Thank you. >> I see. Thank you. >> I see. Thank you. >> Any other questions? >> Any other questions? >> Any other questions? Anyone else? Anyone else? Anyone else? >> Oh yeah. Okay. One more. >> Oh yeah. Okay. One more. >> Oh yeah. Okay. One more. Thanks for the talk. Um, there was quite Thanks for the talk. Um, there was quite Thanks for the talk. Um, there was quite a few mention of eval but you know I'm a few mention of eval but you know I'm a few mention of eval but you know I'm looking at the traces so you know I looking at the traces so you know I looking at the traces so you know I understand the concept of traces but understand the concept of traces but understand the concept of traces but where where did the evals come in when where where did the evals come in when where where did the evals come in when you have that signal that says hey you have that signal that says hey you have that signal that says hey something broke in production. something broke in production. something broke in production. >> Yeah. So so the so the eval typically >> Yeah. So so the so the eval typically >> Yeah. So so the so the eval typically will the um the eval essentially are will the um the eval essentially are will the um the eval essentially are running and being layered on typically running and being layered on typically running and being layered on typically to the production traces something we to the production traces something we to the production traces something we call online evals. Um let me see if this call online evals. Um let me see if this call online evals. Um let me see if this one has an example here of it. Um so so one has an example here of it. Um so so one has an example here of it. Um so so eval actually are data on the trace eval actually are data on the trace eval actually are data on the trace itself and so the agent knows how to uh itself and so the agent knows how to uh itself and so the agent knows how to uh grab the data from traces knows how to grab the data from traces knows how to grab the data from traces knows how to um visualize and you know the skills to um visualize and you know the skills to um visualize and you know the skills to basically pull data for for the basically pull data for for the basically pull data for for the aggregate values of evals across the aggregate values of evals across the aggregate values of evals across the traces so that so the the skills that traces so that so the the skills that traces so that so the the skills that you give uh the harness allow it to get you give uh the harness allow it to get you give uh the harness allow it to get the data on the eval from from the the data on the eval from from the the data on the eval from from the traces. Um so eval are kind of like I traces. Um so eval are kind of like I traces. Um so eval are kind of like I view them as le at least a first view them as le at least a first view them as le at least a first generation eval evals which are elements generation eval evals which are elements generation eval evals which are elements a judge um as a as a AI layer that a judge um as a as a AI layer that a judge um as a as a AI layer that allows you to run periodically and and allows you to run periodically and and allows you to run periodically and and assess your system but it's like it but assess your system but it's like it but assess your system but it's like it but it's adding a little bit more you know it's adding a little bit more you know it's adding a little bit more you know pre-processed information on the data pre-processed information on the data pre-processed information on the data that that and then as signal is running that that and then as signal is running that that and then as signal is running it's using data from the evals that were it's using data from the evals that were it's using data from the evals that were layered on um in addition to all the raw layered on um in addition to all the raw layered on um in addition to all the raw data that it has there Um it but it

data that it has there Um it but it data that it has there Um it but it tends to be like you build an eval for a tends to be like you build an eval for a tends to be like you build an eval for a failure you've seen before a lot of failure you've seen before a lot of failure you've seen before a lot of times. So I have these prompt injection times. So I have these prompt injection times. So I have these prompt injection things that I'm trying to catch or things that I'm trying to catch or things that I'm trying to catch or something or um or or a failure in the something or um or or a failure in the something or um or or a failure in the way it's responded maybe to to something way it's responded maybe to to something way it's responded maybe to to something before. So they they tend to be this before. So they they tend to be this before. So they they tend to be this like you know u at least the LM as a like you know u at least the LM as a like you know u at least the LM as a judge is tends to be like this this judge is tends to be like this this judge is tends to be like this this thing you um preset up and then you can thing you um preset up and then you can thing you um preset up and then you can actually create evaluators for failure. actually create evaluators for failure. actually create evaluators for failure. Say you find this failure that's pretty Say you find this failure that's pretty Say you find this failure that's pretty common and happening all the time. I can common and happening all the time. I can common and happening all the time. I can create an eval so I can catch it next create an eval so I can catch it next create an eval so I can catch it next time. I just you think of it as like time. I just you think of it as like time. I just you think of it as like almost a an AI um assessment that's almost a an AI um assessment that's almost a an AI um assessment that's always running. Uh the other note is the always running. Uh the other note is the always running. Uh the other note is the element as a judge can run really at element as a judge can run really at element as a judge can run really at scale. Well, every you know I have scale. Well, every you know I have scale. Well, every you know I have customers who who lay you know layer customers who who lay you know layer customers who who lay you know layer element as a judge across um their full element as a judge across um their full element as a judge across um their full data set uh where where this tends to be data set uh where where this tends to be data set uh where where this tends to be like you know uh more periodic on a lot like you know uh more periodic on a lot like you know uh more periodic on a lot of data. So cool. Thank you.