Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop

>> Uh thank you all for coming, first of >> Uh thank you all for coming, first of all. all. all. And um And um And um I want to talk today about uh raising I want to talk today about uh raising I want to talk today about uh raising the floor. So, it's this kind of a term the floor. So, it's this kind of a term the floor. So, it's this kind of a term we use a lot. Um we use a lot. Um we use a lot. Um mainly I want to talk about very, very mainly I want to talk about very, very mainly I want to talk about very, very practical, like what do I actually see, practical, like what do I actually see, practical, like what do I actually see, what do we actually see working in the what do we actually see working in the what do we actually see working in the real world, um how do people how are real world, um how do people how are real world, um how do people how are people making their agents better? people making their agents better? people making their agents better? So, the first thing I want to say is So, the first thing I want to say is So, the first thing I want to say is like um I could just I could just say a like um I could just I could just say a like um I could just I could just say a bunch of stuff. Like I think, you know, bunch of stuff. Like I think, you know, bunch of stuff. Like I think, you know, um um um the title of this track is like the title of this track is like the title of this track is like continual learning. I think it's like continual learning. I think it's like continual learning. I think it's like notable that in the real world, there's notable that in the real world, there's notable that in the real world, there's really not that much continual learning, really not that much continual learning, really not that much continual learning, right? Uh if you look at like the labs, right? Uh if you look at like the labs, right? Uh if you look at like the labs, if you look at like products that are in if you look at like products that are in if you look at like products that are in the real world, you really don't see a the real world, you really don't see a the real world, you really don't see a lot of continual learning. So, um lot of continual learning. So, um lot of continual learning. So, um I think it's very easy to like I could, I think it's very easy to like I could, I think it's very easy to like I could, you know, spend 20 minutes just talking you know, spend 20 minutes just talking you know, spend 20 minutes just talking about like, "Hey, here's a bunch of about like, "Hey, here's a bunch of about like, "Hey, here's a bunch of frameworks. Here's a bunch of like frameworks. Here's a bunch of like frameworks. Here's a bunch of like really nice terms." But what I'd rather really nice terms." But what I'd rather really nice terms." But what I'd rather do is actually turn this a little bit do is actually turn this a little bit do is actually turn this a little bit into a dialogue. Um this is not just into a dialogue. Um this is not just into a dialogue. Um this is not just because I procrastinated making a bunch because I procrastinated making a bunch because I procrastinated making a bunch of slides and because Fable was delayed of slides and because Fable was delayed of slides and because Fable was delayed and I was counting on that to make the and I was counting on that to make the and I was counting on that to make the slides, but also because like the slides, but also because like the slides, but also because like the reality is that um there aren't really reality is that um there aren't really reality is that um there aren't really good standards for these things, right?

good standards for these things, right? good standards for these things, right? Like there there's not some Like there there's not some Like there there's not some one-size-fits-all uh solution. And so, one-size-fits-all uh solution. And so, one-size-fits-all uh solution. And so, I'm going to I do have slides, believe I'm going to I do have slides, believe I'm going to I do have slides, believe it or not, but what we're also going to it or not, but what we're also going to it or not, but what we're also going to do is sort of like I'd like to hear from do is sort of like I'd like to hear from do is sort of like I'd like to hear from you guys, people actually building you guys, people actually building you guys, people actually building agents. Like where where have the, you agents. Like where where have the, you agents. Like where where have the, you know, Twitter, you know, eval discourse, know, Twitter, you know, eval discourse, know, Twitter, you know, eval discourse, where has that failed you, right? Where where has that failed you, right? Where where has that failed you, right? Where where is it not working? What are the where is it not working? What are the where is it not working? What are the things you're actually hitting in real things you're actually hitting in real things you're actually hitting in real life? I'd like to talk about that. So, life? I'd like to talk about that. So, life? I'd like to talk about that. So, please like right now start thinking please like right now start thinking please like right now start thinking about your questions, start thinking about your questions, start thinking about your questions, start thinking about the annoying parts of your flow um about the annoying parts of your flow um about the annoying parts of your flow um when you're building agents. And I'd when you're building agents. And I'd when you're building agents. And I'd like to keep a lot of time for Q&A. like to keep a lot of time for Q&A. like to keep a lot of time for Q&A. Um Um Um so, the reality is like a year ago, so, the reality is like a year ago, so, the reality is like a year ago, agents barely existed. Like I remember agents barely existed. Like I remember agents barely existed. Like I remember being at like a speaker dinner here like being at like a speaker dinner here like being at like a speaker dinner here like a year ago and we're like, yeah, do you a year ago and we're like, yeah, do you a year ago and we're like, yeah, do you think like agents will like, you know, think like agents will like, you know, think like agents will like, you know, what you know, will they like keep what you know, will they like keep what you know, will they like keep getting better? Will they not? Um, and I getting better? Will they not? Um, and I getting better? Will they not? Um, and I think the crazy thing is like, you know, think the crazy thing is like, you know, think the crazy thing is like, you know, we're at we were at this point in time a we're at we were at this point in time a we're at we were at this point in time a year ago where everything was like a year ago where everything was like a year ago where everything was like a chatbot mostly, right? Um, I think chatbot mostly, right? Um, I think chatbot mostly, right? Um, I think people here, people in this room people here, people in this room people here, people in this room probably, you know, were a little probably, you know, were a little probably, you know, were a little further ahead. Um, further ahead. Um, further ahead. Um, it was it was a lot simpler then. Uh, if it was it was a lot simpler then. Uh, if it was it was a lot simpler then. Uh, if you remember Evals, the Eval discourse you remember Evals, the Eval discourse you remember Evals, the Eval discourse for like a year or two ago, it would be for like a year or two ago, it would be for like a year or two ago, it would be like, oh, what is the, you know, capital like, oh, what is the, you know, capital like, oh, what is the, you know, capital of the United States? And you're like, of the United States? And you're like, of the United States? And you're like, yeah, and you want to make sure that it yeah, and you want to make sure that it yeah, and you want to make sure that it returns Washington, D.C. and um, that returns Washington, D.C. and um, that returns Washington, D.C. and um, that was an easier time, right? It was like was an easier time, right? It was like was an easier time, right? It was like chatbots were so much more limited in chatbots were so much more limited in chatbots were so much more limited in their sort of flexibility that it was their sort of flexibility that it was their sort of flexibility that it was kind of easy to like kind of easy to like kind of easy to like uh, to do a bunch of like, you know, uh, to do a bunch of like, you know, uh, to do a bunch of like, you know, fact-checking things. Like you knew the fact-checking things. Like you knew the fact-checking things. Like you knew the answer to most questions your users

answer to most questions your users answer to most questions your users would ask almost is is another way of would ask almost is is another way of would ask almost is is another way of saying that. At least like a like 80% of saying that. At least like a like 80% of saying that. At least like a like 80% of them or 90% of them. them or 90% of them. them or 90% of them. Um, but yeah, we have agents being Um, but yeah, we have agents being Um, but yeah, we have agents being deployed in finance, healthcare, deployed in finance, healthcare, deployed in finance, healthcare, defense. defense. defense. And um, I was actually really against And um, I was actually really against And um, I was actually really against the word. One of my like worst takes is the word. One of my like worst takes is the word. One of my like worst takes is like uh, early agents. I was like, ah, I like uh, early agents. I was like, ah, I like uh, early agents. I was like, ah, I hate the word agent. Um, I think there hate the word agent. Um, I think there hate the word agent. Um, I think there were a lot of people that felt the same. were a lot of people that felt the same. were a lot of people that felt the same. It was like, come on, it's an LLM, it's It was like, come on, it's an LLM, it's It was like, come on, it's an LLM, it's a whatever. But I I actually think it's a whatever. But I I actually think it's a whatever. But I I actually think it's it's valuable because I think we're it's valuable because I think we're it's valuable because I think we're seeing that agents are this like almost seeing that agents are this like almost seeing that agents are this like almost self-aware entity, right? They kind of self-aware entity, right? They kind of self-aware entity, right? They kind of like run around their environment, they like run around their environment, they like run around their environment, they have these tools they're using. Um, when have these tools they're using. Um, when have these tools they're using. Um, when they hit, you know, roadblocks, they they hit, you know, roadblocks, they they hit, you know, roadblocks, they start getting really creative, right? start getting really creative, right? start getting really creative, right? And that's what makes agents really And that's what makes agents really And that's what makes agents really powerful, but like that's also what powerful, but like that's also what powerful, but like that's also what makes them like catastrophic oh, well, makes them like catastrophic oh, well, makes them like catastrophic oh, well, I'll just, you know, I'll just like uh, I'll just, you know, I'll just like uh, I'll just, you know, I'll just like uh, decompile this and I'll just, you know, decompile this and I'll just, you know, decompile this and I'll just, you know, like do this thing that you had no idea like do this thing that you had no idea like do this thing that you had no idea that you could have never imagined. Um, that you could have never imagined. Um, that you could have never imagined. Um, sometimes those solutions are are sometimes those solutions are are sometimes those solutions are are helpful, right? Sometimes they're helpful, right? Sometimes they're helpful, right? Sometimes they're they're actually pretty harmful. Um, but they're actually pretty harmful. Um, but they're actually pretty harmful. Um, but yeah, what's very certain is we've come yeah, what's very certain is we've come yeah, what's very certain is we've come a very long way from like next token a very long way from like next token a very long way from like next token prediction. Like if you think about prediction. Like if you think about prediction. Like if you think about chatbots, it was literally just like, chatbots, it was literally just like, chatbots, it was literally just like, oh, it was like you know it's going to oh, it was like you know it's going to oh, it was like you know it's going to what is the next likely word? It was what is the next likely word? It was what is the next likely word? It was very easy to reason about. Um very easy to reason about. Um very easy to reason about. Um and I think the thing that you know on and I think the thing that you know on and I think the thing that you know on the evaluation front, I think the the evaluation front, I think the the evaluation front, I think the reality is like most of the things you'd reality is like most of the things you'd reality is like most of the things you'd read uh online about evals um are really read uh online about evals um are really read uh online about evals um are really still like stuck in this chatbot era.

still like stuck in this chatbot era. still like stuck in this chatbot era. It's very like well like come up with It's very like well like come up with It's very like well like come up with your like 1,000, you know, eval data set your like 1,000, you know, eval data set your like 1,000, you know, eval data set and it's like the reality is like and it's like the reality is like and it's like the reality is like nobody's doing that. Um very few people nobody's doing that. Um very few people nobody's doing that. Um very few people anyway, sorry if you are. Um anyway, sorry if you are. Um anyway, sorry if you are. Um and uh you know, I think what teams have and uh you know, I think what teams have and uh you know, I think what teams have seen over and over again is like yeah, seen over and over again is like yeah, seen over and over again is like yeah, you can do that uh but those evals like you can do that uh but those evals like you can do that uh but those evals like break as soon as you have a new model, break as soon as you have a new model, break as soon as you have a new model, as soon as you like switch harnesses, as soon as you like switch harnesses, as soon as you like switch harnesses, like you have a bunch of like tools that like you have a bunch of like tools that like you have a bunch of like tools that you're like oh yeah, I'm going to make you're like oh yeah, I'm going to make you're like oh yeah, I'm going to make sure that I'm going to write an eval sure that I'm going to write an eval sure that I'm going to write an eval where it has to like call this tool if I where it has to like call this tool if I where it has to like call this tool if I ask it this question and it's like oh ask it this question and it's like oh ask it this question and it's like oh then you switch to like Claude code CLI then you switch to like Claude code CLI then you switch to like Claude code CLI and now 80% of your evals suck and it's and now 80% of your evals suck and it's and now 80% of your evals suck and it's like okay, you can keep doing that but like okay, you can keep doing that but like okay, you can keep doing that but the reality is like the one thing I the reality is like the one thing I the reality is like the one thing I could promise you is that things are could promise you is that things are could promise you is that things are going to keep changing. Like we're not going to keep changing. Like we're not going to keep changing. Like we're not done and so I'd be very careful about done and so I'd be very careful about done and so I'd be very careful about you know, uh investing you know, months you know, uh investing you know, months you know, uh investing you know, months in some sort of eval set that's going to in some sort of eval set that's going to in some sort of eval set that's going to you know, slow you down, right? I think you know, slow you down, right? I think you know, slow you down, right? I think the whole thing here is like you want the whole thing here is like you want the whole thing here is like you want more safety but you you don't want more safety but you you don't want more safety but you you don't want theater and I think that again, I think theater and I think that again, I think theater and I think that again, I think the evals as has been sort of the evals as has been sort of the evals as has been sort of uh prescribed by uh what I call like big uh prescribed by uh what I call like big uh prescribed by uh what I call like big eval. Um I I think that there's this eval. Um I I think that there's this eval. Um I I think that there's this reality where it's kind of like oh you reality where it's kind of like oh you reality where it's kind of like oh you should really should eval but then like should really should eval but then like should really should eval but then like do you actually delay, you know, uh do you actually delay, you know, uh do you actually delay, you know, uh including a new you know, upgrading to including a new you know, upgrading to including a new you know, upgrading to the new model on your product in your the new model on your product in your the new model on your product in your products? Do you actually delay it 2 products? Do you actually delay it 2 products? Do you actually delay it 2 weeks to update your evals or not, weeks to update your evals or not, weeks to update your evals or not, right? I I think most people would say right? I I think most people would say right? I I think most people would say no.

no. no. Um so uh I'm the CTO and co-founder of Um so uh I'm the CTO and co-founder of Um so uh I'm the CTO and co-founder of this company called Raindrop. Um very this company called Raindrop. Um very this company called Raindrop. Um very quickly is like we find critical issues quickly is like we find critical issues quickly is like we find critical issues in production agents, we verify those in production agents, we verify those in production agents, we verify those fixes actually work without unexpected fixes actually work without unexpected fixes actually work without unexpected side effects, and we also simulate side effects, and we also simulate side effects, and we also simulate changes before they land in production changes before they land in production changes before they land in production um based on past behavior. um based on past behavior. um based on past behavior. Uh we're used by the best AI companies Uh we're used by the best AI companies Uh we're used by the best AI companies in the world and Fortune 100s, a lot of in the world and Fortune 100s, a lot of in the world and Fortune 100s, a lot of logos I'm not allowed to put on here logos I'm not allowed to put on here logos I'm not allowed to put on here yet. yet. yet. Um but companies like Vercel, Speak, Um but companies like Vercel, Speak, Um but companies like Vercel, Speak, Framer, Framer, Framer, um and uh I think what it means is we um and uh I think what it means is we um and uh I think what it means is we get this like amazing peak into like get this like amazing peak into like get this like amazing peak into like again, what is actually working in the again, what is actually working in the again, what is actually working in the real world. I think one of our like real world. I think one of our like real world. I think one of our like tenants as a company is that things are tenants as a company is that things are tenants as a company is that things are changing constantly and we have to changing constantly and we have to changing constantly and we have to change what we're doing constantly. And change what we're doing constantly. And change what we're doing constantly. And so I think uh if we we try to be very so I think uh if we we try to be very so I think uh if we we try to be very very honest with our both ourselves and very honest with our both ourselves and very honest with our both ourselves and our customers like what works and what our customers like what works and what our customers like what works and what does not work. We try not to sell things does not work. We try not to sell things does not work. We try not to sell things that don't work. that don't work. that don't work. Um just again, two two things. We have Um just again, two two things. We have Um just again, two two things. We have this like uh open-source tool that like this like uh open-source tool that like this like uh open-source tool that like thousands of people use, maybe you use thousands of people use, maybe you use thousands of people use, maybe you use it yourself. It's called Workshop. Um so it yourself. It's called Workshop. Um so it yourself. It's called Workshop. Um so that's made by us. It's like an that's made by us. It's like an that's made by us. It's like an open-source tracing tool. It's really open-source tracing tool. It's really open-source tracing tool. It's really really cool if you're trying to really cool if you're trying to really cool if you're trying to experiment with like self-healing loops.

experiment with like self-healing loops. experiment with like self-healing loops. I think it's the best way to do that I think it's the best way to do that I think it's the best way to do that because if there's anything it can't do, because if there's anything it can't do, because if there's anything it can't do, uh your agent can just like add it, uh your agent can just like add it, uh your agent can just like add it, which is pretty cool. Um and so highly which is pretty cool. Um and so highly which is pretty cool. Um and so highly highly recommend it. Like again, I know highly recommend it. Like again, I know highly recommend it. Like again, I know thousands of people use it like thousands of people use it like thousands of people use it like people I anyway, I bump into people all people I anyway, I bump into people all people I anyway, I bump into people all the time that use it. Raindrop is like the time that use it. Raindrop is like the time that use it. Raindrop is like our kind of hosted offering that does our kind of hosted offering that does our kind of hosted offering that does issue detection. Think of it like issue detection. Think of it like issue detection. Think of it like Sentry, but detects issues Sentry, but detects issues Sentry, but detects issues for agents. Sorry. for agents. Sorry. for agents. Sorry. Um and we also make howtoeval.com. And Um and we also make howtoeval.com. And Um and we also make howtoeval.com. And so I think it's one of the most popular so I think it's one of the most popular so I think it's one of the most popular resources on how to evaluate AI agents. resources on how to evaluate AI agents. resources on how to evaluate AI agents. It is again, the link is literally in It is again, the link is literally in It is again, the link is literally in the name. It's how to eval.com. the name. It's how to eval.com. the name. It's how to eval.com. Um and I it's it's uh Um and I it's it's uh Um and I it's it's uh uh our attempt at a very very uh our attempt at a very very uh our attempt at a very very no-bullshit guide at at what actually no-bullshit guide at at what actually no-bullshit guide at at what actually works. And I'll be talking a little bit works. And I'll be talking a little bit works. And I'll be talking a little bit about it today. about it today. about it today. Um Um Um I think the like root question that I think the like root question that I think the like root question that we're trying to figure out today we're trying to figure out today we're trying to figure out today together is how do you make your your together is how do you make your your together is how do you make your your agent better, right? It's not even like agent better, right? It's not even like agent better, right? It's not even like what issues does your agent have? Uh it what issues does your agent have? Uh it what issues does your agent have? Uh it is actually how to make your agent is actually how to make your agent is actually how to make your agent better because your agent will have better because your agent will have better because your agent will have issues that issues that issues that potentially you can't solve or potentially you can't solve or potentially you can't solve or not exactly worth solving, right? Like not exactly worth solving, right? Like not exactly worth solving, right? Like um I think that we saw this uh over and um I think that we saw this uh over and um I think that we saw this uh over and over again where it's like um over again where it's like um over again where it's like um you know, you can imagine that um you know, you can imagine that um you know, you can imagine that um th- there are some things that you're th- there are some things that you're th- there are some things that you're like better off waiting for like, you like better off waiting for like, you like better off waiting for like, you know, we know Fable exists now. Maybe know, we know Fable exists now. Maybe know, we know Fable exists now. Maybe Fable, you know, like should you train Fable, you know, like should you train Fable, you know, like should you train your own like Fable level model? It's your own like Fable level model? It's your own like Fable level model? It's like probably not, right? Um, and like probably not, right? Um, and like probably not, right? Um, and there'll be benefits uh when you can there'll be benefits uh when you can there'll be benefits uh when you can just incorporate that into your product.

just incorporate that into your product. just incorporate that into your product. Um, and so there's this actual balance, Um, and so there's this actual balance, Um, and so there's this actual balance, like how do how do I actually make my like how do how do I actually make my like how do how do I actually make my agent better uh with the tools that I agent better uh with the tools that I agent better uh with the tools that I have? have? have? When When When the way that we start thinking about it the way that we start thinking about it the way that we start thinking about it with customers is something like this, with customers is something like this, with customers is something like this, which is like, are you a benchmark maxer which is like, are you a benchmark maxer which is like, are you a benchmark maxer or a floor raiser? or a floor raiser? or a floor raiser? Um, Um, Um, I think that one of the problems when we I think that one of the problems when we I think that one of the problems when we talk about evaluating agents is that the talk about evaluating agents is that the talk about evaluating agents is that the terms are really confused. Like you hear terms are really confused. Like you hear terms are really confused. Like you hear OpenAI has a new like, you know, uh eval OpenAI has a new like, you know, uh eval OpenAI has a new like, you know, uh eval benchmark. And um, you know, they have benchmark. And um, you know, they have benchmark. And um, you know, they have evals, they run evals. And then you hear evals, they run evals. And then you hear evals, they run evals. And then you hear like, oh, well, companies have evals. like, oh, well, companies have evals. like, oh, well, companies have evals. There's like online evals. And like, it There's like online evals. And like, it There's like online evals. And like, it it like the word eval is like uh more or it like the word eval is like uh more or it like the word eval is like uh more or less a meaningless word. It literally is less a meaningless word. It literally is less a meaningless word. It literally is just like you're evaluating something, just like you're evaluating something, just like you're evaluating something, right? It's like a test in some cases. right? It's like a test in some cases. right? It's like a test in some cases. It's a So, it's a little confusing. Um, It's a So, it's a little confusing. Um, It's a So, it's a little confusing. Um, I think it's helpful like I think what I think it's helpful like I think what I think it's helpful like I think what it means is that like uh it means is that like uh it means is that like uh companies start borrowing like the companies start borrowing like the companies start borrowing like the language that like labs are using and language that like labs are using and language that like labs are using and like even copying similar benchmarks, like even copying similar benchmarks, like even copying similar benchmarks, but they're doing completely different but they're doing completely different but they're doing completely different things, right? Like they have completely things, right? Like they have completely things, right? Like they have completely different tools at their disposal. Like different tools at their disposal. Like different tools at their disposal. Like what companies are, you know, like the what companies are, you know, like the what companies are, you know, like the companies that are downstream of models, companies that are downstream of models, companies that are downstream of models, uh they just have very, very different uh they just have very, very different uh they just have very, very different responsibilities than labs. Um, like responsibilities than labs. Um, like responsibilities than labs. Um, like labs are trying to make these super labs are trying to make these super labs are trying to make these super general purpose things. When they fail, general purpose things. When they fail, general purpose things. When they fail, at least on like an API level, my my say at least on like an API level, my my say at least on like an API level, my my say if they get something wrong, it's like if they get something wrong, it's like if they get something wrong, it's like it it's just different. Um, it it's just different. Um, it it's just different. Um, companies are trying to like imbue all companies are trying to like imbue all companies are trying to like imbue all this like company specific uh domain this like company specific uh domain this like company specific uh domain knowledge. Like, oh, here's the shape of knowledge. Like, oh, here's the shape of knowledge. Like, oh, here's the shape of the data and here's what all this data the data and here's what all this data the data and here's what all this data means and here's like how to access it.

means and here's like how to access it. means and here's like how to access it. Um, and so it's very, very different. Um, and so it's very, very different. Um, and so it's very, very different. Um, Um, Um, we have this like funny quiz on on the we have this like funny quiz on on the we have this like funny quiz on on the how to eval site. And um, how to eval site. And um, how to eval site. And um, it's interesting, right? Like it's kind it's interesting, right? Like it's kind it's interesting, right? Like it's kind of like one one of the questions we we of like one one of the questions we we of like one one of the questions we we think about is like, oh, are your think about is like, oh, are your think about is like, oh, are your engineers like or sorry, are your users engineers like or sorry, are your users engineers like or sorry, are your users like domain experts in the thing they're like domain experts in the thing they're like domain experts in the thing they're doing? It's again, is it almost like doing? It's again, is it almost like doing? It's again, is it almost like replacing someone or is it augmenting replacing someone or is it augmenting replacing someone or is it augmenting them? Because if you think about like them? Because if you think about like them? Because if you think about like co-pilot, you know, auto-complete style co-pilot, you know, auto-complete style co-pilot, you know, auto-complete style or cursor what I you know, tab complete or cursor what I you know, tab complete or cursor what I you know, tab complete now. now. now. Uh Uh Uh if it gets something wrong like you can if it gets something wrong like you can if it gets something wrong like you can just delete it, right? Even like Claude just delete it, right? Even like Claude just delete it, right? Even like Claude code CLI or like codex if you're an code CLI or like codex if you're an code CLI or like codex if you're an engineer, it's like it does do things engineer, it's like it does do things engineer, it's like it does do things wrong all the time. wrong all the time. wrong all the time. Um but then when you think about Um but then when you think about Um but then when you think about products like Devin, it's actually gets products like Devin, it's actually gets products like Devin, it's actually gets more interesting, right? Like the more interesting, right? Like the more interesting, right? Like the if if something messes up on on the if if something messes up on on the if if something messes up on on the Claude code side, it could be that like Claude code side, it could be that like Claude code side, it could be that like you don't have something installed you don't have something installed you don't have something installed correctly on your computer. There's a correctly on your computer. There's a correctly on your computer. There's a lot more like user error. There's a lot lot more like user error. There's a lot lot more like user error. There's a lot you leave a lot more up to the the users you leave a lot more up to the the users you leave a lot more up to the the users to get correctly. Um and I think when to get correctly. Um and I think when to get correctly. Um and I think when you start thinking about things like AI you start thinking about things like AI you start thinking about things like AI doctors, for example, it's like uh is a doctors, for example, it's like uh is a doctors, for example, it's like uh is a very very different shape of very very different shape of very very different shape of responsibility as far as like how much responsibility as far as like how much responsibility as far as like how much responsibility a user has in actually responsibility a user has in actually responsibility a user has in actually getting things correctly. Um so, getting things correctly. Um so, getting things correctly. Um so, anyways, it's kind of funny.

anyways, it's kind of funny. anyways, it's kind of funny. Um Um Um just kind of breaking this down. So, we just kind of breaking this down. So, we just kind of breaking this down. So, we think about the ceiling as like what is think about the ceiling as like what is think about the ceiling as like what is the best thing like craziest capability, the best thing like craziest capability, the best thing like craziest capability, emergent capability that your your emergent capability that your your emergent capability that your your product or agent is capable of. Like product or agent is capable of. Like product or agent is capable of. Like things that people would just not expect things that people would just not expect things that people would just not expect that it could do. that it could do. that it could do. And then the floor is like what is the And then the floor is like what is the And then the floor is like what is the worst thing your agent can do? Like worst thing your agent can do? Like worst thing your agent can do? Like recommend a competitor or like delete a recommend a competitor or like delete a recommend a competitor or like delete a bunch of data or like accidentally send bunch of data or like accidentally send bunch of data or like accidentally send a you know, AI slop email to a customer a you know, AI slop email to a customer a you know, AI slop email to a customer because it like technically had access because it like technically had access because it like technically had access to like your email or something. Um to like your email or something. Um to like your email or something. Um and and and again, I think that like again, I think that like again, I think that like the floor is very interesting because I the floor is very interesting because I the floor is very interesting because I think that that is the thing that like think that that is the thing that like think that that is the thing that like breaks user trust. This is the thing breaks user trust. This is the thing breaks user trust. This is the thing that like the reason why people uh like that like the reason why people uh like that like the reason why people uh like if you think about the worst things that if you think about the worst things that if you think about the worst things that could start happening in society, could start happening in society, could start happening in society, whether that's and and things we've whether that's and and things we've whether that's and and things we've already seen um already seen um already seen um whether that's the uh you know, like 400 whether that's the uh you know, like 400 whether that's the uh you know, like 400 kind of psychophancy kind of psychophancy kind of psychophancy um or uh things in that vein, a lot of um or uh things in that vein, a lot of um or uh things in that vein, a lot of it is more on like the floor side rather it is more on like the floor side rather it is more on like the floor side rather than the capability side. than the capability side. than the capability side. Um and anyways, I could talk about that Um and anyways, I could talk about that Um and anyways, I could talk about that for a long time. for a long time. for a long time. Um Um Um I'll kind of skip this.

I'll kind of skip this. I'll kind of skip this. So, So, So, the talk obviously is going to be about the talk obviously is going to be about the talk obviously is going to be about floor raising. floor raising. floor raising. And um And um And um the first thing that we're going to talk the first thing that we're going to talk the first thing that we're going to talk about is like offline evals. Again, we about is like offline evals. Again, we about is like offline evals. Again, we keep it very simple. Um I think that keep it very simple. Um I think that keep it very simple. Um I think that like we said before, things have changed like we said before, things have changed like we said before, things have changed a lot sort of since this chatbot era. a lot sort of since this chatbot era. a lot sort of since this chatbot era. The sort of like, "Oh, you just uh you The sort of like, "Oh, you just uh you The sort of like, "Oh, you just uh you know, like look at, you know, string know, like look at, you know, string know, like look at, you know, string contains, you know, on the uh uh on like contains, you know, on the uh uh on like contains, you know, on the uh uh on like the text output or something." Or even the text output or something." Or even the text output or something." Or even like the style of eval tools that have like the style of eval tools that have like the style of eval tools that have like a prompt playground, like this sort like a prompt playground, like this sort like a prompt playground, like this sort of thing. Like I actually don't know of thing. Like I actually don't know of thing. Like I actually don't know many companies that use some sort of many companies that use some sort of many companies that use some sort of like managed prompt like in the cloud like managed prompt like in the cloud like managed prompt like in the cloud anymore. There's like one or two I can anymore. There's like one or two I can anymore. There's like one or two I can think of. Um and the reality is just think of. Um and the reality is just think of. Um and the reality is just like the prompt is actually like the like the prompt is actually like the like the prompt is actually like the whole thing now. It's like all the code. whole thing now. It's like all the code. whole thing now. It's like all the code. It's all It's your whole harness. It's It's all It's your whole harness. It's It's all It's your whole harness. It's like everything you're connect like It's like everything you're connect like It's like everything you're connect like It's not just like some string where you tell not just like some string where you tell not just like some string where you tell your the agent what to do. Um and so your the agent what to do. Um and so your the agent what to do. Um and so what I think that means is the evals what I think that means is the evals what I think that means is the evals themselves actually should look a lot themselves actually should look a lot themselves actually should look a lot more like code. In other words, like a more like code. In other words, like a more like code. In other words, like a lot more like tests. Whether that's unit lot more like tests. Whether that's unit lot more like tests. Whether that's unit tests, whether that's end-to-end tests. tests, whether that's end-to-end tests. tests, whether that's end-to-end tests. Um they should look a lot more like Um they should look a lot more like Um they should look a lot more like tests. Um tests. Um tests. Um uh Century has this uh a really cool uh Century has this uh a really cool uh Century has this uh a really cool package called like Vitest evals. It's package called like Vitest evals. It's package called like Vitest evals. It's literally just like Vitest with like literally just like Vitest with like literally just like Vitest with like some syntactic sugar on top. Um OpenAI some syntactic sugar on top. Um OpenAI some syntactic sugar on top. Um OpenAI calls this like macro evals. And um calls this like macro evals. And um calls this like macro evals. And um again, I don't think it really matters again, I don't think it really matters again, I don't think it really matters what you call it, but like essentially what you call it, but like essentially what you call it, but like essentially run tests on your agent uh locally run tests on your agent uh locally run tests on your agent uh locally uh is is is the advice. And keep these uh is is is the advice. And keep these uh is is is the advice. And keep these evals as code. Um and again, as much as evals as code. Um and again, as much as evals as code. Um and again, as much as possible, like the I I don't see a lot

possible, like the I I don't see a lot possible, like the I I don't see a lot of companies using the sort of like of companies using the sort of like of companies using the sort of like prompt prompt prompt playground stuff anymore because of how playground stuff anymore because of how playground stuff anymore because of how the shape of agents has really changed. the shape of agents has really changed. the shape of agents has really changed. Uh Uh Uh When when we think about raising the When when we think about raising the When when we think about raising the floor, we think about really like three floor, we think about really like three floor, we think about really like three things. One is that like discovering all things. One is that like discovering all things. One is that like discovering all these like unknown issues that you have these like unknown issues that you have these like unknown issues that you have in your app, like things you're just in your app, like things you're just in your app, like things you're just like not seeing. That's one. like not seeing. That's one. like not seeing. That's one. Two is that for each issue, you really Two is that for each issue, you really Two is that for each issue, you really need to know two things. You need to need to know two things. You need to need to know two things. You need to know when it actually started, and you know when it actually started, and you know when it actually started, and you need to know how many people it affects. need to know how many people it affects. need to know how many people it affects. It sounds like obvious, but I promise It sounds like obvious, but I promise It sounds like obvious, but I promise you that like in the day-to-day of you that like in the day-to-day of you that like in the day-to-day of actually like having an agent, uh you're actually like having an agent, uh you're actually like having an agent, uh you're going to get like, you know, you already going to get like, you know, you already going to get like, you know, you already get thousands of people like, "Oh, I saw get thousands of people like, "Oh, I saw get thousands of people like, "Oh, I saw this weird thing. I saw this weird this weird thing. I saw this weird this weird thing. I saw this weird thing." So, again, the first thing is thing." So, again, the first thing is thing." So, again, the first thing is like, "Is this new?" Cuz if it's not like, "Is this new?" Cuz if it's not like, "Is this new?" Cuz if it's not new, like I probably like I'm going to new, like I probably like I'm going to new, like I probably like I'm going to care about it less. If I if if if I tell care about it less. If I if if if I tell care about it less. If I if if if I tell you like, "Hey, look, this issue started you like, "Hey, look, this issue started you like, "Hey, look, this issue started yesterday." or this issue started like yesterday." or this issue started like yesterday." or this issue started like three or four days ago, suddenly like three or four days ago, suddenly like three or four days ago, suddenly like your mind starts turning and you're your mind starts turning and you're your mind starts turning and you're like, "Oh, what did I do? Like, what like, "Oh, what did I do? Like, what like, "Oh, what did I do? Like, what what what what changed, right? Did we what what what changed, right? Did we what what what changed, right? Did we change model? Did we change you know, so change model? Did we change you know, so change model? Did we change you know, so something else um downstream?" something else um downstream?" something else um downstream?" And again, the second one is like And again, the second one is like And again, the second one is like percent of users. Like, if I'm uh percent of users. Like, if I'm uh percent of users. Like, if I'm uh knowing that it happened to three users knowing that it happened to three users knowing that it happened to three users versus 100,000 users just is uh versus 100,000 users just is uh versus 100,000 users just is uh critical. Because again, I think agents critical. Because again, I think agents critical. Because again, I think agents will have an infinite number of will have an infinite number of will have an infinite number of problems. That's sort of like the the problems. That's sort of like the the problems. That's sort of like the the great and terrible thing about them.

great and terrible thing about them. great and terrible thing about them. It's like by they're like these little It's like by they're like these little It's like by they're like these little stochastic, you know, crazy things stochastic, you know, crazy things stochastic, you know, crazy things exploring everywhere. And so, you just exploring everywhere. And so, you just exploring everywhere. And so, you just uh in order to even start making things uh in order to even start making things uh in order to even start making things better, you you really need to need to better, you you really need to need to better, you you really need to need to know these two things. When it started know these two things. When it started know these two things. When it started and percent of users. and percent of users. and percent of users. Um Um Um and I think also another question we get and I think also another question we get and I think also another question we get a lot is like around like, "Oh, like a lot is like around like, "Oh, like a lot is like around like, "Oh, like I, you know, what should I be doing?" I, you know, what should I be doing?" I, you know, what should I be doing?" And like, the first question I always And like, the first question I always And like, the first question I always ask people is like, "How many users do ask people is like, "How many users do ask people is like, "How many users do you have?" Like, we have customers with you have?" Like, we have customers with you have?" Like, we have customers with millions of users and we have customers millions of users and we have customers millions of users and we have customers with like five. And the real And like, with like five. And the real And like, with like five. And the real And like, to be clear, like to be clear, like to be clear, like uh uh uh customers with five users like customers with five users like customers with five users like especially if let's say it's like an especially if let's say it's like an especially if let's say it's like an internal um app in an enterprise context internal um app in an enterprise context internal um app in an enterprise context where it's like, you know, where it's like, you know, where it's like, you know, giving like very critical information. giving like very critical information. giving like very critical information. Like, it could be very very important to Like, it could be very very important to Like, it could be very very important to get well uh or sorry, to get correctly, get well uh or sorry, to get correctly, get well uh or sorry, to get correctly, but but but it does mean you just like should be it does mean you just like should be it does mean you just like should be have to have a radically different uh have to have a radically different uh have to have a radically different uh approach. Like, so for example, on the approach. Like, so for example, on the approach. Like, so for example, on the like, you know, uh let's say like 10, like, you know, uh let's say like 10, like, you know, uh let's say like 10, 20, 100 million, you know, messages a 20, 100 million, you know, messages a 20, 100 million, you know, messages a day side of things, like experiments day side of things, like experiments day side of things, like experiments become extremely valuable. Uh if you become extremely valuable. Uh if you become extremely valuable. Uh if you have a free tier, you can like have a free tier, you can like have a free tier, you can like uh run experiments on a very small uh run experiments on a very small uh run experiments on a very small sample of your free tier. Um and that sample of your free tier. Um and that sample of your free tier. Um and that can just be extremely extremely useful.

can just be extremely extremely useful. can just be extremely extremely useful. Um obviously, if you have five or 10 Um obviously, if you have five or 10 Um obviously, if you have five or 10 users, like uh, I would not recommend, users, like uh, I would not recommend, users, like uh, I would not recommend, you know, experiments or AB tests, etc. you know, experiments or AB tests, etc. you know, experiments or AB tests, etc. Um, so, uh, this is one of those things Um, so, uh, this is one of those things Um, so, uh, this is one of those things that, like, uh, uh, that, like, uh, uh, that, like, uh, uh, really, really depends on the person. really, really depends on the person. really, really depends on the person. Um, what I want to talk about now, before we what I want to talk about now, before we get into Q&A, get into Q&A, get into Q&A, are like three very, very, very, very are like three very, very, very, very are like three very, very, very, very tactical lessons on the sort of like tactical lessons on the sort of like tactical lessons on the sort of like issue, uh, discovery and analysis side. issue, uh, discovery and analysis side. issue, uh, discovery and analysis side. These are like three things that I I've These are like three things that I I've These are like three things that I I've never heard anyone talk about, like, never heard anyone talk about, like, never heard anyone talk about, like, three things that we've sort of just three things that we've sort of just three things that we've sort of just discovered from first principles as we discovered from first principles as we discovered from first principles as we do stuff at Raindrop. So, any do stuff at Raindrop. So, any do stuff at Raindrop. So, any competitors in the audience, please pay competitors in the audience, please pay competitors in the audience, please pay attention. This is very important. attention. This is very important. attention. This is very important. >> [snorts] >> [snorts] >> [snorts] >> Um, >> Um, >> Um, the first one is that clusters are not the first one is that clusters are not the first one is that clusters are not issues. issues. issues. So, um, the sort of like naive approach So, um, the sort of like naive approach So, um, the sort of like naive approach that we've seen either customers or that we've seen either customers or that we've seen either customers or also, sometimes competitors, also, sometimes competitors, also, sometimes competitors, uh, take is like, well, you just take uh, take is like, well, you just take uh, take is like, well, you just take all the traces and you just cluster it, all the traces and you just cluster it, all the traces and you just cluster it, right? And you get these like clusters. right? And you get these like clusters. right? And you get these like clusters. It could be like useful from like an It could be like useful from like an It could be like useful from like an analysis, uh, you know, like, how we do analysis, uh, you know, like, how we do analysis, uh, you know, like, how we do things like error analysis, like, the things like error analysis, like, the things like error analysis, like, the the the this, you know, the the this, you know, the the this, you know, finding these clusters of things. It finding these clusters of things. It finding these clusters of things. It could be useful to see like, well, could be useful to see like, well, could be useful to see like, well, what's going on in your data, right?

what's going on in your data, right? what's going on in your data, right? Going from like a bunch of logs to like Going from like a bunch of logs to like Going from like a bunch of logs to like something. something. something. Um, Um, Um, the problem is that, like, and and the problem is that, like, and and the problem is that, like, and and again, I I have here, it's useful for again, I I have here, it's useful for again, I I have here, it's useful for one-off analysis, but it just doesn't one-off analysis, but it just doesn't one-off analysis, but it just doesn't really scale well. Um, and there's like really scale well. Um, and there's like really scale well. Um, and there's like a very good reason why we also, you a very good reason why we also, you a very good reason why we also, you know, if you think about like normal know, if you think about like normal know, if you think about like normal telemetry, we we we try to think a lot telemetry, we we we try to think a lot telemetry, we we we try to think a lot about like normal telemetry, what are about like normal telemetry, what are about like normal telemetry, what are the analogies? There's a reason why you the analogies? There's a reason why you the analogies? There's a reason why you don't sort of like take all of your, you don't sort of like take all of your, you don't sort of like take all of your, you know, normal logs and just like start know, normal logs and just like start know, normal logs and just like start clustering it, right? Because clustering it, right? Because clustering it, right? Because when you're building software, you you when you're building software, you you when you're building software, you you need to know like when something need to know like when something need to know like when something started, um, you need to know how much started, um, you need to know how much started, um, you need to know how much it's grown. Um, those things really it's grown. Um, those things really it's grown. Um, those things really matter. So, again, with clusters, it's matter. So, again, with clusters, it's matter. So, again, with clusters, it's very, very hard to reliably, uh, track very, very hard to reliably, uh, track very, very hard to reliably, uh, track over time. Um, like uh, like again, it's over time. Um, like uh, like again, it's over time. Um, like uh, like again, it's like called like temporal clustering and like called like temporal clustering and like called like temporal clustering and there's like research in this, but like, there's like research in this, but like, there's like research in this, but like, it's pretty hard, um, to do reliably. it's pretty hard, um, to do reliably. it's pretty hard, um, to do reliably. You also just like don't have control of You also just like don't have control of You also just like don't have control of boundaries. boundaries. boundaries. Um and this also changes a lot depending Um and this also changes a lot depending Um and this also changes a lot depending on your product. Like um what you on your product. Like um what you on your product. Like um what you consider to be like, you know, the same consider to be like, you know, the same consider to be like, you know, the same issue or not um issue or not um issue or not um is actually very, very unique to every is actually very, very unique to every is actually very, very unique to every company.

company. company. And so you sort of will get these like And so you sort of will get these like And so you sort of will get these like kind of weird clusters. Like, you know, kind of weird clusters. Like, you know, kind of weird clusters. Like, you know, you can imagine each of these is like, you can imagine each of these is like, you can imagine each of these is like, "Oh, wrong price quoted and wrong wrong "Oh, wrong price quoted and wrong wrong "Oh, wrong price quoted and wrong wrong refund calculated." Are like actually refund calculated." Are like actually refund calculated." Are like actually like you'll get a cluster like, you like you'll get a cluster like, you like you'll get a cluster like, you know, know, know, uh uh uh you know, price issues or something. And you know, price issues or something. And you know, price issues or something. And it's like, yeah, sort of. But like it's like, yeah, sort of. But like it's like, yeah, sort of. But like actually these could have like extremely actually these could have like extremely actually these could have like extremely different root causes, right? So so different root causes, right? So so different root causes, right? So so price issues is or like, you know, uh price issues is or like, you know, uh price issues is or like, you know, uh issues calculating is like, you know, as issues calculating is like, you know, as issues calculating is like, you know, as a cluster it's not really that useful. a cluster it's not really that useful. a cluster it's not really that useful. Um and it also again doesn't doesn't Um and it also again doesn't doesn't Um and it also again doesn't doesn't really tell you the things that, you really tell you the things that, you really tell you the things that, you know, we talked about needing. know, we talked about needing. know, we talked about needing. Um Um Um So uh yes. Uh So uh yes. Uh So uh yes. Uh Last one here is going to be uh code Last one here is going to be uh code Last one here is going to be uh code mode actually really scales. Like you've mode actually really scales. Like you've mode actually really scales. Like you've heard about code mode in the context of heard about code mode in the context of heard about code mode in the context of MCPs. MCPs. MCPs. Um Um Um I highly recommend just uh trying to I highly recommend just uh trying to I highly recommend just uh trying to apply this to traces. Like you can just apply this to traces. Like you can just apply this to traces. Like you can just write uh these classifiers and you can write uh these classifiers and you can write uh these classifiers and you can write them and you can run them in a write them and you can run them in a write them and you can run them in a sandbox and you can run them at sandbox and you can run them at sandbox and you can run them at production scale. Um we have a you know, production scale. Um we have a you know, production scale. Um we have a you know, feature that makes this easier, but like feature that makes this easier, but like feature that makes this easier, but like you can do this. So I highly recommend you can do this. So I highly recommend you can do this. So I highly recommend it.

it. it. Last lesson here is that agents are Last lesson here is that agents are Last lesson here is that agents are very, very bad at anomaly detection. So very, very bad at anomaly detection. So very, very bad at anomaly detection. So don't ask your agent to find anomalies. don't ask your agent to find anomalies. don't ask your agent to find anomalies. Uh ask it to investigate anomalies Uh ask it to investigate anomalies Uh ask it to investigate anomalies you've already found. So uh what I mean you've already found. So uh what I mean you've already found. So uh what I mean is like pull out as many deterministic is like pull out as many deterministic is like pull out as many deterministic things as you can like keyword things as you can like keyword things as you can like keyword frequency, right? So if you see a spike frequency, right? So if you see a spike frequency, right? So if you see a spike in like a keyword in like a keyword in like a keyword it doesn't necessarily mean that there's it doesn't necessarily mean that there's it doesn't necessarily mean that there's an issue, but it does mean that you can an issue, but it does mean that you can an issue, but it does mean that you can uh it's like something more tangible uh it's like something more tangible uh it's like something more tangible tractable that you can have an agent tractable that you can have an agent tractable that you can have an agent actually investigate. actually investigate. actually investigate. Um Um Um And I'm going to skip through the rest And I'm going to skip through the rest And I'm going to skip through the rest because we're tight on time. because we're tight on time. because we're tight on time. And I lied to you uh which is that we're And I lied to you uh which is that we're And I lied to you uh which is that we're not going to have enough time for Q&A not going to have enough time for Q&A not going to have enough time for Q&A because I only have a minute left. But, because I only have a minute left. But, because I only have a minute left. But, what I'd love if you could do is, uh, what I'd love if you could do is, uh, what I'd love if you could do is, uh, find me after. Um, I'll be around for find me after. Um, I'll be around for find me after. Um, I'll be around for the next hour and, uh, let's just talk. the next hour and, uh, let's just talk. the next hour and, uh, let's just talk. It's probably a better format than It's probably a better format than It's probably a better format than standing up here and it'll be hard to standing up here and it'll be hard to standing up here and it'll be hard to hear your questions anyway. So, uh, hear your questions anyway. So, uh, hear your questions anyway. So, uh, yeah, thank you guys so much.