Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse

Hello everyone. Hello everyone. Welcome to the workshop on Welcome to the workshop on Welcome to the workshop on continuous continuous agent improvement using agent improvement using Langfuse. You can now Langfuse. You can now Langfuse. You can now go to go to go to langfuse.com/workshop. There are langfuse.com/workshop. There are langfuse.com/workshop. There are steps that we steps that we steps that we will cover today, will cover today, will cover today, and you can go through them again later and you can go through them again later and you can go through them again later at home on your own at home on your own at home on your own , and , and , and there are also additional there are also additional there are also additional steps that we steps that we steps that we will not go will not go will not go through today. We will through today. We will through today. We will show this QR code show this QR code show this QR code during the presentation. during the presentation. So, continuous So, continuous improvement of improvement of improvement of agents. What does this agents. What does this agents. What does this mean? We'll talk a lot mean? We'll talk a lot mean? We'll talk a lot about the about the about the different processes that different processes that different processes that teams teams teams use to use to use to improve their improve their improve their agents, identify agents, identify agents, identify changes, and changes, and changes, and improve them in improve them in improve them in different ways, and you different ways, and you different ways, and you can combine that into a can combine that into a can combine that into a continuous loop. As continuous loop. As continuous loop. As you know, in the SaaS industry we you know, in the SaaS industry we you know, in the SaaS industry we really love cycles. really love cycles. This process and the set of This process and the set of these processes together these processes together these processes together are often called AI are often called AI engineering. So, engineering. So, engineering. So, what is AI engineering, what is AI engineering, what is AI engineering, why is it needed, and why is it needed, and why is it needed, and how does it how does it how does it differ from differ from differ from traditional traditional traditional engineering? The main engineering? The main engineering? The main difference is difference is difference is that AI agents are not that AI agents are not that AI agents are not deterministic.

deterministic. Your system may Your system may appear to be working, appear to be working, appear to be working, while the responses or while the responses or while the responses or actions taken by the actions taken by the actions taken by the agent are agent are agent are incorrect. Sometimes incorrect. Sometimes incorrect. Sometimes they are simply they are simply they are simply wrong. Sometimes they are wrong. Sometimes they are wrong. Sometimes they are also not of good also not of good also not of good quality. This is what quality. This is what quality. This is what AI engineering solves. AI engineering solves. For those For those unfamiliar, Langfuse unfamiliar, Langfuse unfamiliar, Langfuse offers a platform offers a platform offers a platform to implement these to implement these to implement these processes; It is processes; It is processes; It is open open open source, model- source, model- agnostic, and agnostic, and agnostic, and framework- framework- agnostic, so you agnostic, so you agnostic, so you can can can use it with use it with use it with any existing any existing any existing infrastructure. There are infrastructure. There are infrastructure. There are large companies that large companies that large companies that already already already use it every day, use it every day, use it every day, but we also have a but we also have a but we also have a large community of large community of large community of smaller smaller smaller individual individual individual users who users who users who help us a lot help us a lot help us a lot with different with different agent setups and projects. This is definitely one definitely one definitely one way way way we stay we stay we stay on top of where on top of where on top of where this rapidly evolving industry is headed this rapidly evolving industry is headed . By the way, this . By the way, this workshop today workshop today workshop today is being held by is being held by is being held by Annabel and I, and Mark, our Annabel and I, and Mark, our Annabel and I, and Mark, our founder, is also somewhere in the founder, is also somewhere in the founder, is also somewhere in the room. Here he is. We will room. Here he is. We will room. Here he is. We will also walk around the also walk around the also walk around the room to help room to help room to help you during the workshop you during the workshop you during the workshop if you if you if you need anything. So, AI need anything. So, AI engineering—we engineering—we engineering—we like to demonstrate like to demonstrate like to demonstrate or visualize a or visualize a or visualize a set of processes that set of processes that set of processes that humans perform humans perform humans perform using this using this using this graph. We graph. We graph. We call this the AI call this the AI engineering cycle. The exact

engineering cycle. The exact engineering cycle. The exact setup depends a bit setup depends a bit setup depends a bit on the on the companies' infrastructure. It depends companies' infrastructure. It depends on different needs, on different needs, on different needs, but in general they can be but in general they can be but in general they can be divided into these five divided into these five divided into these five steps. And the main thing to steps. And the main thing to steps. And the main thing to remember here is that remember here is that we have two big we have two big we have two big parts. We have an parts. We have an parts. We have an online part, online part, online part, tracing and tracing and tracing and monitoring, and an offline monitoring, and an offline part, where we part, where we part, where we create create create datasets, datasets, datasets, experiment, and experiment, and experiment, and evaluate outside of a evaluate outside of a evaluate outside of a production production production environment. And, environment. And, today we will today we will focus focus focus mainly on the online mainly on the online part, i.e. part, i.e. part, i.e. tracing and tracing and tracing and monitoring. So, monitoring. So, monitoring. So, for tracing, you for tracing, you for tracing, you configure configure configure the tooling, the tooling, the tooling, deploy the application deploy the application deploy the application in a production in a production in a production environment, and see environment, and see environment, and see what traces are coming in, what traces are coming in, what traces are coming in, what your agent is doing, and what your agent is doing, and what your agent is doing, and how it interacts with how it interacts with how it interacts with users. Based on users. Based on users. Based on this, you this, you this, you will will will monitor these routes. monitor these routes. monitor these routes. Monitoring, on the one Monitoring, on the one Monitoring, on the one hand, is automatic hand, is automatic hand, is automatic thanks to evaluators and thanks to evaluators and thanks to evaluators and dashboards, but dashboards, but the role of the role of human feedback should not be underestimated here either. You will human feedback should not be underestimated here either. You will human feedback should not be underestimated here either. You will also collect also collect also collect implicit or explicit implicit or explicit implicit or explicit user feedback user feedback , as well as manually , as well as manually , as well as manually review review review some samples of these some samples of these some samples of these tracks. This together, I don't tracks. This together, I don't tracks. This together, I don't know if you see, oh, the know if you see, oh, the know if you see, oh, the yellow uh yellow uh yellow uh rectangle is rectangle is rectangle is actually pretty actually pretty actually pretty clear, great. Uh, this clear, great. Uh, this clear, great. Uh, this together—the online together—the online part of the part of the part of the AI ​​development cycle—is already AI ​​development cycle—is already AI ​​development cycle—is already a lot. Many teams a lot. Many teams that do not work in that do not work in that do not work in high- high- high- risk industries can risk industries can risk industries can simply start with these simply start with these simply start with these two steps and, two steps and, two steps and, based on the based on the based on the results of results of results of trace monitoring, trace monitoring, trace monitoring, make simple make simple make simple fixes or fixes or fixes or improvements to their improvements to their improvements to their agents, deploy agents, deploy agents, deploy them in a production them in a production them in a production environment, and see

environment, and see environment, and see new new new traces arrive. This is a great traces arrive. This is a great traces arrive. This is a great first step, it will first step, it will first step, it will take you take you take you very far and is a very far and is a very far and is a great way to great way to great way to quickly quickly quickly improve your improve your improve your agents. As your agents. As your agents. As your product becomes a little product becomes a little product becomes a little more mature, your more mature, your more mature, your infrastructure infrastructure infrastructure evolves, or you evolves, or you evolves, or you work in a work in a work in a high- high- high- risk environment where any risk environment where any risk environment where any mistake has bad mistake has bad mistake has bad consequences, it's worth consequences, it's worth consequences, it's worth working on the offline working on the offline part as well. This is necessary part as well. This is necessary part as well. This is necessary for for for structured structured structured testing. So, testing. So, testing. So, your datasets are your datasets are your datasets are your structured your structured your structured test cases. They test cases. They test cases. They will be largely will be largely will be largely inspired by what you inspired by what you inspired by what you see during see during see during monitoring, and when monitoring, and when monitoring, and when you want to make you want to make you want to make changes, you will run changes, you will run changes, you will run an experiment on your an experiment on your an experiment on your datasets, datasets, datasets, evaluate the results of the evaluate the results of the evaluate the results of the experiments, and only experiments, and only experiments, and only if the results if the results if the results are better and you are better and you are better and you haven't made any haven't made any haven't made any regressions, then you regressions, then you regressions, then you deploy it to a deploy it to a deploy it to a production production production environment. Before environment. Before environment. Before we move on to we move on to we move on to the workshop, we want to the workshop, we want to focus very quickly on two focus very quickly on two parts of the online parts of the online cycle. Uh, it's cycle. Uh, it's cycle. Uh, it's tracing and tracing and tracing and monitoring, because monitoring, because monitoring, because it's really important. It it's really important. It it's really important. It all starts with all starts with all starts with your route. Because your route. Because your route. Because if your track is if your track is if your track is bad, you won't be able bad, you won't be able bad, you won't be able to detect everything during the to detect everything during the to detect everything during the monitoring phase.

monitoring phase. monitoring phase. If you don't detect If you don't detect If you don't detect everything during everything during everything during monitoring, your monitoring, your monitoring, your datasets datasets datasets won't be as high quality won't be as high quality won't be as high quality as they could be. You won't as they could be. You won't as they could be. You won't make the most make the most make the most informed decisions informed decisions informed decisions about what about what about what to evaluate, etc. Therefore, it is to evaluate, etc. Therefore, it is to evaluate, etc. Therefore, it is very important very important very important to take enough to take enough to take enough time to time to time to make sure your make sure your make sure your tracks are set up tracks are set up tracks are set up well. So, how do well. So, how do well. So, how do you build a track? A you build a track? A trace consists of trace consists of observations, and observations, and observations, and observations are the observations are the observations are the different steps that an different steps that an different steps that an agent takes agent takes agent takes to produce a response to produce a response to produce a response for the user. These for the user. These for the user. These could be could be could be LLM calls, LLM calls, LLM calls, tool calls, or tool calls, or tool calls, or other things. They other things. They other things. They can also be can also be can also be nested. Each nested. Each nested. Each observation also observation also observation also contains a lot of contains a lot of contains a lot of added data, and this added data, and this added data, and this depends on the type of depends on the type of depends on the type of observation. For an observation. For an observation. For an LLM call, this is LLM call, this is LLM call, this is typically duration, typically duration, typically duration, token cost, token cost, token cost, token usage token usage , model used, , model used, , model used, and prompt version. and prompt version. Anything that comes to Anything that comes to mind that will help mind that will help mind that will help you assess the quality of you assess the quality of you assess the quality of your LLM challenge. And your LLM challenge. And your LLM challenge. And for for for tool calls, things tool calls, things tool calls, things can be completely can be completely can be completely different. For example, different. For example, different. For example, during a web search, during a web search, during a web search, one of the data items might one of the data items might one of the data items might be the query be the query be the query used for used for used for that search. So this is that search. So this is that search. So this is very important so that very important so that you can you can monitor effectively. Regarding monitor effectively. Regarding monitoring—it has monitoring—it has monitoring—it has several different several different several different functions. First, you functions. First, you functions. First, you want to have aggregate want to have aggregate want to have aggregate metrics to see the metrics to see the metrics to see the overall health of your overall health of your overall health of your system over time, system over time, system over time, such as such as such as system cost and quality.

system cost and quality. So, if you've So, if you've set up set up set up raters to raters to raters to measure the quality of measure the quality of measure the quality of responses, you want to responses, you want to responses, you want to make sure it make sure it make sure it doesn't degrade over time doesn't degrade over time . The second very . The second very . The second very important aspect is important aspect is important aspect is detecting signals detecting signals detecting signals through through through user feedback. user feedback. user feedback. As your As your As your application application application scales, you scales, you scales, you will have a lot of traces, will have a lot of traces, will have a lot of traces, and it is simply impossible to review everything and it is simply impossible to review everything and it is simply impossible to review everything . And . And . And user reviews user reviews are a great way are a great way are a great way to highlight exactly those to highlight exactly those to highlight exactly those cases that are worth cases that are worth cases that are worth paying attention to. The third paying attention to. The third is is is feedback and feedback and feedback and evaluation. So, you'll evaluation. So, you'll evaluation. So, you'll set up set up set up automatic automatic automatic raters, and also raters, and also raters, and also add manual add manual add manual markup so you don't markup so you don't markup so you don't miss anything miss anything miss anything new. One important new. One important new. One important point: you have point: you have point: you have automatic automatic automatic evaluators. On the one evaluators. On the one evaluators. On the one hand, they are very hand, they are very hand, they are very scalable. You scalable. You scalable. You can continue to can continue to can continue to apply them to apply them to apply them to all your traces. On all your traces. On all your traces. On the other hand, you are the other hand, you are the other hand, you are unlikely to catch unlikely to catch unlikely to catch very novel or nuanced very novel or nuanced very novel or nuanced cases that you have cases that you have cases that you have n't thought of yourself. Although n't thought of yourself. Although n't thought of yourself. Although user feedback user feedback user feedback and manual markup and manual markup and manual markup scale a little scale a little scale a little worse, that's where you'll worse, that's where you'll worse, that's where you'll discover things you discover things you discover things you never knew you had.

never knew you had. So that's what we'll focus on today. My focus on today. My colleague Annabelle colleague Annabelle colleague Annabelle will review a practical will review a practical will review a practical example, and Mark and I example, and Mark and I example, and Mark and I will walk around the room will walk around the room . If you need . If you need . If you need help, just help, just help, just raise your hand. raise your hand. raise your hand. So, today's So, today's So, today's focus, as focus, as focus, as Lotta already mentioned, is to Lotta already mentioned, is to Lotta already mentioned, is to work effectively with work effectively with work effectively with your your your application's traces. And also application's traces. And also application's traces. And also how to decide what how to decide what how to decide what exactly you want to exactly you want to exactly you want to monitor. So, what monitor. So, what monitor. So, what exactly interests you about exactly interests you about exactly interests you about your application, your application, your application, not in general. And also how not in general. And also how not in general. And also how you can you can you can use use use agents for agents for agents for programming to programming to programming to speed up your work a bit speed up your work a bit speed up your work a bit , but at the same time , but at the same time , but at the same time where it is worth staying where it is worth staying where it is worth staying involved and not involved and not involved and not letting the agent letting the agent letting the agent do everything on do everything on do everything on its own. So its own. So its own. So let's dive into let's dive into let's dive into it. And to do that, we it. And to do that, we it. And to do that, we first want to first want to first want to take a look at our take a look at our take a look at our demo demo demo application for application for application for today. We today. We today. We have prepared for you have prepared for you have prepared for you Specs—an IT Specs—an IT Specs—an IT technical support agent technical support agent technical support agent for dad. This for dad. This for dad. This mini-repository mini-repository mini-repository grew out of a small grew out of a small grew out of a small project for my project for my project for my father, because he was father, because he was father, because he was constantly asking constantly asking constantly asking me on WhatsApp how to me on WhatsApp how to me on WhatsApp how to set up his phone, set up his phone, set up his phone, as well as how to as well as how to as well as how to deal with a deal with a deal with a printer or printer or printer or computer. So I computer. So I computer. So I created this created this created this little agent little agent little agent that, in a productive that, in a productive that, in a productive version, responds to version, responds to version, responds to him via WhatsApp. In him via WhatsApp. In him via WhatsApp. In this case, we this case, we this case, we have a nice have a nice have a nice little interface.

little interface. little interface. As you can As you can As you can imagine, it works like this: imagine, it works like this: imagine, it works like this: we have a user, we have a user, we have a user, my dad or any my dad or any my dad or any other parent. Dad other parent. Dad other parent. Dad asks Specs questions asks Specs questions , and Specs has several , and Specs has several tools available. We tools available. We can get can get can get support contacts support contacts support contacts or information about or information about what phone what phone what phone Dad is using, Dad is using, Dad is using, what what what operating system he has, operating system he has, operating system he has, and what apps and what apps and what apps are installed. are installed. are installed. We also have a We also have a We also have a reference library. These are frequently asked reference library. These are frequently asked reference library. These are frequently asked questions that provide questions that provide questions that provide context on how context on how context on how to solve certain to solve certain to solve certain problems. He can problems. He can problems. He can use these use these use these tools, as well as the tools, as well as the tools, as well as the OpenAI API, to generate OpenAI API, to generate OpenAI API, to generate responses to responses to responses to send to our send to our send to our dad. So this is an dad. So this is an dad. So this is an application that we're going application that we're going to look at together first, and also to look at together first, and also think about what's think about what's think about what's actually interesting actually interesting actually interesting to learn, to to learn, to to learn, to make sure it make sure it make sure it gets the support gets the support gets the support it wants. You can it wants. You can it wants. You can scan the QR code scan the QR code scan the QR code here, find the here, find the here, find the workshop repository workshop repository , and we'll , and we'll , and we'll set everything up together. So don't set everything up together. So don't set everything up together. So don't worry, we'll worry, we'll worry, we'll go through everything step by go through everything step by go through everything step by step. There is also step. There is also step. There is also documentation in documentation in documentation in case you case you case you get confused, you get confused, you get confused, you can refer to can refer to can refer to it, and if you want to it, and if you want to it, and if you want to learn more learn more learn more later, there are later, there are later, there are additional additional additional modules available. So, let modules available. So, let modules available. So, let 's go to 's go to 's go to langfuse.com/workshop. We langfuse.com/workshop. We langfuse.com/workshop. We can see that this is a can see that this is a markdown page, but if we markdown page, but if we click here on click here on click here on langfuse/langfuse workshop, we'll langfuse/langfuse workshop, we'll langfuse/langfuse workshop, we'll be taken to the be taken to the be taken to the repository where you repository where you repository where you can clone can clone can clone it into your IDE. I already it into your IDE. I already it into your IDE. I already did it. So everything is

did it. So everything is did it. So everything is ready. We see that the ready. We see that the ready. We see that the first thing to first thing to first thing to configure is, configure is, configure is, of course, the of course, the of course, the environment variables. And for this, environment variables. And for this, environment variables. And for this, let's create Let's let's create Let's Organize Organize Together on langfuse.com. Together on langfuse.com. We can proceed to We can proceed to launch the program. launch the program. launch the program. I already have an I already have an I already have an organization, but you organization, but you organization, but you can skip to can skip to can skip to launching the app, or launching the app, or if you're on this if you're on this if you're on this page, choose a page, choose a page, choose a region—it doesn't region—it doesn't region—it doesn't matter which one— matter which one— and create an and create an and create an account. Once you account. Once you account. Once you log in, you will see log in, you will see log in, you will see this page. So, a project this page. So, a project this page. So, a project will be automatically will be automatically will be automatically created for you, and you created for you, and you created for you, and you will be on this will be on this will be on this page where you can page where you can page where you can get your get your get your API keys by clicking " API keys by clicking " create new create new create new API keys". So maybe I'll API keys". So maybe I'll API keys". So maybe I'll give you a minute to give you a minute to give you a minute to go to our site and get go to our site and get go to our site and get everything set up, that's going to be the very everything set up, that's going to be the very everything set up, that's going to be the very first thing we first thing we first thing we need to do together. need to do together. And I will also show the steps here, which you will also steps here, which you will also find on our find on our find on our workshop website.

workshop website. Yes. So, quick Yes. So, quick registration. registration. registration. Creating a new Creating a new Creating a new account. And account. And account. And for those who have already for those who have already for those who have already done so, we done so, we done so, we have prepared the have prepared the have prepared the OpenAI key for today. You OpenAI key for today. You OpenAI key for today. You can can can scan it. Of course, scan it. Of course, scan it. Of course, we we we will change it later. But for will change it later. But for will change it later. But for now, this is an API key now, this is an API key now, this is an API key that you can that you can that you can use. You use. You use. You can add it to can add it to can add it to your file. And then we'll your file. And then we'll your file. And then we'll begin. Yes, begin. Yes, begin. Yes, just copy the just copy the just copy the .env.example file and .env.example file and .env.example file and navigate to it. navigate to it. Good. So, when we Good. So, when we cloned cloned cloned the repository, the first thing the repository, the first thing the repository, the first thing we need to we need to we need to do is quickly do is quickly do is quickly install the install the install the dependencies. We are dependencies. We are dependencies. We are running a running a running a TypeScript application here TypeScript application here TypeScript application here that uses the that uses the that uses the OpenAI Azure SDK. And so we OpenAI Azure SDK. And so we OpenAI Azure SDK. And so we can quickly can quickly can quickly install all the install all the install all the dependencies. They are dependencies. They are dependencies. They are available. And then we available. And then we available. And then we can run can run can run it with the command npm run dev.

it with the command npm run dev. I'll pull it out here. I'll pull it out here. It opens It opens It opens right here. I will also right here. I will also right here. I will also reveal it here. And we reveal it here. And we reveal it here. And we see our example see our example see our example application. By the way, application. By the way, application. By the way, if I'm in a bit of a hurry if I'm in a bit of a hurry , here are the steps on the , here are the steps on the langfuse.com/workshop setup page, how you langfuse.com/workshop setup page, how you can go through everything can go through everything can go through everything step by step, step by step, step by step, configuring the .env file configuring the .env file configuring the .env file and running and running and running the application. So the application. So the application. So let's see what we let's see what we let's see what we have here. We have our have here. We have our have here. We have our dad support agent dad support agent dad support agent and he is already greeting us with and he is already greeting us with and he is already greeting us with some questions. some questions. some questions. If I have a lot of If I have a lot of If I have a lot of questions, we also have questions, we also have questions, we also have examples of questions that examples of questions that examples of questions that can be asked. We can be asked. We can be asked. We may ask, "How do may ask, "How do may ask, "How do I turn on Bluetooth on my I turn on Bluetooth on my I turn on Bluetooth on my iPhone?" And Spex takes iPhone?" And Spex takes iPhone?" And Spex takes a moment, a moment, a moment, using using the tools we the tools we need in the background to need in the background to need in the background to get the answer. And get the answer. And get the answer. And then we then we then we respond to our respond to our respond to our dad in this dad in this dad in this case with step-by-step case with step-by-step case with step-by-step instructions on how to, well, instructions on how to, well, instructions on how to, well, turn on Bluetooth on the turn on Bluetooth on the turn on Bluetooth on the device. And what device. And what device. And what happened in the background now? This is happened in the background now? This is happened in the background now? This is exactly the interesting exactly the interesting exactly the interesting part that we part that we part that we want to look at. And want to look at. And want to look at. And if we now if we now if we now go back to go back to go back to the trace, and this is exactly the trace, and this is exactly the trace, and this is exactly what should happen to what should happen to what should happen to you, if you you, if you you, if you run the application run the application run the application and ask one and ask one and ask one question, then we'll question, then we'll question, then we'll see the see the see the first trace appear first trace appear . So, every line . So, every line . So, every line here is an here is an here is an observation. We observation. We observation. We also see that they also see that they also see that they have slightly different have slightly different have slightly different icons, and these icons, and these icons, and these different icons different icons different icons mean different things.

mean different things. mean different things. So if you So if you So if you click on one of click on one of click on one of those, we'll see that those, we'll see that those, we'll see that all of this originates in all of this originates in all of this originates in our, um, our, um, our, um, agent observation agent observation , and below that we have , and below that we have , and below that we have a few generations, as a few generations, as a few generations, as well as well as well as tool calls, where tool calls, where tool calls, where some some some context is derived from. Um, so this is context is derived from. Um, so this is context is derived from. Um, so this is already a fully already a fully already a fully instrumented instrumented instrumented application, and there's also a application, and there's also a application, and there's also a step-by-step guide on step-by-step guide on how to actually how to actually how to actually set it up. But set it up. But set it up. But we also have a we also have a we also have a LangChain skill that you can LangChain skill that you can LangChain skill that you can use for use for use for automatic automatic automatic configuration. And configuration. And configuration. And today we want to today we want to today we want to focus on focus on focus on working with these working with these working with these traces, rather than on traces, rather than on traces, rather than on their step-by-step their step-by-step their step-by-step configuration. So, configuration. So, configuration. So, yeah, here's our yeah, here's our yeah, here's our trace. And if we trace. And if we trace. And if we now go back to now go back to now go back to our LangChain com our LangChain com /workshop page, we can /workshop page, we can /workshop page, we can go to the go to the monitoring workshop module, which is what we monitoring workshop module, which is what we actually want to actually want to cover together today, as cover together today, as Lott already mentioned. So, in Lott already mentioned. So, in Lott already mentioned. So, in the context of our the context of our dad support app, what does monitoring actually mean? So when monitoring actually mean? So when Dad interacts with Dad interacts with Dad interacts with the app, the app, the app, of course we can't see what of course we can't see what of course we can't see what he's doing because we're he's doing because we're he's doing because we're not sitting next to not sitting next to not sitting next to him. This is the him. This is the him. This is the whole point of an agent. But whole point of an agent. But whole point of an agent. But we actually want we actually want we actually want to track what's to track what's to track what's happening. And we happening. And we happening. And we want to not just want to not just want to not just track what track what track what exactly the agent is doing exactly the agent is doing exactly the agent is doing , but also understand , but also understand , but also understand their meaning. So we their meaning. So we their meaning. So we thought about what thought about what thought about what could actually go could actually go could actually go wrong in a wrong in a wrong in a support application. And the first is that support application. And the first is that the data may not be the data may not be the data may not be consistent with consistent with consistent with the application.

the application. the application. For example, if I For example, if I For example, if I ask the question: "How do ask the question: "How do ask the question: "How do I set up I set up I set up Bluetooth?". Um, then I Bluetooth?". Um, then I Bluetooth?". Um, then I ask myself, "Okay, but ask myself, "Okay, but ask myself, "Okay, but how does this actually how does this actually how does this actually work?" Where is this button work?" Where is this button ? And whoever is asking this ? And whoever is asking this , we might want to , we might want to , we might want to catch that because catch that because catch that because maybe something wasn't maybe something wasn't clear enough or clear enough or detailed enough for the detailed enough for the detailed enough for the question to be question to be answered correctly. So let's answered correctly. So let's set this up together. set this up together. Let's start by, yes, Let's start by, yes, copying this copying this copying this Git command, Git checkout Git command, Git checkout Git command, Git checkout our our our monitoring checkpoint. And then monitoring checkpoint. And then monitoring checkpoint. And then we can we can we can go back to our go back to our go back to our IDE. And run this IDE. And run this IDE. And run this command, and now we command, and now we command, and now we see that we are moving see that we are moving see that we are moving into the monitoring phase of into the monitoring phase of into the monitoring phase of our workshop, so our workshop, so our workshop, so this is a checkpoint. this is a checkpoint. And now we want And now we want to connect these to connect these to connect these estimators to our estimators to our estimators to our application. So, if we application. So, if we application. So, if we scroll down a bit scroll down a bit scroll down a bit , we'll see that the , we'll see that the , we'll see that the first thing we first thing we first thing we need to do need to do need to do together is together is together is set up the set up the set up the LangFuse evaluator model. So I'll LangFuse evaluator model. So I'll LangFuse evaluator model. So I'll move on to our move on to our move on to our application here. I'll close the application here. I'll close the tracing part. And I'll tracing part. And I'll go to the go to the go to the evaluator tab. Here I see evaluator tab. Here I see evaluator tab. Here I see that I can start working that I can start working that I can start working with evaluators, and there are two with evaluators, and there are two with evaluators, and there are two different possibilities. I different possibilities. I different possibilities. I can configure can configure can configure software evaluators software evaluators software evaluators for more for more for more deterministic deterministic deterministic checks of my checks of my checks of my application, and I can also application, and I can also application, and I can also configure configure configure LLM-based evaluators.

LLM-based evaluators. They are more They are more suitable for suitable for suitable for implicit things that you implicit things that you implicit things that you can get from the can get from the can get from the conversation history and conversation history and conversation history and actually want to actually want to actually want to reveal there. We also reveal there. We also reveal there. We also see that there are some see that there are some see that there are some templates here, and our templates here, and our templates here, and our user disagreement template user disagreement template is one of the ones we is one of the ones we is one of the ones we provide here. So provide here. So provide here. So if we click if we click if we click here, we'll see that here, we'll see that here, we'll see that we need to we need to we need to set up an LLM set up an LLM connection. I'm connection. I'm connection. I'm going to go here and going to go here and going to go here and stop stop stop the demo really quickly because the demo really quickly because the demo really quickly because you shouldn't see the you shouldn't see the you shouldn't see the API key. The laptop API key. The laptop API key. The laptop is out of battery. So, is out of battery. So, is out of battery. So, ultimately, it's an ultimately, it's an ultimately, it's an opportunity opportunity opportunity to use a certain to use a certain to use a certain model to assess model to assess model to assess what's happening what's happening . And so that MGM. Give me a . And so that MGM. Give me a . And so that MGM. Give me a second. Good. So, second. Good. So, second. Good. So, I just I just I just set up a new LLM set up a new LLM connection. And in connection. And in connection. And in this case, I'm this case, I'm this case, I'm using the OpenAI API key again using the OpenAI API key again using the OpenAI API key again . I'll be back. Yes.

. I'll be back. Yes. . I'll be back. Yes. So, I plugged in the So, I plugged in the OpenAI API key that we OpenAI API key that we OpenAI API key that we set up together. set up together. set up together. Of course, you Of course, you Of course, you can also connect can also connect can also connect all other types of models all other types of models . And you can . And you can . And you can use use use your own proxy keys your own proxy keys your own proxy keys if you have if you have if you have your your your own gateway configured. And now own gateway configured. And now own gateway configured. And now we will return to the we will return to the we will return to the evaluator page. evaluator page. evaluator page. Click "create Click "create Click "create evaluator" and evaluator" and evaluator" and select our template again. select our template again. And let's choose a model. And And let's choose a model. And now we see that the now we see that the now we see that the first thing we can first thing we can first thing we can do together is do together is do together is filter out the filter out the filter out the actual actual actual observations that observations that observations that we want to target. we want to target. we want to target. And in this case, we And in this case, we And in this case, we want to choose not want to choose not want to choose not a generation, but an agent, a generation, but an agent, a generation, but an agent, which is the root of all the which is the root of all the which is the root of all the tracing that we tracing that we tracing that we have here. You have here. You have here. You may also see other may also see other may also see other options, such as options, such as options, such as selecting selecting selecting tool calls or tool calls or tool calls or generations, but in generations, but in generations, but in our case we our case we our case we want to focus want to focus want to focus only on the input only on the input , i.e., viewing the , i.e., viewing the user's messages. So, going back to going back to our instructions, we our instructions, we our instructions, we have a certain way of have a certain way of have a certain way of choosing these different ones.

choosing these different ones. different parts of different parts of the observations. We the observations. We the observations. We take an agent, we have take an agent, we have take an agent, we have our evaluator prompt, our evaluator prompt, our evaluator prompt, and here below we see that and here below we see that and here below we see that through the curly braces through the curly braces through the curly braces we pull up we pull up we pull up the context. To the context. To the context. To target the target the target the part of the trace part of the trace part of the trace we need, we we need, we we need, we need to customize need to customize need to customize this JSON path and this JSON path and this JSON path and make sure make sure make sure we select the correct we select the correct we select the correct element. For element. For element. For conversation history, we want conversation history, we want conversation history, we want to cover the entire array of to cover the entire array of to cover the entire array of messages. So we messages. So we messages. So we copy it here. And copy it here. And copy it here. And we insert it here. we insert it here. we insert it here. Now below we see Now below we see that it displays the that it displays the that it displays the message history of message history of message history of your trace. your trace. your trace. It collects all the It collects all the It collects all the interaction: what interaction: what interaction: what Dad asked, what the Dad asked, what the Dad asked, what the tools did, and what tools did, and what tools did, and what response was response was response was generated. As a generated. As a generated. As a second step to second step to second step to detect detect detect user disagreement, we user disagreement, we user disagreement, we also need to take also need to take also need to take the last message the last message the last message from them and thus from them and thus from them and thus obtain obtain obtain information. Below we information. Below we information. Below we see that see that see that the message "how do the message "how do the message "how do I turn on Bluetooth I turn on Bluetooth I turn on Bluetooth on my phone" on my phone" on my phone" is passed to this is passed to this is passed to this estimator. If we estimator. If we estimator. If we do this now, we do this now, we do this now, we will have the first will have the first will have the first working working working real-time estimator.

real-time estimator. real-time estimator. Imagine a Imagine a Imagine a request comes in where Dad says, " request comes in where Dad says, " Okay, how do I Okay, how do I Okay, how do I turn on Bluetooth?" And turn on Bluetooth?" And turn on Bluetooth?" And then he clarifies: “Oh, then he clarifies: “Oh, then he clarifies: “Oh, but that button is but that button is but that button is n’t actually here. I don't n’t actually here. I don't n’t actually here. I don't know where to find her, and know where to find her, and know where to find her, and it's not true what you're it's not true what you're it's not true what you're saying." We will saying." We will saying." We will record this because it is an record this because it is an record this because it is an interesting event—a signal interesting event—a signal interesting event—a signal that allows us that allows us that allows us to understand where exactly the to understand where exactly the to understand where exactly the problems arise. And problems arise. And , probably, out of the first 500 , probably, out of the first 500 , probably, out of the first 500 questions, not all 500 questions, not all 500 questions, not all 500 were unsuccessful, were unsuccessful, were unsuccessful, but there were five, six, but there were five, six, but there were five, six, or seven cases where or seven cases where or seven cases where we didn't explain clearly enough we didn't explain clearly enough we didn't explain clearly enough and and and needed more needed more needed more detail. So, detail. So, detail. So, user disagreement user disagreement is our first is our first is our first example. Let's see example. Let's see example. Let's see what else might be what else might be what else might be interesting. In our interesting. In our interesting. In our case, we have an case, we have an case, we have an example where we example where we example where we use a use a use a software software software evaluator evaluator evaluator to check whether the to check whether the to check whether the text is text is text is only in uppercase. only in uppercase. only in uppercase. So, if I think So, if I think So, if I think about how I sometimes about how I sometimes about how I sometimes interact with Claude or interact with Claude or interact with Claude or Codex, when I'm Codex, when I'm Codex, when I'm annoyed, I might annoyed, I might annoyed, I might write in all write in all write in all caps to get caps to get caps to get my point across. And I think my point across. And I think my point across. And I think many of you have many of you have many of you have done this in done this in done this in chats as well. We see, chats as well. We see, chats as well. We see, especially in chats, that especially in chats, that especially in chats, that this is actually a good this is actually a good this is actually a good way to detect way to detect user annoyance. In this user annoyance. In this case, it's capital case, it's capital case, it's capital letters, but letters, but letters, but alternatively it alternatively it alternatively it could be swear could be swear could be swear words or, say, words or, say, words or, say, certain terms that you certain terms that you certain terms that you shouldn't normally shouldn't normally shouldn't normally use in use in use in chat. But when it's an chat. But when it's an chat. But when it's an agent, you may be agent, you may be agent, you may be more inclined to do more inclined to do more inclined to do so. So we want so. So we want so. So we want to set up a to set up a to set up a code-based evaluator so we code-based evaluator so we code-based evaluator so we can make sure we can make sure we can make sure we notice when notice when notice when it's irritated.

it's irritated. it's irritated. So, as you can see here So, as you can see here , we have a small , we have a small , we have a small piece of Python code. piece of Python code. And if we And if we scroll down a little bit, we'll scroll down a little bit, we'll scroll down a little bit, we'll see that the see that the estimator function itself is defined here. It again targets the It again targets the JSON path part. JSON path part. JSON path part. That is, it extracts a That is, it extracts a That is, it extracts a specific message from specific message from specific message from the user and then the user and then the user and then checks if there are any checks if there are any checks if there are any capital letters in it. And if capital letters in it. And if capital letters in it. And if there are uppercase letters, there are uppercase letters, there are uppercase letters, we return the we return the we return the logical value " logical value " true". So, let's true". So, let's true". So, let's copy this so we copy this so we copy this so we can set up can set up can set up the connection. I the connection. I the connection. I only copy the bottom part, only copy the bottom part, only copy the bottom part, as the top as the top as the top part is actually part is actually part is actually static. And I'm static. And I'm static. And I'm setting up an setting up an setting up an additional evaluator, additional evaluator, additional evaluator, in this case in this case in this case code-based. We also code-based. We also code-based. We also support support support TypeScript code. So if you TypeScript code. So if you TypeScript code. So if you prefer prefer prefer TypeScript, it's possible. But TypeScript, it's possible. But TypeScript, it's possible. But now we want to now we want to now we want to select Python code. And we'll select Python code. And we'll select Python code. And we'll call it "all call it "all call it "all caps" caps" annoyed. And annoyed. And since we're since we're since we're using a using a using a boolean, boolean, boolean, let's add "is".

let's add "is". let's add "is". So we will always So we will always So we will always know: if it's " know: if it's " true," then he is true," then he is true," then he is really really really annoyed and writes in annoyed and writes in annoyed and writes in capital letters. And capital letters. And capital letters. And now I'm going to now I'm going to now I'm going to replace this part at the replace this part at the replace this part at the bottom with our Of course. I'm bottom with our Of course. I'm bottom with our Of course. I'm copying this again. And copying this again. And copying this again. And of course, you can of course, you can of course, you can write any write any write any custom code you custom code you custom code you want here. So, whatever want here. So, whatever want here. So, whatever you want to evaluate you want to evaluate — whether it's specific words — whether it's specific words — whether it's specific words you're looking for or, you're looking for or, you're looking for or, for example, a for example, a for example, a brand name that brand name that brand name that should always be spelled should always be spelled should always be spelled correctly. For example, correctly. For example, correctly. For example, in the name Langfuse, in the name Langfuse, in the name Langfuse, the letter F is sometimes the letter F is sometimes the letter F is sometimes capitalized. We usually capitalized. We usually capitalized. We usually write it in lowercase. write it in lowercase. write it in lowercase. So you can also So you can also So you can also track track track things like that to see where things like that to see where something might be going something might be going something might be going wrong. And, yeah, let's wrong. And, yeah, let's wrong. And, yeah, let's set it up. So, we set it up. So, we set it up. So, we created our second created our second created our second estimator. But of course, estimator. But of course, estimator. But of course, we also need to we also need to we also need to think about, uh, what kind of think about, uh, what kind of think about, uh, what kind of observations do observations do observations do we want we want we want to target? So, to target? So, to target? So, let's take let's take let's take another look here. Um, we another look here. Um, we another look here. Um, we say, "Okay, say, "Okay, say, "Okay, sure, we want sure, we want sure, we want to run this on to run this on to run this on current current current observations." So observations." So , we want to know what's , we want to know what's , we want to know what's happening in the happening in the happening in the production production production environment. And we're environment. And we're environment. And we're again focusing again focusing again focusing on, uh, the type of on, uh, the type of on, uh, the type of observation—the agent observation—the agent . And here I can choose . And here I can choose . And here I can choose this. We choose an this. We choose an this. We choose an agent again. And here on the right agent again. And here on the right agent again. And here on the right we also see that we also see that we also see that we can do a we can do a we can do a little test

little test little test run. And here we run. And here we run. And here we see that in this see that in this see that in this case it, uh, case it, uh, case it, uh, returns "false" because returns "false" because returns "false" because we're only asking here we're only asking here we're only asking here to turn Bluetooth on and to turn Bluetooth on and to turn Bluetooth on and off. And once off. And once off. And once we're happy with everything we're happy with everything we're happy with everything , uh, we , uh, we , uh, we can, uh, click can, uh, click can, uh, click "execute." Uh, "execute." Uh, "execute." Uh, a little side note: a little side note: a little side note: we're working with observations right now, we're working with observations right now, we're working with observations right now, but but but of course this of course this of course this can also be run on can also be run on can also be run on any any any experiments you might experiments you might want to want to want to run in an run in an run in an autonomous autonomous autonomous improvement process, improvement process, improvement process, instead of instead of instead of running them on running them on running them on real real real production data. So, yeah, production data. So, yeah, production data. So, yeah, for now we're happy with, for now we're happy with, for now we're happy with, uh, just running on uh, just running on uh, just running on current current current observations. And observations. And observations. And let's also let's also let's also click "execute" click "execute" click "execute" here. Perfectly. Uh, so here. Perfectly. Uh, so here. Perfectly. Uh, so let's check out let's check out let's check out how it how it how it actually turned out. Uh, because actually turned out. Uh, because actually turned out. Uh, because of course we want to know of course we want to know of course we want to know if it works too. And if it works too. And if it works too. And for that, I for that, I for that, I turn to our turn to our turn to our IT support agent. IT support agent. Let's start with Let's start with the question: "How do I the question: "How do I the question: "How do I turn Bluetooth back on turn Bluetooth back on ?". Let's wait a second. ?". Let's wait a second. ?". Let's wait a second. And then we will get And then we will get And then we will get our answers. I our answers. I 'm zooming in. Yes , I can try. , I can try. , I can try. For me, it's also a bit For me, it's also a bit For me, it's also a bit petty. I used to petty. I used to petty. I used to think I should have think I should have think I should have gotten more gotten more gotten more glasses. And yes, now glasses. And yes, now glasses. And yes, now we see it perfectly we see it perfectly . Um, so let's . Um, so let's . Um, so let's get annoyed. Here it is at get annoyed. Here it is at get annoyed. Here it is at first. And, of course, first. And, of course, first. And, of course, our agent somehow our agent somehow our agent somehow copes with this, because copes with this, because copes with this, because we have certain we have certain we have certain specifications in the specifications in the specifications in the system request. And system request. And system request. And I might also, I might also, I might also, uh, need uh, need uh, need an answer first.

an answer first. an answer first. So, let's see if So, let's see if So, let's see if we get more we get more we get more details. And then I details. And then I details. And then I might ask, "No, there's might ask, "No, there's might ask, "No, there's no such no such no such Bluetooth icon there." Just to Bluetooth icon there." Just to argue a little with the argue a little with the agent. And he says I'm agent. And he says I'm agent. And he says I'm right. So, right. So, right. So, let's see let's see let's see how this will show up in how this will show up in how this will show up in our traces. our traces. our traces. Okay, we see that we have a Okay, we see that we have a few more, uh, entries few more, uh, entries here. And let's here. And let's here. And let's maybe check maybe check maybe check this one out. Okay, we this one out. Okay, we this one out. Okay, we also have also have also have some evaluators already. some evaluators already. some evaluators already. Let's dig Let's dig Let's dig a little deeper into what's a little deeper into what's a little deeper into what's really really really going on here. So, uh going on here. So, uh , we see that this is , we see that this is , we see that this is not the same anymore. Yes, we have a not the same anymore. Yes, we have a user message: "How do user message: "How do I turn on Bluetooth?" We I turn on Bluetooth?" We I turn on Bluetooth?" We get an answer. get an answer. get an answer. And then we see that And then we see that And then we see that this is the result this is the result this is the result we get we get we get back. Okay, Dad, don't back. Okay, Dad, don't back. Okay, Dad, don't worry. Let's worry. Let's worry. Let's try something else, uh, try something else, uh, try something else, uh, first. So, here first. So, here first. So, here we had some, uh, we had some, uh, we had some, uh, disagreements, and disagreements, and disagreements, and here we can here we can here we can see such a see such a see such a small score. So in small score. So in small score. So in the upper left the upper left the upper left corner, it's probably corner, it's probably corner, it's probably still small. We are increasing.

still small. We are increasing. still small. We are increasing. So now we see So now we see So now we see that our root that our root that our root observation, as a observation, as a observation, as a general agent, is general agent, is general agent, is actually labeled actually labeled actually labeled as " as " as " user disagreement—true" user disagreement—true" , uh, because here , uh, because here , uh, because here the user actually the user actually the user actually said, "Okay, said, "Okay, said, "Okay, actually, uh, in actually, uh, in actually, uh, in this case also this case also this case also all all all caps, uh, caps, uh, caps, uh, actually not true, and actually not true, and actually not true, and that didn't work." And, that didn't work." And, that didn't work." And, uh, we can uh, we can uh, we can see it right here see it right here see it right here and have this signal. and have this signal. and have this signal. So we could So we could So we could theoretically theoretically theoretically filter out filter out filter out observations that observations that observations that have, uh, maybe " have, uh, maybe " user disagreement is user disagreement is true" or also, uh true" or also, uh , "frustration in , "frustration in , "frustration in capital letters is capital letters is true," and true," and true," and thus find the ones that are thus find the ones that are thus find the ones that are really interesting for us to really interesting for us to really interesting for us to work with. Good. So these are the work with. Good. So these are the work with. Good. So these are the ones where we get ones where we get ones where we get something from the more or less, uh something from the more or less, uh , productive , productive , productive behavior of, uh, people. behavior of, uh, people. behavior of, uh, people. There is also one There is also one There is also one additional thing that is additional thing that is additional thing that is often interesting often interesting often interesting to track in a to track in a to track in a work application. work application. work application. Mainly, uh, Mainly, uh, Mainly, uh, is someone trying to is someone trying to is someone trying to do something that do something that do something that really shouldn't be really shouldn't be really shouldn't be done, uh, with the done, uh, with the done, uh, with the appropriate, uh, appropriate, uh, appropriate, uh, application. So we application. So we application. So we call this, uh, an call this, uh, an call this, uh, an out-of- out-of- out-of- scope request. So in scope request. So in scope request. So in our system our system our system request, uh, we request, uh, we request, uh, we specify here that specify here that specify here that the agent should assist the agent should assist the agent should assist with all types of with all types of with all types of IT IT IT phone support questions. Er, but phone support questions. Er, but phone support questions. Er, but of course, er, this also of course, er, this also of course, er, this also applies to different applies to different applies to different types of devices. So types of devices. So types of devices. So maybe a printer or maybe a printer or maybe a printer or a computer that a computer that a computer that he wants to, uh, he wants to, uh, he wants to, uh, get answers get answers get answers to questions from. But right now to questions from. But right now to questions from. But right now we're limited to, uh, this, uh we're limited to, uh, this, uh we're limited to, uh, this, uh , specific , specific , specific scope. So for scope. So for scope. So for this we can also this we can also , uh, set up , uh, set up , uh, set up these types of evaluators.

these types of evaluators. these types of evaluators. And we call them in And we call them in And we call them in this case now " this case now " out of out of out of scope request". And here scope request". And here scope request". And here we have a different, uh, we have a different, uh, we have a different, uh, dynamic. We're dynamic. We're dynamic. We're not just looking at the not just looking at the user's last message or user's last message or message history, message history, message history, uh, but here we're uh, but here we're uh, but here we're actually also actually also actually also looking at a looking at a looking at a system query. And system query. And system query. And because we because we because we define all of this define all of this define all of this through a JSON path, we have through a JSON path, we have through a JSON path, we have the ability to the ability to the ability to target it target it target it specifically as well. So, specifically as well. So, specifically as well. So, uh, one last time, let's uh, one last time, let's uh, one last time, let's set up this set up this set up this estimator. And for that, estimator. And for that, estimator. And for that, we go to our we go to our we go to our evaluators page. evaluators page. We can click We can click on “evaluators” and on “evaluators” and on “evaluators” and select our select our select our out-of-scope request template once again out-of-scope request template once again out-of-scope request template once again . . Now we want to Now we want to target the target the target the generation because we generation because we generation because we also want to also want to also want to reference the reference the reference the system query. system query. system query. Now let's Now let's Now let's compare our compare our compare our variables again. So, we variables again. So, we variables again. So, we take the system take the system take the system prompt, which in this prompt, which in this prompt, which in this case is the very first case is the very first case is the very first message message message sent in this sent in this sent in this interaction. And then we interaction. And then we interaction. And then we also take the last also take the last also take the last message from message from message from the user that the user that the user that comes in. And in this comes in. And in this comes in. And in this prompt, we now prompt, we now prompt, we now define: okay, we define: okay, we define: okay, we also want to also want to also want to constantly compare constantly compare what the system what the system what the system should actually do should actually do should actually do and what the agent should do and what the agent should do and what the agent should do with what the user is actually with what the user is actually with what the user is actually trying to do trying to do trying to do . So, . So,

now it's connected. Variables are not displayed at this time Variables are not displayed at this time . And . And . And let's do that let's do that let's do that too. Okay, so too. Okay, so too. Okay, so every time every time every time they try to they try to they try to do something with do something with do something with the app that they the app that they the app that they really shouldn't really shouldn't really shouldn't , we'll be able to , we'll be able to , we'll be able to learn about it and learn about it and make an make an informed decision based on that information informed decision based on that information . Do we want to . Do we want to . Do we want to fix this to fix this to fix this to expand the scope of expand the scope of expand the scope of our our our app, or maybe app, or maybe app, or maybe we are having trouble we are having trouble we are having trouble communicating to communicating to communicating to users users users what exactly they should be what exactly they should be what exactly they should be doing with our doing with our doing with our app? And in this app? And in this app? And in this case, of course, it's case, of course, it's case, of course, it's just IT support, but just IT support, but just IT support, but maybe in maybe in maybe in larger larger larger applications, people sometimes applications, people sometimes applications, people sometimes try to do try to do try to do something and really something and really something and really misunderstand misunderstand misunderstand how the system is supposed to be how the system is supposed to be used. And used. And so we can so we can , yes, detect , yes, detect , yes, detect similar things similar things similar things happening and happening and happening and delve into the delve into the delve into the details. Perfectly. So, this is how details. Perfectly. So, this is how details. Perfectly. So, this is how we can we can we can set up set up set up evaluators and evaluators and evaluators and monitors for our monitors for our monitors for our application. And I think you can application. And I think you can application. And I think you can clearly see that it's clearly see that it's clearly see that it's not about not about not about just copying what's just copying what's just copying what's already there and what we already there and what we already there and what we can use.

can use. can use. Of course, a lot of Of course, a lot of Of course, a lot of this could be this could be this could be reused reused reused for a chat app, but it's for a chat app, but it's for a chat app, but it's really important to really important to really important to think about what think about what think about what typical failure modes typical failure modes typical failure modes we experience in our we experience in our we experience in our type of app, rather than type of app, rather than type of app, rather than trying to copy trying to copy trying to copy what's already out there, but really what's already out there, but really what's already out there, but really looking at what's looking at what's looking at what's going on. So going on. So going on. So when we want to when we want to when we want to work with this now work with this now , we could, , we could, , we could, for example, now go for example, now go for example, now go to using these to using these to using these estimates now to estimates now to estimates now to also filter out also filter out also filter out certain specific certain specific certain specific traces and traces and traces and results. Here we results. Here we results. Here we have two, and that way have two, and that way have two, and that way we get we get we get an idea of ​​how an idea of ​​how an idea of ​​how often these things often these things often these things actually go actually go actually go wrong. Perfectly. So, wrong. Perfectly. So, wrong. Perfectly. So, this way we can this way we can this way we can connect monitors connect monitors connect monitors and see what's and see what's and see what's going on. And going on. And going on. And let's now let's now let's now look at some look at some look at some production data to production data to production data to see what's see what's see what's actually actually actually happening and how it happening and how it happening and how it works in our works in our works in our application. And for this, application. And for this, application. And for this, we move to we move to we move to the end of our the end of our the end of our monitoring module. And we monitoring module. And we monitoring module. And we have a script here to have a script here to have a script here to add some add some add some production production production information. And information. And information. And thus, we have a little thus, we have a little thus, we have a little more data to more data to more data to analyze. So, I'm analyze. So, I'm analyze. So, I'm going back to my going back to my going back to my repository. And I'm copying repository. And I'm copying repository. And I'm copying our team. You our team. You our team. You can imagine that in can imagine that in can imagine that in the background we the background we the background we have a set of traces as a have a set of traces as a have a set of traces as a dataset. And this dataset. And this dataset. And this information also information also information also contains certain estimates.

contains certain estimates. And we load this And we load this directly into our directly into our directly into our application, into our application, into our application, into our LangFuse instance. So LangFuse instance. So LangFuse instance. So let's check let's check let's check if they are already if they are already if they are already loaded. loaded. loaded. We check our We check our We check our filter again based on the ratings. And filter again based on the ratings. And filter again based on the ratings. And now we see that now we see that now we see that yes, new data has emerged yes, new data has emerged yes, new data has emerged . We actually . We actually . We actually now have a lot, um, now have a lot, um, a lot of data, several a lot of data, several pages of traces. And pages of traces. And pages of traces. And now the question, of course, is now the question, of course, is how to work with this how to work with this how to work with this and how and how and how to find out what's going to find out what's going to find out what's going wrong and what's working well wrong and what's working well wrong and what's working well in our application? in our application? And all this data is And all this data is usually usually usually used to used to used to understand understand understand what exactly needs to be what exactly needs to be what exactly needs to be improved improved improved offline. offline.

So these working So these working traces, especially traces, especially traces, especially those that seem those that seem those that seem interesting, can interesting, can interesting, can be used to be used to be used to create a dataset create a dataset create a dataset that you will apply in the that you will apply in the that you will apply in the autonomous autonomous autonomous improvement process to improvement process to improvement process to make the application make the application make the application better. But today better. But today better. But today we want to we want to we want to focus on the focus on the focus on the monitoring monitoring monitoring and tracing part. So and tracing part. So and tracing part. So let's see let's see let's see how we can how we can how we can use our use our use our programming agent programming agent to find to find to find larger problems larger problems larger problems in this application. in this application. To do this, let's To do this, let's install the install the install the LangFuse skill together so we LangFuse skill together so we LangFuse skill together so we can can can use it. use it. use it. We have a We have a We have a small team here small team here small team here in the in the in the setup phase for this. setup phase for this. I usually just I usually just ask my programming agent ask my programming agent to install it for me , but you can also , but you can also , but you can also install it yourself install it yourself install it yourself . So we have the . So we have the . So we have the CLI, as well as our CLI, as well as our CLI, as well as our skill, and it can be skill, and it can be skill, and it can be installed, in my installed, in my installed, in my case right now, case right now, case right now, using Claude. I using Claude. I using Claude. I trust my trust my trust my application. And what application. And what application. And what happens now is that happens now is that happens now is that it loads our CLI, it loads our CLI, it loads our CLI, and our CLI exposes and our CLI exposes and our CLI exposes all the APIs that all the APIs that all the APIs that we have, so you we have, so you we have, so you can always can always can always access access access your traces, your your traces, your your traces, your scores, your datasets, your scores, your datasets, your experiment results, and experiment results, and run them run them run them through the command through the command through the command line directly line directly line directly or with your or with your programming agent. And thus work more work more efficiently with LangFuse. And efficiently with LangFuse. And efficiently with LangFuse. And this is also true this is also true this is also true when you are trying to

when you are trying to when you are trying to instrument your instrument your instrument your application from the application from the application from the beginning and get beginning and get beginning and get your first your first your first traces. This is something traces. This is something traces. This is something you can you can you can use to use to use to automatically add automatically add automatically add tracing to your tracing to your tracing to your application and application and application and get your first get your first get your first insights. So, I'm insights. So, I'm insights. So, I'm installing the installing the installing the LangFuse skill right now, just to LangFuse skill right now, just to LangFuse skill right now, just to see how it see how it see how it works. I'm moving on works. I'm moving on works. I'm moving on to the Codex now. I already to the Codex now. I already to the Codex now. I already installed it in the installed it in the installed it in the background. And background. And background. And now I want now I want now I want to figure out what exactly is going wrong to figure out what exactly is going wrong to figure out what exactly is going wrong in my in my in my traces. So, traces. So, traces. So, what steps are going what steps are going what steps are going wrong, what indicators do wrong, what indicators do wrong, what indicators do we have there? And I we have there? And I we have there? And I ask my ask my ask my programming agent, " programming agent, " programming agent, " Please use the Please use the Please use the LangFuse skill to LangFuse skill to LangFuse skill to find problems in find problems in find problems in our traces." our traces." Of course, Codex Of course, Codex will need some will need some will need some time now, but to time now, but to time now, but to understand what is understand what is understand what is happening: Codex is happening: Codex is happening: Codex is now accessing the now accessing the now accessing the actual actual actual traces that we traces that we traces that we have in our have in our have in our application. It application. It application. It checks what checks what checks what questions I asked, what questions I asked, what questions I asked, what ratings we see there, ratings we see there, ratings we see there, whether there are many whether there are many whether there are many caps messages caps messages caps messages indicating indicating indicating irritation, as well as irritation, as well as irritation, as well as where the user where the user where the user disagrees, and disagrees, and disagrees, and even more than what even more than what even more than what we have officially we have officially we have officially set up. It set up. It set up. It also checks if there are any also checks if there are any also checks if there are any issues with our issues with our issues with our latency, if there are any latency, if there are any latency, if there are any issues with issues with issues with tool calls, or if tool calls, or if tool calls, or if anything in general went anything in general went anything in general went wrong that we didn't want to wrong that we didn't want to . To find where . To find where . To find where the system is silently the system is silently the system is silently crashing in the work crashing in the work crashing in the work environment, although we environment, although we environment, although we certainly don't certainly don't certainly don't want that. Since this is going to take a while to want that. Since this is going to take a while to want that. Since this is going to take a while to work on, work on, work on, I think this is a good

I think this is a good I think this is a good time to wait for the time to wait for the time to wait for the Codex and answer a Codex and answer a Codex and answer a few questions if you have any few questions if you have any few questions if you have any . Okay, . Okay, . Okay, let's see if we've let's see if we've let's see if we've got the result yet. got the result yet. got the result yet. Yes, we received the report. Yes, we received the report. I could save this I could save this as markdown, but for now I'll as markdown, but for now I'll as markdown, but for now I'll briefly show you the one I briefly show you the one I briefly show you the one I prepared prepared prepared in advance. And in advance. And in advance. And I swear, I'm not in the business of I swear, I'm not in the business of I swear, I'm not in the business of adjusting the adjusting the adjusting the results. We results. We results. We can look can look can look at the actual report as well, I at the actual report as well, I at the actual report as well, I just wanted to just wanted to just wanted to show what it show what it show what it looks like. So, our looks like. So, our programming agent programming agent analyzed the working analyzed the working analyzed the working traces that we traces that we traces that we uploaded there. It uploaded there. It uploaded there. It was about 120 was about 120 was about 120 traces or 70 traces or 70 traces or 70 traces. He traces. He traces. He actually looked at actually looked at actually looked at all the different dimensions where all the different dimensions where all the different dimensions where something could have gone something could have gone something could have gone wrong. And now we have a wrong. And now we have a wrong. And now we have a pretty detailed pretty detailed pretty detailed report of how many report of how many report of how many traces there were, traces there were, traces there were, how many observations how many observations , and how many of , and how many of , and how many of them were them were them were tool calls. And what do we tool calls. And what do we tool calls. And what do we have there? And then he have there? And then he have there? And then he also generated also generated also generated some conclusions for us. And the some conclusions for us. And the some conclusions for us. And the first conclusion here is that first conclusion here is that we have a hidden we have a hidden problem with the quality of problem with the quality of problem with the quality of search in the search in the search in the help library. So this is help library. So this is help library. So this is actually something that we didn't actually something that we didn't actually something that we didn't evaluate, but we evaluate, but we evaluate, but we also didn't also didn't also didn't realize before that our realize before that our information retrieval tool, where we information retrieval tool, where we look at what the look at what the look at what the right right right answer might be, is actually answer might be, is actually answer might be, is actually quietly failing and not quietly failing and not quietly failing and not doing what it's supposed to doing what it's supposed to doing what it's supposed to do. So, this is the do. So, this is the do. So, this is the first thing we see.

first thing we see. first thing we see. But we also see But we also see But we also see that that that data acquisition errors data acquisition errors data acquisition errors increase latency increase latency increase latency and cost. So, and cost. So, and cost. So, we have a certain problem we have a certain problem we have a certain problem with repeated with repeated tool calls, where we tool calls, where we find certain find certain find certain problems. And we also problems. And we also problems. And we also see that there is a see that there is a see that there is a gap in the gap in the gap in the estimator's coverage estimator's coverage estimator's coverage that relates to that relates to that relates to recent traffic recent traffic . And we don't actually . And we don't actually . And we don't actually evaluate everything, but we evaluate everything, but we only get a signal from a subset of the only get a signal from a subset of the tracks that are tracks that are tracks that are happening there. And happening there. And happening there. And just briefly just briefly just briefly check if Codex has check if Codex has check if Codex has now also published the now also published the now also published the same. Okay, he same. Okay, he same. Okay, he also found that also found that also found that the version of the prompt was the version of the prompt was the version of the prompt was wrong. And then wrong. And then wrong. And then he also talks he also talks he also talks about delay tags for about delay tags for about delay tags for retries. And retries. And retries. And we have a we have a we have a monitoring separation. Yes, and monitoring separation. Yes, and monitoring separation. Yes, and that's how we can that's how we can that's how we can find these problems in find these problems in find these problems in our application, but our application, but our application, but we can also we can also we can also make sure that make sure that make sure that we find a way to we find a way to we find a way to fix them. And so, fix them. And so, fix them. And so, yes, improving yes, improving yes, improving the app over time and the app over time and the app over time and making sure that we making sure that we , yes, that we , yes, that we , yes, that we fix the problems fix the problems that actually that actually that actually arise. Okay, so arise. Okay, so we get a we get a suggestion here to suggestion here to suggestion here to keep it. Um, but keep it. Um, but keep it. Um, but let's see let's see let's see if we can if we can if we can get a fix get a fix get a fix here too. So, what's the here too. So, what's the here too. So, what's the most interesting part? I most interesting part? I most interesting part? I actually want to actually want to actually want to get rid of the get rid of the get rid of the search help library.

search help library. search help library. Because if it Because if it Because if it doesn't really doesn't really doesn't really help us help us help us use these use these use these tools, then we tools, then we tools, then we can remove it. can remove it. can remove it. Um, and we Um, and we Um, and we can can can use our use our use our skill for that too. Our skill for that too. Our library library search assistance tool. And search assistance tool. And search assistance tool. And since we now since we now since we now have the have the have the repository context, it repository context, it repository context, it can actually also can actually also can actually also help us help us help us fix problems in fix problems in fix problems in the code, not just in the the code, not just in the language tool settings. And yes, now he's And yes, now he's back to back to back to work and fixing work and fixing work and fixing it for us in our it for us in our it for us in our app. But let's app. But let's app. But let's now take a now take a now take a moment to take a moment to take a moment to take a step back and step back and step back and understand what understand what understand what really needs to happen with this really needs to happen with this really needs to happen with this . And for . And for . And for that, we that, we that, we go back to go back to go back to our slides here again our slides here again our slides here again . So what . So what . So what we focused on today is the we focused on today is the part of getting part of getting signals from our signals from our signals from our production runs. So production runs. So production runs. So we first we first we first set up our set up our set up our tracing and tracing and tracing and made sure we made sure we made sure we had a step-by-step had a step-by-step had a step-by-step overview of what was overview of what was overview of what was actually actually actually happening, what happening, what happening, what tools tools tools our agent was using our agent was using our agent was using , what the input and , what the input and , what the input and output were, and output were, and output were, and we got a signal from that we got a signal from that we got a signal from that . But what should . But what should . But what should happen to such happen to such happen to such signals so that the agent signals so that the agent signals so that the agent continuously continuously continuously improves over improves over improves over time? And to do this, on the time? And to do this, on the time? And to do this, on the one hand, we one hand, we one hand, we need to need to need to maintain our maintain our maintain our evaluation criteria.

evaluation criteria. evaluation criteria. So, we had So, we had So, we had a question, for example, a question, for example, a question, for example, how to ensure that we how to ensure that we how to ensure that we don't overestimate don't overestimate don't overestimate one thing or that it doesn't give one thing or that it doesn't give one thing or that it doesn't give us any useful us any useful us any useful information. And so, information. And so, information. And so, given the signals, we given the signals, we given the signals, we need to make sure need to make sure need to make sure that over time we have a that over time we have a that over time we have a quality system of quality system of quality system of evaluators to evaluators to evaluators to get the get the get the information we need. But information we need. But information we need. But most importantly, we most importantly, we most importantly, we also need to also need to also need to maintain our maintain our maintain our dataset. When you dataset. When you dataset. When you build an application build an application , you may think you , you may think you , you may think you know exactly what know exactly what users want to do, but over users want to do, but over time you will realize time you will realize time you will realize that there are many edge that there are many edge that there are many edge cases that you did not cases that you did not cases that you did not consider consider consider in advance. That's why in advance. That's why in advance. That's why you need to you need to you need to maintain your maintain your maintain your data set over time, data set over time, data set over time, create successful create successful create successful examples, add them examples, add them examples, add them to autonomous assessments, and to autonomous assessments, and to autonomous assessments, and then confidently then confidently then confidently implement changes. implement changes. You can make this You can make this part of your CI/CD and part of your CI/CD and part of your CI/CD and ensure that no ensure that no ensure that no regressions occur regressions occur regressions occur every time you make a every time you make a every time you make a change. Today we change. Today we change. Today we have only covered a have only covered a have only covered a part of the part of the part of the AI ​​development cycle where we AI ​​development cycle where we AI ​​development cycle where we focus on focus on focus on online assessments, but online assessments, but online assessments, but of course, that is far from of course, that is far from of course, that is far from everything. The main thing is everything. The main thing is to understand what to understand what to understand what is happening: is happening: is happening: to capture interesting to capture interesting to capture interesting moments, whether it is a moments, whether it is a moments, whether it is a user's disagreement, the user's disagreement, the user's disagreement, the use of use of use of caps, obscene caps, obscene caps, obscene language, or any language, or any language, or any other quality assessment.

other quality assessment. Based on this, you Based on this, you build and, importantly, build and, importantly, build and, importantly, continuously maintain continuously maintain continuously maintain datasets. All this is datasets. All this is datasets. All this is not done at not done at not done at once. You need to once. You need to once. You need to constantly work constantly work constantly work on this and close the on this and close the on this and close the loop so that the dataset loop so that the dataset loop so that the dataset reflects the real reflects the real reflects the real actions of your actions of your actions of your users. And on users. And on users. And on these datasets you these datasets you these datasets you can already conduct can already conduct can already conduct experiments and experiments and experiments and evaluate their evaluate their evaluate their results. So you results. So you results. So you will always have will always have will always have an understanding of an understanding of an understanding of how high-quality how high-quality how high-quality your application is. And your application is. And your application is. And when you deploy when you deploy when you deploy updates, you can updates, you can updates, you can be confident in the be confident in the be confident in the outcome. This is the outcome. This is the outcome. This is the main part of main part of main part of our workshop. I our workshop. I our workshop. I just wanted to briefly just wanted to briefly just wanted to briefly mention a few mention a few mention a few points. Today we points. Today we points. Today we focused only on focused only on focused only on this part. But on this part. But on this part. But on our website you our website you our website you can see that can see that can see that this LangFuse workshop this LangFuse workshop this LangFuse workshop consists of eight consists of eight consists of eight stages, where you stages, where you stages, where you go through each go through each go through each individual module. individual module. individual module. We were We were We were only able to touch on these only able to touch on these only able to touch on these two today, but LangFuse also has two today, but LangFuse also has prompt management. You can prompt management. You can customize these customize these customize these datasets. And this is a datasets. And this is a datasets. And this is a step-by-step guide that will step-by-step guide that will guide you guide you guide you through all these steps through all these steps through all these steps so you can so you can set everything up correctly. And we of course set everything up correctly. And we of course also recommend also recommend also recommend using this using this using this skill for skill for skill for customization customization customization with your with your with your programming agent. If programming agent. If programming agent. If we were in your we were in your we were in your shoes and had to shoes and had to shoes and had to start tomorrow, we would start tomorrow, we would start tomorrow, we would do exactly that.

do exactly that. do exactly that. So, we would take our So, we would take our So, we would take our repository. We would repository. We would repository. We would use the use the use the LangFuse skill to set up LangFuse skill to set up LangFuse skill to set up tracing and really tracing and really tracing and really understand what is understand what is understand what is happening in happening in happening in your application through your application through your application through this command. Then we would this command. Then we would this command. Then we would manually review manually review manually review the tracks to really the tracks to really the tracks to really understand what users were doing understand what users were doing understand what users were doing . And then . And then . And then eventually we would start eventually we would start eventually we would start building out our building out our building out our routes and routes and routes and event setups event setups event setups for productive for productive for productive routes to also routes to also routes to also start optimizing start optimizing start optimizing offline. OK. offline. OK. offline. OK. Thank you very much everyone. I'll Thank you very much everyone. I'll Thank you very much everyone. I'll answer a answer a answer a few more questions, but few more questions, but few more questions, but thank you very much for participating; thank you very much for participating; thank you very much for participating; Marcus is here, Lotta is here, Marcus is here, Lotta is here, Marcus is here, Lotta is here, I am here. And we would be I am here. And we would be I am here. And we would be very happy to discuss very happy to discuss very happy to discuss anything with you. anything with you. anything with you. We also have an exhibition We also have an exhibition We also have an exhibition stand. So please stand. So please come by over the come by over the come by over the next few next few next few days and chat days and chat days and chat with us. with us.