Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust

Hello everyone. Me Hello everyone. Me My name is Doug Guthrie. I My name is Doug Guthrie. I My name is Doug Guthrie. I solutions engineer at solutions engineer at solutions engineer at Braintrust. I have several. Braintrust. I have several. Braintrust. I have several. colleagues in the room who colleagues in the room who colleagues in the room who will help if you will help if you will help if you get stuck somewhere under get stuck somewhere under get stuck somewhere under time of day time of day time of day workshop. Besides workshop. Besides workshop. Besides me all the time me all the time me all the time interrupted during interrupted during interrupted during calls, so if calls, so if calls, so if Do you want to stop me? Do you want to stop me? Do you want to stop me? at any time and at any time and at any time and to ask a question, to ask a question, to ask a question, I don't mind. I don't mind. I don't mind. I hope you already I hope you already I hope you already managed to scan managed to scan managed to scan QR code. It leads to this QR code. It leads to this QR code. It leads to this repository. This is what repository. This is what repository. This is what we will we will we will use use use today within today within today within workshop to workshop to workshop to understand how understand how understand how to provide to provide to provide observability in observability in observability in Braintrust and how Braintrust and how Braintrust and how apply all of it apply all of it apply all of it functions for functions for functions for creating better creating better creating better agents. Actually, exactly agents. Actually, exactly agents. Actually, exactly that's why we're here. How about us? that's why we're here. How about us? that's why we're here. How about us? actually create actually create actually create quality agents? I quality agents? I quality agents? I I'll leave it on the screen. I'll leave it on the screen. I'll leave it on the screen. for another 10 seconds. I still for another 10 seconds. I still for another 10 seconds. I still I'll show it later in I'll show it later in I'll show it later in presentations when we presentations when we presentations when we directly directly directly let's move on to let's move on to let's move on to workshop. So, if workshop. So, if workshop. So, if you didn't have time, you you didn't have time, you you didn't have time, you you can just you can just you can just google DPGuthrie on google DPGuthrie on google DPGuthrie on GitHub and find this GitHub and find this GitHub and find this the repository is there. Class the repository is there. Class . So, we are here to . So, we are here to . So, we are here to to master to master to master AI observability AI observability , and I'll show you how.

, and I'll show you how. , and I'll show you how. looks in context looks in context looks in context using Braintrust. using Braintrust. Short plan. I have Short plan. I have there are several slides to there are several slides to there are several slides to lay the foundation lay the foundation lay the foundation observability, and observability, and observability, and I will tell you how, on I will tell you how, on I will tell you how, on in our opinion, do this in our opinion, do this in our opinion, do this best. So I best. So I best. So I I will create the basis for I will create the basis for I will create the basis for What does it look like? I will give What does it look like? I will give What does it look like? I will give you have an idea about you have an idea about you have an idea about how to isolate how to isolate how to isolate useful signal from useful signal from useful signal from this huge this huge this huge noise flow. You, noise flow. You, noise flow. You, You've probably already seen it. You've probably already seen it. You've probably already seen it. tracing and tracing and tracing and understand that agents, understand that agents, understand that agents, which you create, which you create, which you create, generate a huge generate a huge generate a huge amount of data. AND amount of data. AND amount of data. AND to understand everything to understand everything to understand everything this to get this to get this to get at least some useful at least some useful at least some useful signal or insight, — signal or insight, — The task is difficult. The task is difficult. The task is difficult. This is precisely what This is precisely what This is precisely what lies the complexity. WITH lies the complexity. WITH lies the complexity. WITH on the one hand, it is necessary on the one hand, it is necessary on the one hand, it is necessary be able to track be able to track be able to track data and have a platform data and have a platform data and have a platform for for for observability. WITH observability. WITH observability. WITH on the other hand, appears on the other hand, appears on the other hand, appears question: how do we question: how do we question: how do we use this for use this for use this for receiving valuables receiving valuables receiving valuables conclusions? So I conclusions? So I conclusions? So I I'll show you how it works in I'll show you how it works in I'll show you how it works in Braintrust, and then we Braintrust, and then we Braintrust, and then we Let's go through everything, Let's go through everything, Let's go through everything, what I described and showed what I described and showed what I described and showed on slides, during on slides, during on slides, during practical part. practical part. practical part. As I said, if you As I said, if you As I said, if you you have any questions, don't you have any questions, don't you have any questions, don't Feel free to interrupt. Feel free to interrupt. . I don't mind. I, . I don't mind. I, . I don't mind. I, At least I'll try.

At least I'll try. At least I'll try. leave some time leave some time leave some time at the end for at the end for at the end for questions and questions and questions and answers. So answers. So answers. So you can leave yours you can leave yours you can leave yours question at the end. question at the end. A brief introduction to A brief introduction to Braintrust, if you haven't already Braintrust, if you haven't already Braintrust, if you haven't already familiar. We- familiar. We- evaluation platform evaluation platform evaluation platform (Evals) and (Evals) and (Evals) and observability ( observability ( observability) AI. Our observability) AI. Our observability) AI. Our the only goal— the only goal— to help our to help our to help our to create customers to create customers to create customers the best agents. We the best agents. We the best agents. We we help our we help our we help our customers to understand, customers to understand, customers to understand, What is quality and how is it What is quality and how is it What is quality and how is it improve in real, improve in real, improve in real, measured way. measured way. One of the most One of the most exciting parts exciting parts exciting parts of my work—this of my work—this of my work—this possibility possibility possibility to communicate with those to communicate with those to communicate with those customers you customers you customers you see above see above see above right. I can right. I can right. I can to understand exactly how to understand exactly how to understand exactly how they build their own they build their own they build their own agents. And many of agents. And many of agents. And many of this knowledge actually this knowledge actually this knowledge actually fall into our fall into our fall into our product. They also product. They also product. They also used in used in used in my other work with my other work with my other work with customers. So I customers. So I customers. So I I am talking to I am talking to I am talking to by Cloudflare engineers, with by Cloudflare engineers, with by Cloudflare engineers, with Dropbox engineers, and Dropbox engineers, and Dropbox engineers, and I find out that they I find out that they I find out that they do: what grades do: what grades do: what grades create, as create, as create, as relate to ours relate to ours relate to ours new features, what is it new features, what is it new features, what is it gives and what gives and what gives and what automation they automation they automation they build on the basis of build on the basis of build on the basis of this. I hope so.

this. I hope so. this. I hope so. Today I will be able to give Today I will be able to give Today I will be able to give do you have any idea do you have any idea do you have any idea about this. I already mentioned about this. I already mentioned about this. I already mentioned about this, right? We- about this, right? We- evaluation platform evaluation platform evaluation platform and observability and observability . If you think about it . If you think about it . If you think about it assessment and assessment and assessment and observability, observability, observability, so it's like two sides so it's like two sides so it's like two sides one medal. Together one medal. Together one medal. Together they are what they are what they are what will allow you will allow you will allow you create agents create agents create agents of the highest quality. If you of the highest quality. If you of the highest quality. If you not yet a customer not yet a customer not yet a customer Braintrust and not Braintrust and not Braintrust and not use our use our use our platform today, platform today, platform today, hard hard hard I recommend I recommend I recommend to return and to return and to return and look at Braintrust look at Braintrust look at Braintrust in a month or two. in a month or two. Today in the industry we Today in the industry we we see mostly we see mostly we see mostly very passive way very passive way very passive way observation of observation of observation of data. This is what I'm talking about. data. This is what I'm talking about. data. This is what I'm talking about. said—possibility said—possibility said—possibility get get get analytical conclusions. analytical conclusions. analytical conclusions. Braintrust is moving, on Braintrust is moving, on Braintrust is moving, on in my opinion, a little in my opinion, a little in my opinion, a little more to what we more to what we more to what we we call active we call active we call active observability. observability. observability. This is giving you This is giving you This is giving you basic tools basic tools , the basics so that you , the basics so that you , the basics so that you received the necessary received the necessary received the necessary conclusions, not conclusions, not conclusions, not extracting data from extracting data from extracting data from platforms and not platforms and not platforms and not sorting out sorting out sorting out alone, that they alone, that they alone, that they mean for your mean for your mean for your agent and what is needed agent and what is needed agent and what is needed do for him do for him do for him improvement. Therefore, improvement. Therefore, improvement. Therefore, active active active observability— observability— this is the way we this is the way we this is the way we we can provide you we can provide you we can provide you these conclusions so that you these conclusions so that you these conclusions so that you created agents created agents created agents of the highest quality. Here of the highest quality. Here of the highest quality. Here so short so short so short Getting to know Braintrust. I Getting to know Braintrust. I Getting to know Braintrust. I I will remind you of this.

I will remind you of this. I will remind you of this. at the end, but we have at the end, but we have at the end, but we have there is a stand at the exhibition. there is a stand at the exhibition. there is a stand at the exhibition. It seems he It seems he It seems he opens opens opens around 5 o'clock. Come around 5 o'clock. Come . I will be there today. . I will be there today. . I will be there today. in the evening. Find in the evening. Find in the evening. Find me, put me, put me, put additional questions. additional questions. additional questions. We will be here too. We will be here too. We will be here too. all week. Therefore, all week. Therefore, all week. Therefore, the basis of all this, the basis of all this, the basis of all this, basis basis basis observability observability observability in general, as I think, in general, as I think, in general, as I think, most of you already most of you already most of you already knows, there is tracing. You knows, there is tracing. You knows, there is tracing. You everyone should understand everyone should understand everyone should understand the steps your the steps your the steps your agent, right? What agent, right? What agent, right? What he does it to he does it to he does it to get the final get the final get the final result. And not result. And not result. And not not only this, but also what not only this, but also what not only this, but also what input data for these input data for these input data for these tools? Which tools? Which tools? Which the results of their the results of their the results of their work? And if you work? And if you work? And if you start thinking about start thinking about start thinking about all the different steps that all the different steps that all the different steps that can make your can make your can make your agent, it might be enough agent, it might be enough agent, it might be enough to grow quickly. AND to grow quickly. AND to grow quickly. AND ability ability ability to figure this out— to figure this out— this is a challenge that we this is a challenge that we this is a challenge that we trying to solve trying to solve . But without this . But without this . But without this tracing, right? tracing, right? tracing, right? We don't really We don't really We don't really we can understand that we can understand that we can understand that happening and where happening and where happening and where quality actually quality actually quality actually getting worse. So getting worse. So getting worse. So opportunity, firstly, opportunity, firstly, opportunity, firstly, track all these track all these track all these data, and then data, and then data, and then to receive certain to receive certain to receive certain conclusions are where conclusions are where conclusions are where Branch will provide some Branch will provide some Branch will provide some value. But in general value. But in general if you are thinking about if you are thinking about if you are thinking about observability, observability, observability, if in your stack if in your stack if in your stack no part no part no part tracing similar to tracing similar to tracing similar to this, then this is black this, then this is black this, then this is black box. You will be box. You will be box. You will be very difficult to detect very difficult to detect very difficult to detect these quality problems these quality problems these quality problems or, you know, indicate or, you know, indicate or, you know, indicate Codex, Cloud Code or whatever Codex, Cloud Code or whatever Codex, Cloud Code or whatever other than that you other than that you other than that you use to use to use to understand that understand that understand that is happening. If is happening. If is happening. If this data is not available, you are not this data is not available, you are not this data is not available, you are not you will be able to find those you will be able to find those you will be able to find those places where the agent places where the agent places where the agent is wrong and where is it is wrong and where is it is wrong and where is it can be improved. Therefore can be improved. Therefore , tracing is , tracing is , tracing is basis. I think if basis. I think if basis. I think if look at look at look at the evaluation side and,

the evaluation side and, the evaluation side and, of course, on of course, on of course, on observability, observability, observability, what are we talking about today what are we talking about today what are we talking about today we say, that's all we say, that's all we say, that's all concentrates concentrates concentrates around our around our around our ability to record ability to record ability to record everything that ours does everything that ours does everything that ours does agent. So, one more thing agent. So, one more thing agent. So, one more thing short note short note short note regarding Brain Trust. We have regarding Brain Trust. We have regarding Brain Trust. We have many different many different many different ways. So I'm a little ways. So I'm a little ways. So I'm a little I'll skip ahead. I'll skip ahead. I'll skip ahead. The seminar we The seminar we The seminar we we conduct, is based we conduct, is based we conduct, is based on, you know, imitations on, you know, imitations on, you know, imitations support agent. Himself support agent. Himself support agent. Himself an agent is not something an agent is not something an agent is not something important, but it important, but it important, but it built on SDK built on SDK built on SDK OpenAI agents. We have OpenAI agents. We have OpenAI agents. We have many different many different many different integrations with all integrations with all integrations with all different frameworks different frameworks different frameworks agents. We have agents. We have agents. We have integration with all integration with all integration with all AI providers. AI providers. AI providers. We have different languages. We have different languages. We have different languages. programming. So, programming. So, programming. So, depending on how depending on how depending on how you create your own you create your own you create your own agent, is it an example agent, is it an example agent, is it an example today on Python, is this today on Python, is this today on Python, is this could be TypeScript, Java, could be TypeScript, Java, could be TypeScript, Java, Ruby or Go. We are quite Ruby or Go. We are quite Ruby or Go. We are quite flexible because flexible because flexible because we allow you we allow you we allow you use any use any -what approach do you -what approach do you -what approach do you you already have in mind you already have in mind you already have in mind programming languages, programming languages, programming languages, or method of construction or method of construction or method of construction agent: through agent: through agent: through framework, OpenTelemetry or framework, OpenTelemetry or framework, OpenTelemetry or own orchestration— own orchestration— BrainTrust allows BrainTrust allows BrainTrust allows without hindrance without hindrance without hindrance integrate with any integrate with any what. And I think if what. And I think if what. And I think if look at the market, look at the market, look at the market, if you try if you try if you try understand how understand how understand how track work track work track work this agent, you this agent, you this agent, you in general will be needed in general will be needed in general will be needed opportunity to work opportunity to work opportunity to work with many different with many different with many different things. As you can see things. As you can see things. As you can see here, there are many different here, there are many different here, there are many different ways to integrate ways to integrate ways to integrate your stack at Braintrust. Therefore your stack at Braintrust. Therefore , again, my , again, my , again, my test agent test agent test agent today today today uses SDK for uses SDK for uses SDK for OpenAI agents, but we OpenAI agents, but we OpenAI agents, but we definitely not limited definitely not limited definitely not limited only this. This word "

only this. This word " flywheel, I, flywheel, I, flywheel, I, I think I pronounce it. I think I pronounce it. I think I pronounce it. I don't know, a hundred of them. I don't know, a hundred of them. I don't know, a hundred of them. times a day. times a day. The general idea, and that's it , about which we received , about which we received , about which we received many reviews from many reviews from many reviews from our customers, our customers, our customers, is that is that is that best agents best agents best agents are being created today are being created today are being created today exactly according to this model exactly according to this model . And I mean that . And I mean that . And I mean that productive productive productive the environment has the environment has the environment has influence the development. influence the development. You must have a very You must have a very simple and convenient in simple and convenient in simple and convenient in creating a "flywheel", creating a "flywheel", creating a "flywheel", which allows you to enter which allows you to enter which allows you to enter change, fix something change, fix something change, fix something in the agent, in the agent, in the agent, to protest, or not to protest, or not to protest, or not led to these changes led to these changes led to these changes regression or regression or regression or improved it. You improved it. You improved it. You must have must have must have confidence that confidence that confidence that to deploy this to deploy this to deploy this agent in agent in agent in productive productive productive environment. When you environment. When you environment. When you Are you deploying it or not? Are you deploying it or not? Are you deploying it or not? Yes? We need Yes? We need Yes? We need to be able to to be able to to be able to to watch him. to watch him. to watch him. However, before that However, before that However, before that we are observing we are observing we are observing we need to add something more we need to add something more —that which is possible, —that which is possible, —that which is possible, not directly highlighted in not directly highlighted in not directly highlighted in this flywheel, but this flywheel, but this flywheel, but you need to be able to you need to be able to you need to be able to to receive certain to receive certain to receive certain insights at scale. insights at scale. Most customers, Most customers, especially those who were especially those who were especially those who were on the slide earlier, on the slide earlier, on the slide earlier, have huge have huge have huge scale, and scale, and scale, and get insights get insights get insights manually or manually or manually or view this data view this data view this data a person would be a person would be a person would be incredibly difficult.

incredibly difficult. incredibly difficult. Therefore we need Therefore we need Therefore we need add level add level add level intelligence on top intelligence on top intelligence on top this, and this is what this, and this is what this, and this is what Braintrust will offer, but Braintrust will offer, but Braintrust will offer, but I think that if you I think that if you I think that if you think about the process think about the process think about the process creating a better creating a better creating a better agent in general, you agent in general, you agent in general, you need such a " need such a " flywheel". You flywheel". You flywheel". You need to create need to create need to create a system in which data from a system in which data from a system in which data from productive productive productive environments affect environments affect environments affect on how you change on how you change on how you change your agent. That's it. your agent. That's it. your agent. That's it. I am partially today. I am partially today. I am partially today. I will show you during our I will show you during our I will show you during our workshop: how we workshop: how we workshop: how we we can automatically we can automatically we can automatically to receive these to receive these to receive these insights from our insights from our insights from our support agent, support agent, support agent, bring them back into the process bring them back into the process bring them back into the process development, to contribute development, to contribute development, to contribute changes and launch changes and launch changes and launch assessment, right? assessment, right? Such a cycle becomes, Such a cycle becomes, you know, just you know, just you know, just incredibly important. incredibly important. incredibly important. And I hope this And I hope this And I hope this will become very simple will become very simple will become very simple a matter to which a matter to which a matter to which most companies most companies most companies and developers and developers and developers will be able to easily will be able to easily will be able to easily connect. Therefore, connect. Therefore, connect. Therefore, when I say "signal", in when I say "signal", in when I say "signal", in in this context I have in this context I have in this context I have meaning two different meaning two different meaning two different things that we also things that we also things that we also we will set up during we will set up during we will set up during workshop. I have on workshop. I have on workshop. I have on meaning of the assessment. It meaning of the assessment. It meaning of the assessment. It there may be, you know, there may be, you know, there may be, you know, appraisers. It's simple.

appraisers. It's simple. appraisers. It's simple. way to measure way to measure way to measure work results work results work results your agent. Do you have your agent. Do you have your agent. Do you have a certain type of error, a certain type of error, a certain type of error, or are you trying or are you trying or are you trying to understand something about to understand something about to understand something about agent when he agent when he agent when he receives certain incoming receives certain incoming receives certain incoming data. We can data. We can data. We can create a lot create a lot create a lot different types of assessments. different types of assessments. different types of assessments. One of them, which you One of them, which you One of them, which you you see on the left, it's you see on the left, it's you see on the left, it's program evaluation. program evaluation. program evaluation. They are usually very They are usually very They are usually very fast, cheap and fast, cheap and fast, cheap and easy to perform. easy to perform. easy to perform. You can imagine that I You can imagine that I You can imagine that I I'm trying to check. I'm trying to check. I'm trying to check. data schema. I data schema. I data schema. I I'm trying to understand. I'm trying to understand. I'm trying to understand. was it called was it called was it called tool A before tool A before tool A before tool B? Or tool B? Or tool B? Or was this caused by was this caused by was this caused by correct correct correct sequence? It sequence? It sequence? It things that can be things that can be things that can be get very quickly get very quickly get very quickly and without need and without need and without need engage LLM. More engage LLM. More engage LLM. More subjective, however, is subjective, however, is subjective, however, is judge based on LLM. We judge based on LLM. We judge based on LLM. We we want to add some we want to add some we want to add some human understanding of human understanding of human understanding of this tracking, or this tracking, or this tracking, or wrong? Until that one wrong? Until that one wrong? Until that one interactions that just interactions that just interactions that just took place, with this took place, with this took place, with this assessment case or assessment case or assessment case or even in mode even in mode even in mode online. So you can online. So you can online. So you can to provide LLM certain to provide LLM certain to provide LLM certain criteria for how you criteria for how you criteria for how you want to evaluate want to evaluate want to evaluate result or result or result or conversation, and she has conversation, and she has conversation, and she has to give a certain assessment.

to give a certain assessment. And, probably, there is also And, probably, there is also fourth option, fourth option, fourth option, but the third one you but the third one you but the third one you you see on the right, it's you see on the right, it's you see on the right, it's LLM judge, agreed with LLM judge, agreed with LLM judge, agreed with human assessment. human assessment. It is also important how you It is also important how you can you imagine, can you imagine, can you imagine, if I have a judge LLM, if I have a judge LLM, if I have a judge LLM, which issues which issues which issues results, and we results, and we results, and we we accept them we accept them we accept them based on the decision, we based on the decision, we based on the decision, we It is worth understanding whether these It is worth understanding whether these It is worth understanding whether these the results are the results are the results are high-quality. Therefore, it has high-quality. Therefore, it has high-quality. Therefore, it has to be present to be present to be present certain element certain element certain element human verification. human verification. human verification. Especially if you Especially if you Especially if you use these use these use these judges in the regime judges in the regime judges in the regime real time, right? real time, right? real time, right? Yes? You are evaluating Yes? You are evaluating Yes? You are evaluating real interactions real interactions real interactions users with your users with your users with your agent, you must agent, you must agent, you must to be able to to be able to to be able to to provide some to provide some to provide some interface for interface for interface for users to users to users to they could provide they could provide they could provide feedback for feedback for feedback for calibration of these calibration of these calibration of these judges, right? This is not judges, right? This is not judges, right? This is not the case when the case when the case when can be configured and can be configured and can be configured and forget. Your agent forget. Your agent forget. Your agent will develop, and will develop, and will develop, and with him and yours with him and yours with him and yours evaluations. Fourth evaluations. Fourth evaluations. Fourth option, example option, example option, example which I have on which I have on which I have on seminar, is seminar, is seminar, is programmatic evaluation, but programmatic evaluation, but programmatic evaluation, but she uses LLM "under" she uses LLM "under" she uses LLM "under" "hood". You can "hood". You can "hood". You can imagine what logic is imagine what logic is imagine what logic is branching or branching or branching or just some more just some more just some more advanced or complex advanced or complex advanced or complex things that you things that you things that you need to be done on need to be done on need to be done on based on a certain logic, based on a certain logic, based on a certain logic, which one do you want which one do you want which one do you want encode in your encode in your encode in your program evaluation. So program evaluation. So program evaluation. So you can do and you can do and you can do and it. These are some of the things it. These are some of the things it. These are some of the things which, it seems to me, which, it seems to me, which, it seems to me, I heard from different I heard from different I heard from different conversations with our conversations with our conversations with our customers. I think customers. I think customers. I think again, again, again, treat this with respect treat this with respect treat this with respect a certain proportion a certain proportion a certain proportion skepticism. This is not skepticism. This is not skepticism. This is not will always be will always be will always be suitable for 100% suitable for 100% scenarios every time. It scenarios every time. It scenarios every time. It where I saw a lot where I saw a lot where I saw a lot successes among our successes among our successes among our customer base.

customer base. customer base. Ability to understand Ability to understand Ability to understand justification that justification that justification that granted LLM for granted LLM for granted LLM for rating, rating, rating, incredibly useful, incredibly useful, incredibly useful, especially during especially during especially during human verification, human verification, human verification, when you say, "Hey, I when you say, "Hey, I when you say, "Hey, I I understand why she does this. I understand why she does this. I understand why she does this. published. This is later published. This is later published. This is later Tell me how I Tell me how I Tell me how I I can change my I can change my I can change my evaluations. Regarding binary evaluations. Regarding binary evaluations. Regarding binary assessments, especially on assessments, especially on assessments, especially on beginning when we beginning when we beginning when we we release the agent in we release the agent in we release the agent in operation, I operation, I operation, I I think it's binary. I think it's binary. I think it's binary. the score is better than, the score is better than, the score is better than, you know, grades A B C D and you know, grades A B C D and you know, grades A B C D and and so on. And you should and so on. And you should and so on. And you should to be incredible to be incredible to be incredible strict. You should strict. You should strict. You should to create really to create really to create really complex examples. complex examples. complex examples. If you get 100% If you get 100% for all of their own for all of their own for all of their own estimates, and today estimates, and today estimates, and today we probably do we probably do we probably do Let's do it, because it's ours. Let's do it, because it's ours. Let's do it, because it's ours. kind of toy kind of toy kind of toy example, if you example, if you example, if you you get 100% for you get 100% for you get 100% for estimates—first, estimates—first, estimates—first, probably in your probably in your probably in your dataset dataset dataset not strong enough not strong enough not strong enough test cases. Yours test cases. Yours test cases. Yours estimates, probably estimates, probably estimates, probably not strict enough. not strict enough. not strict enough. So you should pay attention So you should pay attention So you should pay attention attention to this and attention to this and attention to this and in accordance in accordance in accordance calibrate calibrate calibrate system. Another system. Another system. Another the moment that I the moment that I the moment that I noticed, again, noticed, again, noticed, again, with a touch of skepticism— with a touch of skepticism— adding examples adding examples adding examples in your tips in your tips in your tips usually helps, and usually helps, and maybe it's worth it maybe it's worth it maybe it's worth it use use use a more powerful model for a more powerful model for a more powerful model for evaluation than for evaluation than for evaluation than for generation, right?

generation, right? generation, right? Let's do our best Let's do our best Let's do our best let's bring it closer to let's bring it closer to let's bring it closer to human human human subjectivity. subjectivity. subjectivity. Obviously, this can Obviously, this can Obviously, this can vary depending vary depending vary depending from the type of traffic, from the type of traffic, from the type of traffic, truth? If we truth? If we truth? If we let's start talking about let's start talking about let's start talking about online assessment and online assessment and online assessment and things like that, then, things like that, then, things like that, then, actually, there are two actually, there are two actually, there are two different types of assessments, different types of assessments, different types of assessments, at least the way we do at least the way we do at least the way we do We see this in BrainTrust. THERE ARE We see this in BrainTrust. THERE ARE We see this in BrainTrust. THERE ARE offline assessment, offline assessment, offline assessment, Yes? These are our evals ( Yes? These are our evals ( evaluations). We evaluations). We evaluations). We trying to get trying to get trying to get signal whether signal whether signal whether should we launch should we launch should we launch This is work. This will give This is work. This will give This is work. This will give we understand, or we understand, or we understand, or Did it get better? Did it get better? Did it get better? Has there been a regression? You Has there been a regression? You Has there been a regression? You can you imagine how can you imagine how can you imagine how connect this to... connect this to... Sorry. This is mine. Sorry. This is mine. Sorry. This is mine. daughter. Yes, so you daughter. Yes, so you daughter. Yes, so you can you imagine how can you imagine how can you imagine how connect it, right? connect it, right? connect it, right? Yes? To CI. You can Yes? To CI. You can Yes? To CI. You can imagine how to connect imagine how to connect imagine how to connect this is for some this is for some this is for some automated automated automated systems. Again, systems. Again, systems. Again, where do you run this where do you run this where do you run this offline. And on the other hand offline. And on the other hand offline. And on the other hand side, the main idea or side, the main idea or side, the main idea or what we are talking about here what we are talking about here what we are talking about here we say, it's like we say, it's like we say, it's like receive a signal from receive a signal from receive a signal from of what actually of what actually of what actually takes place in takes place in takes place in working environment?

working environment? working environment? And one of the things you And one of the things you And one of the things you you can do, is you can do, is you can do, is use these use these use these online assessments and online assessments and online assessments and evaluate the traffic that evaluate the traffic that evaluate the traffic that comes to BrainTrust. AND comes to BrainTrust. AND comes to BrainTrust. AND again, the idea in again, the idea in again, the idea in in order to add in order to add in order to add such a signal where you are like such a signal where you are like such a signal where you are like user can user can user can to come in and say, "Hey, to come in and say, "Hey, to come in and say, "Hey, my accuracy rating my accuracy rating my accuracy rating routing routing routing is zero in these is zero in these is zero in these examples. Is there examples. Is there examples. Is there some kind of pattern, about some kind of pattern, about some kind of pattern, about which one should I know, which one should I know, which one should I know, based on this based on this based on this feedback, right? So me feedback, right? So me feedback, right? So me need to move from need to move from need to move from huge volume huge volume huge volume data to something data to something data to something much smaller and much smaller and much smaller and more convenient for more convenient for more convenient for processing". Second processing". Second processing". Second signal—is a function in signal—is a function in signal—is a function in BrainTrust called " BrainTrust called " Topics. The idea here is Topics. The idea here is Topics. The idea here is because you want because you want because you want understand understand understand patterns in patterns in patterns in your data. You your data. You your data. You Do you want to go out for a bit? Do you want to go out for a bit? Do you want to go out for a bit? further beyond the "known" further beyond the "known" further beyond the "known" unknowns", right? unknowns", right? unknowns", right? So, your estimates, which So, your estimates, which So, your estimates, which you have already created under you have already created under you have already created under offline time- offline time- evaluation, is evaluation, is evaluation, is failure modes that you, failure modes that you, failure modes that you, perhaps already discovered. perhaps already discovered. perhaps already discovered. These are things you already These are things you already These are things you already check, but this, check, but this, check, but this, certainly not all certainly not all certainly not all your work range your work range your work range agent or all possible agent or all possible agent or all possible the mistakes he made the mistakes he made the mistakes he made may collide. AND may collide. AND may collide. AND so we need so we need so we need understand more. And " understand more. And " Topics" is one of Topics" is one of Topics" is one of things that will help things that will help things that will help to provide this. We to provide this. We to provide this. We must have must have must have ability to understand ability to understand ability to understand hidden errors. You hidden errors. You hidden errors. You you can see these you can see these you can see these quite traditional quite traditional quite traditional estimates, as actual estimates, as actual estimates, as actual accuracy and moderation.

accuracy and moderation. accuracy and moderation. They will only reveal They will only reveal They will only reveal a very small part a very small part a very small part errors. And probably errors. And probably errors. And probably you have a little more you have a little more you have a little more specialized specialized specialized the grades you the grades you the grades you use. And in use. And in use. And in in this context in this context in this context What are you doing again? What are you doing again? What are you doing again? launch, launch, launch, use this use this use this a subset of the scores to a subset of the scores to a subset of the scores to to understand whether there is to understand whether there is to understand whether there is improvement or regression improvement or regression . But there is a wide . But there is a wide . But there is a wide range of options range of options range of options uses that are not uses that are not uses that are not cover your cover your cover your test scenarios where test scenarios where test scenarios where users users users actually actually actually are checking your are checking your are checking your agent, trying agent, trying agent, trying to get a certain to get a certain to get a certain the result, which, the result, which, the result, which, maybe there is not quite maybe there is not quite maybe there is not quite correct. So, we correct. So, we correct. So, we we want to cover these we want to cover these we want to cover these isolated cases. We isolated cases. We isolated cases. We we want to cover these we want to cover these we want to cover these atypical situations and atypical situations and atypical situations and to structure to structure to structure flywheel of development, or flywheel of development, or flywheel of development, or wrong? We want wrong? We want wrong? We want return these return these return these examples in our examples in our examples in our datasets. We datasets. We datasets. We we want to spend we want to spend we want to spend evaluation. We want evaluation. We want , possibly create , possibly create , possibly create new estimates based on new estimates based on new estimates based on this reverse this reverse this reverse communication. So what is it? communication. So what is it? communication. So what is it? Topics? Topics in general, or Topics? Topics in general, or Topics? Topics in general, or at a high level, — at a high level, — this is the way for us this is the way for us this is the way for us understand understand understand patterns in patterns in patterns in your data. We your data. We your data. We we can create we can create we can create clusters of these clusters of these clusters of these patterns and patterns and patterns and to give you an idea of to give you an idea of to give you an idea of the frequency with which they the frequency with which they the frequency with which they arise. Also arise. Also arise. Also It is important to note that It is important to note that It is important to note that exist as built-in, exist as built-in, exist as built-in, and custom and custom and custom aspects. Under this I have aspects. Under this I have aspects. Under this I have meaning that when you meaning that when you meaning that when you go to your go to your go to your account account account BrainTrust, you can BrainTrust, you can BrainTrust, you can activate the function activate the function activate the function Topics and see immediately Topics and see immediately Topics and see immediately tasks, problems and tasks, problems and tasks, problems and mood. So I can mood. So I can mood. So I can understand: "Hey, huh?"

understand: "Hey, huh?" understand: "Hey, huh?" what is the typical intention what is the typical intention what is the typical intention user? As user? As user? As feel feel feel users during users during users during interactions with my interactions with my interactions with my agent? And what are they? agent? And what are they? agent? And what are they? common problems in common problems in common problems in agent's work? So, I agent's work? So, I agent's work? So, I I can use I can use I can use this taxonomy where I don't this taxonomy where I don't this taxonomy where I don't I will definitely provide it. I will definitely provide it. I will definitely provide it. in advance, because LLM in advance, because LLM in advance, because LLM creates it for me creates it for me creates it for me as part of this as part of this as part of this conveyor. But I also conveyor. But I also conveyor. But I also I can add my own. I can add my own. I can add my own. customizable customizable customizable aspects. I can aspects. I can aspects. I can create your own create your own create your own request and own request and own request and own conveyor to detect conveyor to detect conveyor to detect these patterns in these patterns in these patterns in your data, right? your data, right? your data, right? Regarding things that Regarding things that Regarding things that I am interested. Er... I am interested. Er... Let's get back to this. Let's get back to this. This, you know, is simple. This, you know, is simple. This, you know, is simple. example illustration, example illustration, example illustration, which you can which you can which you can to see from to see from to see from generated topics generated topics . But why do you have this? . But why do you have this? . But why do you have this? use? I use? I use? I already hinted at a little already hinted at a little already hinted at a little this, but you want this, but you want this, but you want avoid "blind spots" avoid "blind spots" ". You want to understand ". You want to understand ". You want to understand all the different ways in which all the different ways in which all the different ways in which which users which users which users interact with your interact with your interact with your agents, and then agents, and then agents, and then use this use this use this for decision-making for decision-making for decision-making as a developer, right? It as a developer, right? It as a developer, right? It not even necessary not even necessary not even necessary there must be a problem or there must be a problem or there must be a problem or malfunction. It malfunction. It malfunction. It can be can be can be potential requests for potential requests for potential requests for new features or new new features or new new features or new your possibilities your possibilities your possibilities agent based on agent based on agent based on because users because users because users actually with him actually with him actually with him interact. So, this is, interact. So, this is, interact. So, this is, in my opinion, one of in my opinion, one of in my opinion, one of the ways in which I see the ways in which I see , many of our , many of our , many of our users users users start this start this start this use- use- in fact, to in fact, to in fact, to create this or, create this or, create this or, let's say so, let's say so, let's say so, understand that understand that understand that is happening, and is happening, and is happening, and to determine how it should to determine how it should to determine how it should to look expensive to look expensive to look expensive a card for an agent, right?

a card for an agent, right? a card for an agent, right? This is starting to show This is starting to show This is starting to show things we don't talk about things we don't talk about things we don't talk about must have known must have known must have known regarding the work of these regarding the work of these regarding the work of these agents in agents in agents in production production production environment. By environment. By environment. By behind the scenes it is behind the scenes it is behind the scenes it is actually very actually very actually very complex conveyor. You complex conveyor. You complex conveyor. You you see there on the left you see there on the left you see there on the left preprocessor. So, preprocessor. So, preprocessor. So, if you remember mine if you remember mine if you remember mine the first slide with that the first slide with that the first slide with that tracing, then this tracing, then this tracing, then this very, very small very, very small very, very small should be compared to should be compared to should be compared to that, as I saw, that, as I saw, that, as I saw, generate our generate our generate our clients. We need clients. We need clients. We need pre-process pre-process pre-process this data into something this data into something this data into something meaningful, so that meaningful, so that meaningful, so that transfer them to LLM. So transfer them to LLM. So transfer them to LLM. So we provide you with what we provide you with what we provide you with what we call we call we call standard standard standard preprocessor. We are preprocessor. We are preprocessor. We are we take what you mean we take what you mean we take what you mean you will see in our you will see in our you will see in our viewing streams, and viewing streams, and viewing streams, and you'll see it in a bit you'll see it in a bit you'll see it in a bit later, but it's an exchange later, but it's an exchange later, but it's an exchange messages, right? messages, right? messages, right? This is the user, this is This is the user, this is This is the user, this is assistant, these are challenges assistant, these are challenges assistant, these are challenges tools. We tools. We tools. We previously previously previously we are processing this we are processing this we are processing this a massive footprint in something a massive footprint in something a massive footprint in something meaningful, significant meaningful, significant meaningful, significant Less. Then we Less. Then we Less. Then we we pass this on to the facet, we pass this on to the facet, we pass this on to the facet, Yes? Facet is what I am Yes? Facet is what I am Yes? Facet is what I am just described: just described: just described: problems, tasks, problems, tasks, problems, tasks, tonality, yours tonality, yours tonality, yours own built-ins own built-ins own built-ins facets. What are we facets. What are we facets. What are we we are gathering here we are gathering here we are gathering here to do: these to do: these to do: these pre-treated pre-treated pre-treated data will be transferred data will be transferred data will be transferred to the facet, and we them to the facet, and we them to the facet, and we them Let's summarize. For Let's summarize. For Let's summarize. For this summary we this summary we this summary we create a vector create a vector create a vector embedding in embedding in embedding in our backend. Except our backend. Except our backend. Except as soon as we as soon as we as soon as we we have this vector we have this vector we have this vector embedding, we embedding, we embedding, we we can create that we can create that we can create that what do we call what do we call what do we call topic map. It topic map. It topic map. It approximately what you approximately what you approximately what you seen on seen on seen on previous slide previous slide previous slide on the diagram on the diagram on the diagram scattering. This is what scattering. This is what scattering. This is what allows us allows us allows us to mark our tracks to mark our tracks to mark our tracks and classify them.

and classify them. and classify them. And if I have these And if I have these And if I have these tags and tags and tags and classifiers, for classifiers, for classifiers, for me as a user me as a user me as a user it becomes incredible it becomes incredible it becomes incredible simply, and also simply, and also simply, and also incredibly simple incredibly simple incredibly simple for me as for me as for me as user who user who user who uses coding- uses coding- Braintrust-based agent, Braintrust-based agent, Braintrust-based agent, start filtering start filtering start filtering this noise, this noise, this noise, using all using all using all these are extremely these are extremely these are extremely rich data. Important rich data. Important rich data. Important note: conveyor, note: conveyor, note: conveyor, which you see, which you see, which you see, facets and step facets and step facets and step embedding embedding embedding actually actually actually use use use hosted by us hosted by us hosted by us model. So we just model. So we just model. So we just accomplished a lot accomplished a lot accomplished a lot behind the scenes work, behind the scenes work, behind the scenes work, to create not to create not to create not just a function that just a function that just a function that can only be shown in can only be shown in can only be shown in demo. Of course, we demo. Of course, we demo. Of course, we let's see something here let's see something here let's see something here today, but it has today, but it has today, but it has work at scale. work at scale. work at scale. You must have You must have You must have ability to launch ability to launch ability to launch This is 100% your tracks. This is 100% your tracks. This is 100% your tracks. This, in fact, is the goal. This, in fact, is the goal. This, in fact, is the goal. our topic pipeline. our topic pipeline. our topic pipeline. This, uh, to get into This, uh, to get into This, uh, to get into details, perhaps details, perhaps details, perhaps even too much, 6 even too much, 6 even too much, 6 cents per million cents per million cents per million tokens at the entrance and 40 tokens at the entrance and 40 tokens at the entrance and 40 cents at the exit. cents at the exit. cents at the exit. Compare this with Compare this with Compare this with other models that other models that other models that you could you could you could use as LLM use as LLM -judge for creation -judge for creation -judge for creation the same signal, and you the same signal, and you the same signal, and you will get enough will get enough will get enough a strong option a strong option a strong option using, using, using, especially if you especially if you especially if you you will begin to see you will begin to see you will begin to see value in topics, value in topics, value in topics, labels and labels and labels and classifiers that classifiers that classifiers that we generate as we generate as we generate as result. Uh, I already result. Uh, I already result. Uh, I already I mentioned this a little bit.

I mentioned this a little bit. : why do we do this : why do we do this : why do we do this created? We have created? We have created? We have customers who customers who customers who generate huge generate huge generate huge data volumes. And one of data volumes. And one of data volumes. And one of things that we things that we things that we decided on their own decided on their own decided on their own beginning, is beginning, is beginning, is possibility possibility possibility receive this data, and receive this data, and receive this data, and then allow then allow then allow to our customers to our customers to our customers to make inquiries to them to make inquiries to them to make inquiries to them almost instantly. What do you have? almost instantly. What do you have? almost instantly. What do you have? we were not there ourselves we were not there ourselves we were not there ourselves beginning, so it is beginning, so it is beginning, so it is possibilities something from possibilities something from possibilities something from to do this, right? AND to do this, right? AND to do this, right? AND right here in the game right here in the game right here in the game enters active enters active enters active observability. observability. observability. How are we now? How are we now? How are we now? get insights from get insights from get insights from all this data, all this data, all this data, especially from these especially from these especially from these volumes? Here it is. volumes? Here it is. volumes? Here it is. this one appears this one appears this one appears challenge. So if you have challenge. So if you have challenge. So if you have is so complicated is so complicated is so complicated conveyor, architecture conveyor, architecture conveyor, architecture which allows which allows which allows to work on a large scale, to work on a large scale, to work on a large scale, and I mean like and I mean like and I mean like technical perspective technical perspective , as valuable as I am , as valuable as I am , as valuable as I am mentioned earlier, then mentioned earlier, then mentioned earlier, then we can provide we can provide we can provide really interesting really interesting really interesting the foundation on which the foundation on which the foundation on which our customers can our customers can our customers can build your own build your own build your own system. After all, we system. After all, we system. After all, we we try to give we try to give we try to give our customers the opportunity our customers the opportunity our customers the opportunity to build such to build such to build such a system that improves a system that improves a system that improves the quality of their work the quality of their work the quality of their work agents very, very agents very, very agents very, very quickly. OK.

quickly. OK. quickly. OK. I hope I don't offend anyone. I hope I don't offend anyone. I hope I don't offend anyone. not lost during not lost during not lost during presentations and you all presentations and you all presentations and you all ready for the workshop. ready for the workshop. Raise your hands, who? Raise your hands, who? still need a QR code or still need a QR code or still need a QR code or link to link to link to GitHub repository? Yes. GitHub repository? Yes. Yes, I will hold it. Yes, I will hold it. here are thirty seconds. here are thirty seconds. My team My team uses Lightspin. I uses Lightspin. I uses Lightspin. I I think Lightspin is I think Lightspin is I think Lightspin is so-so. Can you so-so. Can you so-so. Can you give me a list give me a list give me a list arguments so that my arguments so that my arguments so that my the team understood, the team understood, the team understood, why should they why should they why should they use this use this use this instead of Lightspin? instead of Lightspin? Wow, you're right about that. Wow, you're right about that. You ask your competitors, You ask your competitors, You ask your competitors, Yes? Wow. Wasn't there before Yes? Wow. Wasn't there before Yes? Wow. Wasn't there before I'm ready for this. I I'm ready for this. I I'm ready for this. I definitely not the one who definitely not the one who definitely not the one who will stand here and will stand here and will stand here and will criticize someone will criticize someone . Hmm, I really think so.

. Hmm, I really think so. . Hmm, I really think so. Which of us is stronger? Which of us is stronger? Which of us is stronger? proposal, and I think proposal, and I think proposal, and I think that she is stronger in that she is stronger in that she is stronger in many different many different many different aspects. Ahem, aspects. Ahem, aspects. Ahem, of course, not in of course, not in of course, not in to some certain one to some certain one to some certain one order. One of them— order. One of them— this is the flexibility that I this is the flexibility that I this is the flexibility that I showed you on another showed you on another showed you on another slide. This is an opportunity slide. This is an opportunity slide. This is an opportunity to work with many to work with many to work with many in many languages in many languages in many languages programming languages programming languages , I apologize, , I apologize, , I apologize, many many many frameworks, it's very frameworks, it's very frameworks, it's very universal way universal way universal way creating your creating your creating your agent. Uh, LangSmith, I agent. Uh, LangSmith, I agent. Uh, LangSmith, I I think it also has something to do with I think it also has something to do with I think it also has something to do with this, but they this, but they this, but they definitely definitely definitely oriented to LangGraph, oriented to LangGraph, oriented to LangGraph, uh LangChain, deep uh LangChain, deep uh LangChain, deep agents and the like agents and the like agents and the like things, and I think they have things, and I think they have things, and I think they have there are functions that there are functions that there are functions that intended only for intended only for intended only for this. Um, another thing, this. Um, another thing, this. Um, another thing, which I think about if which I think about if which I think about if to go back, to go back, to go back, I actually had I actually had I actually had slide about it, but slide about it, but slide about it, but BrainTrust created BrainTrust created BrainTrust created own database own database own database at the end of 2024. AND at the end of 2024. AND at the end of 2024. AND the reason why we the reason why we the reason why we created this database created this database created this database data, is that, data, is that, data, is that, that Ankur, our that Ankur, our that Ankur, our CEO CEO , realized very early , realized very early , realized very early the trajectory we are on the trajectory we are on the trajectory we are on moved from the point of view moved from the point of view moved from the point of view agents. So he, we agents. So he, we agents. So he, we used used used ClickHouse behind the scenes in ClickHouse behind the scenes in ClickHouse behind the scenes in that time and very quickly that time and very quickly that time and very quickly understood when understood when understood when attracted such attracted such attracted such customers you have there customers you have there customers you have there seen, Netflix, Microsoft and seen, Netflix, Microsoft and seen, Netflix, Microsoft and Dropbox, what ClickHouse is simple Dropbox, what ClickHouse is simple Dropbox, what ClickHouse is simple "went to bed" very quickly "went to bed" very quickly "went to bed" very quickly when scaling. AND when scaling. AND when scaling. AND we had to do we had to do we had to do something else. So exactly something else. So exactly something else. So exactly here is the team here is the team here is the team engineers and anchors engineers and anchors engineers and anchors created BrainStore. It created BrainStore. It created BrainStore. It our own our own our own specially specially specially backend developed.

backend developed. backend developed. I think at that time people I think at that time people I think at that time people considered us considered us considered us crazy. They crazy. They crazy. They thought, "Hey, there's everyone thought, "Hey, there's everyone thought, "Hey, there's everyone these alternatives with these alternatives with these alternatives with open source. THERE ARE open source. THERE ARE open source. THERE ARE ClickHouse. There are such things, ClickHouse. There are such things, ClickHouse. There are such things, which ones actually, you know which ones actually, you know , allow us , allow us , allow us do this and do that do this and do that do this and do that "Okay." What do you see? "Okay." What do you see? "Okay." What do you see? now, and especially, as now, and especially, as now, and especially, as you probably see from you probably see from you probably see from LangSmith's side, they just LangSmith's side, they just LangSmith's side, they just created their own database created their own database created their own database data. Um, Arise, they data. Um, Arise, they data. Um, Arise, they just created just created just created own database own database own database about 6 months ago. about 6 months ago. about 6 months ago. I think people are starting to I think people are starting to I think people are starting to to realize that this to realize that this to realize that this actually the problem is, actually the problem is, actually the problem is, which is not resolved which is not resolved which is not resolved ready-made solutions, and ready-made solutions, and ready-made solutions, and you need something you need something you need something individual. And therefore individual. And therefore , in this regard, I , in this regard, I , in this regard, I I think we're about a year away. I think we're about a year away. I think we're about a year away. we are ahead of some we are ahead of some we are ahead of some from these competitors. AND from these competitors. AND from these competitors. AND where it manifests itself, where it manifests itself, where it manifests itself, so it is in the topics. It so it is in the topics. It so it is in the topics. It ability to generate ability to generate ability to generate real conclusions on real conclusions on real conclusions on based on data in based on data in based on data in on a large scale. Ahem on a large scale. Ahem , there are probably others. , there are probably others. , there are probably others. If you want, you know, If you want, you know, If you want, you know, find me later, find me later, find me later, we can we can we can discuss this discuss this discuss this more details. But, um, I more details. But, um, I more details. But, um, I I would say that the last one I would say that the last one I would say that the last one point, such as point, such as point, such as having your own having your own having your own backend and things that backend and things that backend and things that we were able to build we were able to build we were able to build on its basis, is on its basis, is on its basis, is what actually what actually what actually allowed us, as allowed us, as allowed us, as me, stand out me, stand out me, stand out among other players in among other players in among other players in in this area. Yes. Yes, you in this area. Yes. Yes, you . Yes. Yes, I have.

. Yes. Yes, I have. . Yes. Yes, I have. mind, during mind, during mind, during workshop, sorry, workshop, sorry, workshop, sorry, if anyone didn't hear, if anyone didn't hear, if anyone didn't hear, the question was about the question was about the question was about examples of topics and a little examples of topics and a little examples of topics and a little more understanding more understanding more understanding how, perhaps, how, perhaps, how, perhaps, the process is working, and whether the process is working, and whether the process is working, and whether we create something we create something we create something really meaningful really meaningful really meaningful excellent because excellent because excellent because traces relative traces relative traces relative are the same. They will be are the same. They will be are the same. They will be semantically similar. semantically similar. semantically similar. Um, I hope you Um, I hope you Um, I hope you you will see this during you will see this during you will see this during workshop. I think that workshop. I think that workshop. I think that customers who customers who customers who use this in use this in use this in production, production, production, probably not probably not probably not would agree with you. would agree with you. would agree with you. That's why they do it. That's why they do it. That's why they do it. are used because are used because are used because they see these they see these they see these differences, and they differences, and they differences, and they see how they see how they see how they appear as different appear as different appear as different classifiers, like classifiers, like classifiers, like things they still talk about things they still talk about things they still talk about didn't know until didn't know until didn't know until launched this launched this launched this conveyor. Hmm, I think so. conveyor. Hmm, I think so. conveyor. Hmm, I think so. part of the case part of the case part of the case is to is to is to to try a little to try a little to try a little independently and truly independently and truly independently and truly add your own add your own add your own Traces. For example, in Traces. For example, in Traces. For example, in I have fictitious demos I have fictitious demos data, where am I, you know, data, where am I, you know, data, where am I, you know, used a script, used a script, used a script, to create certain to create certain to create certain the differences that we the differences that we the differences that we We generate here. I think We generate here. I think We generate here. I think you will naturally see you will naturally see you will naturally see even more so, even more so, even more so, when real when real when real users users users will interact with will interact with will interact with agent. Um, Tracey agent. Um, Tracey agent. Um, Tracey they probably look like they probably look like they probably look like somewhat similar, right? We somewhat similar, right? We somewhat similar, right? We we call the same ones we call the same ones we call the same ones tools. All tools. All tools. All passes through that passes through that passes through that the work process itself.

the work process itself. the work process itself. But I think, you know, But I think, you know, But I think, you know, depending on what depending on what depending on what what does your agent do and what what does your agent do and what what does your agent do and what your users your users your users do with your do with your do with your agent, and I think agent, and I think agent, and I think perhaps, retreating perhaps, retreating perhaps, retreating a little back, a lot a little back, a lot a little back, a lot Which of us builds? Which of us builds? Which of us builds? agents who become agents who become agents who become more complex, or more complex, or more complex, or build systems that build systems that build systems that are becoming more complex. AND are becoming more complex. AND are becoming more complex. AND this creates, I think, this creates, I think, this creates, I think, some differences, some differences, some differences, which will be visible on which will be visible on which will be visible on screen. But you screen. But you screen. But you you will see when you will see when you will see when create an account, in create an account, in create an account, in you will actually be you will actually be you will actually be free account. free account. free account. You can send You can send You can send data up to a certain limit. data up to a certain limit. data up to a certain limit. You can even You can even You can even enable themes to enable themes to enable themes to to see if it's true to see if it's true to see if it's true it generates something it generates something it generates something similar. So we similar. So we similar. So we just encouraging just encouraging just encouraging you even after you even after you even after this workshop this workshop this workshop to experiment with to experiment with to experiment with hereby. Hmm, and look, hereby. Hmm, and look, hereby. Hmm, and look, will it generate for will it generate for will it generate for Are there any interesting topics for you? Are there any interesting topics for you? . Yes, go ahead. . Yes, go ahead. bring your own bring your own own traces, so own traces, so own traces, so that you don't show them that you don't show them that you don't show them to the client you to the client you to the client you use use use own own own Yes. Yes, the question is this. Yes. Yes, the question is this. : can we : can we : can we use themes use themes , perhaps, for example, , perhaps, for example, , perhaps, for example, in our own in our own in our own sessions with Cloud Code, Codex sessions with Cloud Code, Codex sessions with Cloud Code, Codex etc? Certainly.

etc? Certainly. etc? Certainly. One of the things you One of the things you One of the things you you will see if you will see if you will see if switch to ours switch to ours switch to ours integrations page, integrations page, integrations page, there is integration with Cloud Code, there is integration with Cloud Code, there is integration with Cloud Code, integration with Codex, uh, integration with Codex, uh, integration with Codex, uh, Cursor, OpenCode. And you can Cursor, OpenCode. And you can ...I am currently working with ...I am currently working with ...I am currently working with several clients, several clients, several clients, who are trying who are trying who are trying to do exactly that and to do exactly that and to do exactly that and to overlay themes to overlay themes to overlay themes on top of this. I think on top of this. I think on top of this. I think many...I definitely many...I definitely many...I definitely I don't know where from. I don't know where from. I don't know where from. this question arose, this question arose, this question arose, but, as far as I'm concerned, but, as far as I'm concerned, but, as far as I'm concerned, most engineers most engineers most engineers today, most today, most today, most engineering teams engineering teams engineering teams use these use these use these tools, and now, tools, and now, tools, and now, I think it is necessary. I think it is necessary. I think it is necessary. understanding understanding understanding efficiency, or efficiency, or efficiency, or how good how good how good these things really these things really these things really improve the working improve the working improve the working process, and therefore process, and therefore process, and therefore things like that. And I things like that. And I things like that. And I I think that probably more I think that probably more I think that probably more need to be configured need to be configured need to be configured a certain mechanism, but a certain mechanism, but a certain mechanism, but if you just want if you just want if you just want understand that people understand that people understand that people do in these different do in these different do in these different tools, I tools, I tools, I I think that topics are I think that topics are I think that topics are very good way very good way very good way start to understand it. start to understand it. Yes. So, we have Yes. So, we have several different several different several different hosting options. hosting options. hosting options. One of them is full One of them is full One of them is full SaaS. This is what you all SaaS. This is what you all SaaS. This is what you all will be will be will be use use use today. We have both today. We have both today. We have both BYOC model, and BYOC model, and BYOC model, and hybrid model. AND, hybrid model. AND, hybrid model. AND, maybe to get away maybe to get away maybe to get away even further from this even further from this even further from this Question: BrainTrust with Question: BrainTrust with Question: BrainTrust with the first day was the first day was the first day was designed for designed for designed for hybrid support hybrid support hybrid support deployment. Under this deployment. Under this deployment. Under this I mean, uh, I mean, uh, I mean, uh, control plane, control plane, control plane, web application, right?

web application, right? web application, right? The metadata we The metadata we The metadata we we keep, always we keep, always we keep, always will be placed in will be placed in will be placed in us. Data plane, us. Data plane, us. Data plane, that is, yours that is, yours that is, yours tracing, sets tracing, sets tracing, sets data, experiments, data, experiments, data, experiments, actually can be actually can be actually can be hosted in your VPC, hosted in your VPC, hosted in your VPC, in your in your in your infrastructure. We infrastructure. We infrastructure. We we are expanding into all we are expanding into all we are expanding into all main cloud main cloud main cloud providers. But yes, providers. But yes, providers. But yes, actually, we have you actually, we have you actually, we have you you know, those clients, you know, those clients, you know, those clients, which you saw there, I which you saw there, I which you saw there, I I would bet that 90% of them I would bet that 90% of them —these are the ones that —these are the ones that —these are the ones that use use use self-hosting. self-hosting. Yes. Yes, sorry. IN Yes. Yes, sorry. IN green shirt. Yes, green shirt. Yes, green shirt. Yes, the question is how you the question is how you the question is how you working with long-term working with long-term working with long-term conversations for conversations for conversations for using themes? Some of using themes? Some of using themes? Some of this...I'll show you this...I'll show you this...I'll show you this is during the workshop, this is during the workshop, this is during the workshop, but there are different but there are different but there are different settings that settings that settings that can be configured, can be configured, can be configured, working with different working with different working with different tracings, or tracings, or tracings, or can be can be can be caused only by caused only by caused only by a certain period of time. a certain period of time.

So, depending on So, depending on that, generally speaking, that, generally speaking, that, generally speaking, I think there is understanding. I think there is understanding. I think there is understanding. how long how long how long usually last usually last usually last conversations that help conversations that help conversations that help to determine to determine to determine settings that you settings that you settings that you create. But I don't create. But I don't create. But I don't I know, participating in I know, participating in I know, participating in workshop, probably workshop, probably workshop, probably answers this answers this answers this a little better. But I a little better. But I a little better. But I I think we can. I think we can. I think we can. answer this. answer this. answer this. Steeply. How about another one? Steeply. How about another one? Steeply. How about another one? one question, and one question, and one question, and then we will begin. Yes, then we will begin. Yes, then we will begin. Yes, this is a preprocessor, we this is a preprocessor, we this is a preprocessor, we we provide you we provide you we provide you standard option. standard option. standard option. You can You can You can use use use own special own special own special preprocessor. If in preprocessor. If in preprocessor. If in you have data that exists you have data that exists you have data that exists beyond what beyond what beyond what we can sort out from we can sort out from we can sort out from incoming and outgoing incoming and outgoing incoming and outgoing data or what data or what data or what enters the LLM and enters the LLM and enters the LLM and comes out of it. You have comes out of it. You have comes out of it. You have there is information in there is information in there is information in metadata. Do you have metadata. Do you have metadata. Do you have data in other data in other data in other ranges that we, ranges that we, ranges that we, maybe not maybe not maybe not we process we process we process standard standard standard preprocessor, you preprocessor, you preprocessor, you you can create your own you can create your own you can create your own own. One more thing, own. One more thing, own. One more thing, which I think I'm talking about which I think I'm talking about which I think I'm talking about remembered, —this remembered, —this remembered, —this facets. You facets. You facets. You you can add your own you can add your own you can add your own aspects. You can aspects. You can aspects. You can create your own create your own create your own system that system that system that trying trying trying to understand or for to understand or for to understand or for detection detection detection patterns in patterns in patterns in within BrainTrust. But yes, within BrainTrust. But yes, within BrainTrust. But yes, I think another one is very I think another one is very I think another one is very cool feature cool feature cool feature this conveyor in that this conveyor in that that he is completely that he is completely that he is completely flexible at its best flexible at its best flexible at its best understanding this understanding this understanding this words. Class. I see 30- words. Class. I see 30- minute timer.

minute timer. minute timer. Although I think we have Although I think we have Although I think we have there is time until 11:00. Class. there is time until 11:00. Class. So, I assume that So, I assume that everyone has access to everyone has access to everyone has access to seminar, QR code or seminar, QR code or seminar, QR code or at least until at least until at least until repository. Is it worth it? repository. Is it worth it? repository. Is it worth it? we need to increase it a little we need to increase it a little we need to increase it a little scale? OK, scale? OK, scale? OK, the situation may be a little the situation may be a little the situation may be a little vary depending on vary depending on vary depending on depending on whether you have depending on whether you have depending on whether you have organization in BrainTrust. organization in BrainTrust. organization in BrainTrust. As I said, Noah As I said, Noah As I said, Noah here, behind, and maybe here, behind, and maybe here, behind, and maybe help if you have help if you have help if you have difficulties will arise. difficulties will arise. difficulties will arise. Besides, you know, Besides, you know, Besides, you know, if you just want if you just want if you just want shout a little at shout a little at shout a little at me, this too me, this too me, this too normally. So, normally. So, normally. So, obviously, the first thing obviously, the first thing obviously, the first thing we will need—this we will need—this we will need—this organization in BrainTrust, organization in BrainTrust, organization in BrainTrust, and I will walk through this and I will walk through this and I will walk through this manual along with manual along with manual along with you so that we can start you so that we can start you so that we can start understand everything that I understand everything that I understand everything that I just described. We just described. We just described. We we want to create this we want to create this we want to create this mechanism, and I think we mechanism, and I think we mechanism, and I think we a little more a little more a little more let's focus on let's focus on let's focus on online side, but online side, but online side, but just know what is there and just know what is there and just know what is there and the reverse side— the reverse side— evaluations (evals), and in evaluations (evals), and in evaluations (evals), and in I have one too.

I have one too. I have one too. evaluation as evaluation as evaluation as part of this, what about part of this, what about part of this, what about We can talk. We can talk. So, a little teaser So, a little teaser came out by accident, but came out by accident, but came out by accident, but we will create a new one we will create a new one we will create a new one organization. If this organization. If this organization. If this the first organization, the first organization, the first organization, that you create, you that you create, you that you create, you need to go for need to go for need to go for this link. On this link. On this link. On you may start you may start you may start there will be slightly different steps. there will be slightly different steps. You may be asked You may be asked to indicate to indicate to indicate AI providers and AI providers and AI providers and generate an API key. generate an API key. Please do it. Please do it. it. I also have it. I also have it. I also have steps for this. You steps for this. You steps for this. You you will see a little different you will see a little different you will see a little different process, because in process, because in process, because in I have already been created. I have already been created. I have already been created. organizations in BrainTrust. organizations in BrainTrust. organizations in BrainTrust. Do you see such a Do you see such a Do you see such a the same screen as now the same screen as now the same screen as now do I have? Quickly do I have? Quickly do I have? Quickly Raise your hands. Or Raise your hands. Or Raise your hands. Or most people most people most people create a new one create a new one create a new one BrainTrust organization BrainTrust organization BrainTrust organization for the first time? Okay, cool.

for the first time? Okay, cool. for the first time? Okay, cool. Perfectly. I am pleased. Perfectly. I am pleased. Perfectly. I am pleased. to see it. We to see it. We to see it. We created an organization created an organization . Is the next step . Is the next step . Is the next step will be created will be created will be created project? Sorry? project? Sorry? O. Problems with O. Problems with the internet. I love it the internet. I love it the internet. I love it it. Okay, so no one it. Okay, so no one it. Okay, so no one cannot participate. cannot participate. Okay, that's a shame. I don't know. Okay, that's a shame. I don't know. was anyone here was anyone here was anyone here last year. We have last year. We have last year. We have was exactly the same was exactly the same was exactly the same problem. Well, we problem. Well, we problem. Well, we let's just go through let's just go through let's just go through this, and I this, and I this, and I I will try not to I will try not to I will try not to to hurry, knowing that to hurry, knowing that to hurry, knowing that no one else here no one else here no one else here doesn't do this, but doesn't do this, but So, you will see that So, you will see that we created we created we created organization. You too organization. You too organization. You too you will see where we are you will see where we are you will see where we are automatically created automatically created automatically created "my project". You can "my project". You can "my project". You can delete this if delete this if delete this if you want. We are going you want. We are going you want. We are going create your own create your own create your own a project for this a project for this a project for this seminar. I will call seminar. I will call seminar. I will call his AIE workshop. All code his AIE workshop. All code his AIE workshop. All code in the repository in the repository in the repository assumes that you are assumes that you are assumes that you are done. Of course you done. Of course you done. Of course you you can't do this you can't do this you can't do this to do, but to change to do, but to change to do, but to change something else in something else in something else in repositories, but I repositories, but I repositories, but I just wanted just wanted just wanted to emphasize this, to emphasize this, to emphasize this, if you follow if you follow if you follow progress of work. Simply progress of work. Simply progress of work. Simply know what you need know what you need know what you need need need need to make sure that to make sure that to make sure that the project you the project you the project you defined here, defined here, defined here, coincides with the fact that in coincides with the fact that in coincides with the fact that in you are in the repository.

you are in the repository. So, two more things that So, two more things that we will need to we will need to we will need to to finish this: to- to finish this: to- first, we need first, we need first, we need generate or generate or generate or create AI- create AI- provider. So if provider. So if provider. So if you will go to BrainTrust you will go to BrainTrust you will go to BrainTrust and open and open and open settings, you settings, you settings, you you will see a place where you will see a place where you will see a place where can be customized can be customized can be customized their AI providers. their AI providers. I'll click on this. I'll click on this. link above. There link above. There link above. There there must be a lot there must be a lot there must be a lot different places where different places where different places where you can add an API key you can add an API key you can add an API key or what we just did or what we just did or what we just did released—the federation released—the federation released—the federation identification identification identification workloads workloads workloads for some of our for some of our for some of our basic AI- basic AI- providers. But ours providers. But ours providers. But ours organizations also organizations also organizations also often do or often do or often do or build their own locks. build their own locks. And they want And they want connect your gateway connect your gateway connect your gateway to BrainTrust. They can to BrainTrust. They can to BrainTrust. They can do it for do it for do it for with help with help with help special special special provider. You provider. You provider. You provide the main provide the main provide the main specification, which specification, which specification, which this gateway responds, this gateway responds, this gateway responds, as we as we as we authenticate to authenticate to authenticate to him and other types him and other types him and other types URL authentication. URL authentication. URL authentication. So in general you can So in general you can So in general you can connect to connect to connect to many different AI- many different AI- different providers different providers different providers ways. I haven't yet.

ways. I haven't yet. ways. I haven't yet. saw that someone didn't saw that someone didn't saw that someone didn't managed to connect managed to connect managed to connect to something with BrainTrust. to something with BrainTrust. Another thing, perhaps, Another thing, perhaps, if you follow if you follow if you follow process and process and process and using a gateway, using a gateway, using a gateway, but this gateway has but this gateway has but this gateway has certain firewalls, you, certain firewalls, you, certain firewalls, you, you will probably see you will probably see you will probably see some mistakes from this some mistakes from this some mistakes from this side. Keep in mind, side. Keep in mind, side. Keep in mind, that there are ways that there are ways that there are ways to grant access to certain to grant access to certain to grant access to certain IP addresses to those IP addresses to those IP addresses to those places that may places that may places that may only allow only allow only allow certain IP addresses, certain IP addresses, certain IP addresses, for example, through a gateway for example, through a gateway . I will stop the show. . I will stop the show. . I will stop the show. screen per second, screen per second, screen per second, so that you don't see mine so that you don't see mine so that you don't see mine API key. Okay, I API key. Okay, I API key. Okay, I I'll generate more now.

I'll generate more now. I'll generate more now. one API key. Good. one API key. Good. OK. Therefore, OK. Therefore, summing up summing up summing up done: I have moved done: I have moved done: I have moved to the section to the section to the section AI providers. I AI providers. I AI providers. I added API keys for OpenAI added API keys for OpenAI added API keys for OpenAI and Anthropic. Reason, and Anthropic. Reason, and Anthropic. Reason, why is this necessary, why is this necessary, why is this necessary, is that is that is that if you are setting up if you are setting up if you are setting up online assessment and online assessment and online assessment and use use use judges, you judges, you judges, you will have to choose will have to choose will have to choose a certain model. Another a certain model. Another a certain model. Another moment: when you moment: when you moment: when you cause these assessments, cause these assessments, cause these assessments, you use you use you use own models. Only own models. Only own models. Only the exception is the conveyor belt the exception is the conveyor belt the exception is the conveyor belt the topic that the topic that the topic that uses our uses our uses our built-in models built-in models built-in models BrainTrust for complex BrainTrust for complex BrainTrust for complex tasks. OK. Therefore, tasks. OK. Therefore, tasks. OK. Therefore, we actually did it we actually did it we actually did it "zero step". We "zero step". We "zero step". We created a new one created a new one created a new one organization. We have organization. We have organization. We have a project called "AI a project called "AI a project called "AI "e-workshop". We have "e-workshop". We have "e-workshop". We have AI and API provider AI and API provider key. If you still key. If you still key. If you still They didn't, of course.

They didn't, of course. They didn't, of course. Now we can Now we can Now we can clone this one clone this one clone this one repository. repository. I don't seem to be doing anything right. I don't seem to be doing anything right. updated in the code. OK. updated in the code. OK. updated in the code. OK. Again. Again. Is anyone for or against? ? Is everything okay?

? Is everything okay? ? Is everything okay? need to increase need to increase need to increase scale? Class. Therefore, scale? Class. Therefore, scale? Class. Therefore, let's start with this let's start with this let's start with this work. Of course, we work. Of course, we work. Of course, we we will we will we will use use use various tools, various tools, various tools, to ensure to ensure to ensure work. I work. I work. I I use UV as I use UV as I use UV as package manager for package manager for package manager for installation of all installation of all installation of all my Python dependencies. my Python dependencies. So, if you don't have one yet So, if you don't have one yet UV installed, UV installed, UV installed, use this use this use this team. I have it. team. I have it. team. I have it. already exists, so I just already exists, so I just already exists, so I just I will run "uv sync" and add I will run "uv sync" and add I will run "uv sync" and add multiple dependencies multiple dependencies multiple dependencies for development. This has for development. This has for development. This has create a virtual create a virtual create a virtual environment in your environment in your environment in your directories and directories and directories and install all install all install all required components required components required components for this seminar. for this seminar. The next step is for us to The next step is for us to let's create our .env file, let's create our .env file, let's create our .env file, an example of which we have an example of which we have an example of which we have already exists. You can already exists. You can already exists. You can just copy just copy just copy its. Everything you need its. Everything you need its. Everything you need need to do—this need to do—this need to do—this add API key, add API key, add API key, generated on generated on generated on on the Braintrust side, and on the Braintrust side, and on the Braintrust side, and also choose a model also choose a model also choose a model by default. It by default. It by default. It it is only necessary that it is only necessary that it is only necessary that in case you in case you in case you want to start a chat- want to start a chat- interface or, interface or, interface or, seems to be holding seems to be holding seems to be holding evaluation. But this evaluation. But this evaluation. But this two things that you two things that you two things that you need to create need to create need to create or change in the file or change in the file or change in the file .env that you just created .env that you just created .env that you just created copied to copied to copied to run this run this run this seminar. Here. There are others.

seminar. Here. There are others. seminar. Here. There are others. instructions in this instructions in this instructions in this .env file regarding what .env file regarding what .env file regarding what you can also you can also you can also change, for example, change, for example, change, for example, if you want if you want if you want to adhere to those to adhere to those to adhere to those best practices, best practices, best practices, which we talked about which we talked about which we talked about previously, regarding previously, regarding previously, regarding creation or creation or creation or using the best using the best using the best or a more powerful model or a more powerful model or a more powerful model for evaluation, you for evaluation, you for evaluation, you can you do this can you do this can you do this also. Just if you also. Just if you also. Just if you You don't know what's going on here. You don't know what's going on here. You don't know what's going on here. be, there is this file be, there is this file be, there is this file .env that we just created .env that we just created .env that we just created copied. Again copied. Again copied. Again okay, I'm just changing okay, I'm just changing okay, I'm just changing or add my API key or add my API key or add my API key BrainTrust, and then I add BrainTrust, and then I add BrainTrust, and then I add standard model. AND standard model. AND standard model. AND this standard model, this standard model, this standard model, of course, there will be of course, there will be of course, there will be accessible through accessible through accessible through providers that I providers that I providers that I set to set to set to on the BrainTrust side. So, on the BrainTrust side. So, on the BrainTrust side. So, if I chose OpenAI, I if I chose OpenAI, I if I chose OpenAI, I I will use I will use I will use GPT model, Anthropic Sonnet and GPT model, Anthropic Sonnet and GPT model, Anthropic Sonnet and and so on. So simple and so on. So simple and so on. So simple make sure that these make sure that these make sure that these two things coincide. two things coincide. Another thing that will Another thing that will very important for very important for very important for this seminar, is this seminar, is this seminar, is installing BrainTrust CLI installing BrainTrust CLI . This will allow you . This will allow you . This will allow you launch random launch random launch random SQL, execute SQL, execute SQL, execute evaluation, evaluation, evaluation, browse magazines browse magazines browse magazines and create sets and create sets and create sets data. Remember that one data. Remember that one data. Remember that one reverse mechanism reverse mechanism reverse mechanism the connection that I just made the connection that I just made the connection that I just made described, many of these described, many of these described, many of these things are part of it things are part of it , and we can , and we can , and we can integrate this into integrate this into integrate this into our agent to our agent to our agent to he started doing it he started doing it he started doing it independently, that you independently, that you independently, that you you will see at the end you will see at the end you will see at the end seminar. So, you seminar. So, you seminar. So, you See what I have here.

See what I have here. See what I have here. already installed already installed already installed BrainTrust CLI. Me too BrainTrust CLI. Me too BrainTrust CLI. Me too I will execute `bt setup skills` and I will execute `bt setup skills` and I will execute `bt setup skills` and I will use I will use I will use Codex. You can Codex. You can Codex. You can use Cloud use Cloud use Cloud Code, Open Code, Cursor, Qwen or Code, Open Code, Cursor, Qwen or Code, Open Code, Cursor, Qwen or Gemini. This is also Gemini. This is also Gemini. This is also supported, but supported, but supported, but the idea is that the idea is that the idea is that what will we give to our what will we give to our what will we give to our access agent access agent access agent skills so that he skills so that he skills so that he understood how understood how understood how create sets create sets create sets data, run data, run data, run evaluation, writing SQL evaluation, writing SQL evaluation, writing SQL etc. Yes. etc. Yes. Switch to what? Oh, yes. This is very beautiful. Oh, yes. This is very beautiful. idea. I think so. idea. I think so. idea. I think so. never this here never this here never this here did. Any ideas? did. Any ideas?

What is it? Control B? Let's do it. What is it? Control B? Let's do it. it. Sorry? Command it. Sorry? Command it. Sorry? Command Shift B? What to enter?

Shift B? What to enter? OK. I think you all OK. I think you all hoped to see hoped to see as I wander as I wander as I wander terminal, without having terminal, without having terminal, without having no idea what no idea what no idea what I do. So we have everything. I do. So we have everything. I do. So we have everything. good. This is better for good. This is better for good. This is better for everyone. Perfectly. Sorry everyone. Perfectly. Sorry everyone. Perfectly. Sorry for this. Okay, great. for this. Okay, great. Next we Next we initialize the CLI. I initialize the CLI. I initialize the CLI. I I will do something different, I will do something different, I will do something different, because I already because I already because I already initialized me too initialized me too initialized me too you need to go through OAuth in you need to go through OAuth in you need to go through OAuth in Brain Trust to get Brain Trust to get Brain Trust to get access to another access to another access to another organizations. So I organizations. So I organizations. So I I will do it. Oh. So, if I will do it. Oh. So, if I will do it. Oh. So, if you already have CLI Brain Trust and you already have CLI Brain Trust and you already have CLI Brain Trust and you created a new one you created a new one you created a new one organization, you organization, you organization, you go through this process and go through this process and go through this process and just choose the one just choose the one just choose the one organization that organization that organization that just created. IN just created. IN just created. IN in my case it was in my case it was in my case it was ABC DPG. Steeply. And then I ABC DPG. Steeply. And then I ABC DPG. Steeply. And then I I will run BT init and I will run BT init and I will run BT init and I will look for ABC. It's simple.

I will look for ABC. It's simple. I will look for ABC. It's simple. will create a config.json file will create a config.json file will create a config.json file in the directory, so that when in the directory, so that when in the directory, so that when BT system startup BT system startup BT system startup knew which one knew which one knew which one organizations and organizations and organizations and project to apply. project to apply. You can add this to You can add this to your repository. your repository. your repository. Now all your Now all your Now all your developers will have developers will have developers will have easier way easier way easier way to understand to which to understand to which to understand to which organizations and organizations and organizations and BT project has BT project has BT project has to apply for to apply for to apply for evaluation and inquiry evaluation and inquiry evaluation and inquiry data. Perfectly. data. Perfectly. data. Perfectly. Let's go now. Let's go now. Let's go now. let's move on to let's move on to let's move on to setting things up, setting things up, setting things up, which we will need which we will need which we will need for the workshop. I for the workshop. I for the workshop. I I'll just fill in the database. I'll just fill in the database. I'll just fill in the database. data. I will miss. data. I will miss. data. I will miss. smoke command, but smoke command, but smoke command, but I'll run make ready. If in I'll run make ready. If in I'll run make ready. If in you are not make or you are not make or you are not make or access to it, access to it, access to it, below are the ones below are the ones below are the ones commands that you commands that you commands that you you can do it. you can do it. Now it's simple. Now it's simple. checks if we are all checks if we are all checks if we are all right right right set up. Does the set up. Does the set up. Does the us CLI Braintrust? Does the us CLI Braintrust? Does the us CLI Braintrust? Does the us Braintrust API key? Is there us Braintrust API key? Is there us Braintrust API key? Is there we have a model for we have a model for we have a model for default? And therefore default? And therefore default? And therefore similar. I think now similar. I think now similar. I think now we can move we can move we can move further. To give you further. To give you further. To give you idea of ​​an example idea of ​​an example idea of ​​an example the use that we the use that we the use that we we are considering. This is very we are considering. This is very we are considering. This is very a simple chatbot, but a simple chatbot, but a simple chatbot, but it is important to me it is important to me it is important to me understand if everything understand if everything understand if everything connected correctly connected correctly . Do we see how the logs . Do we see how the logs . Do we see how the logs coming to Braintrust?

coming to Braintrust? coming to Braintrust? So, I just So, I just So, I just pressed the button in pressed the button in pressed the button in our chat. Now we our chat. Now we our chat. Now we we need to see the recording we need to see the recording we need to see the recording at Braintrust with a question at Braintrust with a question , which we just , which we just , which we just put. Here put. Here put. Here trace example, trace example, trace example, just created on just created on just created on based on our based on our based on our integration with OpenAI Agents integration with OpenAI Agents integration with OpenAI Agents SDK. Hmm, I think maybe, SDK. Hmm, I think maybe, SDK. Hmm, I think maybe, about this, uh, about this, uh, about this, uh, just to show just to show just to show What does it look like to you? What does it look like to you? What does it look like to you? some integration, uh, some integration, uh, some integration, uh, at Braintrust. This is obviously at Braintrust. This is obviously at Braintrust. This is obviously just one example. Ahem, just one example. Ahem, just one example. Ahem, but all we actually but all we actually but all we actually We do, this is it. It We do, this is it. It We do, this is it. It what we pull up from what we pull up from what we pull up from Braintrust SDK, and we Braintrust SDK, and we Braintrust SDK, and we we install we install we install trace processor trace processor trace processor per processor per processor per processor Braintrust tracing. Uh, Braintrust tracing. Uh, Braintrust tracing. Uh, there is also a simpler one there is also a simpler one there is also a simpler one way to do it. way to do it. way to do it. There is an automatic There is an automatic There is an automatic instrumentation, this instrumentation, this instrumentation, this changing just one, um, changing just one, um, changing just one, um, line. The reason why I line. The reason why I line. The reason why I did this, is to did this, is to did this, is to because me too because me too because me too I use our I use our I use our gateway under the hood. Ahem, gateway under the hood. Ahem, gateway under the hood. Ahem, maybe I'm a little bit of that maybe I'm a little bit of that maybe I'm a little bit of that hid. At Braintrust hid. At Braintrust hid. At Braintrust there is also a gateway. So, to there is also a gateway. So, to there is also a gateway. So, to all those all those all those AI providers, AI providers, AI providers, which you have set up in which you have set up in which you have set up in Braintrust, we are now Braintrust, we are now Braintrust, we are now we can get we can get we can get access, uh, from a single access, uh, from a single access, uh, from a single interface, uh, what are we interface, uh, what are we interface, uh, what are we here we do, but in here we do, but in here we do, but in we are set up we are set up we are set up tracing from our tracing from our tracing from our SDK directly for agents SDK directly for agents SDK directly for agents OpenAI. We are installing OpenAI. We are installing OpenAI. We are installing this processor this processor this processor tracing, uh, on tracing, uh, on tracing, uh, on trace processor trace processor trace processor Braintrust. And again, Braintrust. And again, Braintrust. And again, this means that all these this means that all these this means that all these routes that routes that routes that are generated, all these are generated, all these are generated, all these generated spans generated spans , now, uh, they're coming , now, uh, they're coming , now, uh, they're coming at Braintrust. Uh, maybe we

at Braintrust. Uh, maybe we at Braintrust. Uh, maybe we let's do it...Actually, let's do it...Actually, let's do it...Actually, no, we will not no, we will not no, we will not do it from here, do it from here, do it from here, because it's not light because it's not light because it's not light regime. Okay, so in regime. Okay, so in regime. Okay, so in I have one too, uh, I have one too, uh, I have one too, uh, the script is here, or I have it the script is here, or I have it the script is here, or I have it there's, uh, the code here for there's, uh, the code here for there's, uh, the code here for launch assessments ( launch assessments ( evals). And, of course, this is not evals). And, of course, this is not evals). And, of course, this is not there must be, uh, there must be, uh, there must be, uh, the main goal, uh, the main goal, uh, the main goal, uh, this seminar, this seminar, this seminar, truth? We are a little truth? We are a little truth? We are a little we are talking about we are talking about we are talking about observability. observability. observability. But, but I think, uh, c But, but I think, uh, c But, but I think, uh, c context context context observability observability observability you need to think you need to think you need to think about the flywheel. And, and about the flywheel. And, and about the flywheel. And, and part of this part of this part of this there is a flywheel there is a flywheel there is a flywheel evaluation. Um, things, evaluation. Um, things, evaluation. Um, things, which we can which we can which we can understand and deduce understand and deduce understand and deduce about what happened in about what happened in about what happened in production, have production, have production, have return to our return to our return to our evaluations. Uh, and that's why evaluations. Uh, and that's why evaluations. Uh, and that's why in the repository, which in in the repository, which in in the repository, which in there are you, there are also, uh, there are you, there are also, uh, there are you, there are also, uh, kind of fictitious kind of fictitious kind of fictitious dataset, right? IN dataset, right? IN dataset, right? IN there are several, uh, there are several, uh, there are several, uh, test cases, test cases, test cases, which we now want which we now want which we now want to drive through our to drive through our to drive through our evaluation. So, what evaluation. So, what evaluation. So, what we just did, we just did, we just did, again, again, again, used our used our used our command interface command interface command interface Braintrust line, created Braintrust line, created Braintrust line, created data set and data set and data set and actually downloaded actually downloaded actually downloaded a series of lines there. Ahem, a series of lines there. Ahem, a series of lines there. Ahem, now I will start now I will start now I will start evaluation. Again evaluation. Again evaluation. Again okay, this, uh, will pass okay, this, uh, will pass okay, this, uh, will pass through this dataset through this dataset . He, so to speak, . He, so to speak, . He, so to speak, will process each line will process each line will process each line in this dataset in this dataset in this dataset and will call our and will call our and will call our agent. And now we agent. And now we agent. And now we let's keep all this, yeah let's keep all this, yeah let's keep all this, yeah I would say, uh, all this I would say, uh, all this I would say, uh, all this telemetry and, uh, telemetry and, uh, telemetry and, uh, the results of this the results of this the results of this evaluation, uh, evaluation, uh, evaluation, uh, directly in directly in directly in Braintrust. But you too Braintrust. But you too Braintrust. But you too can you imagine what this is can you imagine what this is can you imagine what this is will be extremely will be extremely will be extremely informative when we informative when we informative when we let's run this one let's run this one let's run this one mechanism when mechanism when mechanism when let's begin to understand and let's begin to understand and let's begin to understand and to make changes, because to make changes, because to make changes, because usually you want usually you want usually you want compare it to something compare it to something . You want to understand, . You want to understand, . You want to understand, has it gotten better or not?

has it gotten better or not? has it gotten better or not? worse compared to worse compared to worse compared to last time, when last time, when last time, when you spent this you spent this you spent this assessment. So, we have… assessment. So, we have… Actually, I just Actually, I just Actually, I just I will go to Braintrust. I will go to Braintrust. I will go to Braintrust. So, now we have So, now we have So, now we have see on our see on our see on our page page page experiments experiments experiments assessment results, assessment results, assessment results, which we launched. You which we launched. You which we launched. You you see a series of ratings, you see a series of ratings, you see a series of ratings, which we have defined for which we have defined for which we have defined for this particular one this particular one this particular one use case use case . You can also . You can also . You can also see, and maybe I see, and maybe I see, and maybe I I'll move on to this. I'll move on to this. I'll move on to this. a little later, what do we have a little later, what do we have a little later, what do we have there are different assessments, right? there are different assessments, right? there are different assessments, right? Yes? As I already described Yes? As I already described Yes? As I already described earlier, there are estimates that earlier, there are estimates that earlier, there are estimates that to a greater extent to a greater extent to a greater extent are based on code. are based on code. are based on code. For example, calls For example, calls For example, calls necessary necessary necessary tools: "Hey, in tools: "Hey, in tools: "Hey, in I have this one. I have this one. I have this one. question. That's what I am question. That's what I am question. That's what I am "I'm waiting." And I also have "I'm waiting." And I also have "I'm waiting." And I also have a little more ratings a little more ratings a little more ratings based on an LLM judge. based on an LLM judge. based on an LLM judge. For example, For example, For example, resolving queries resolving queries resolving queries in support and in support and in support and quality of use quality of use quality of use tools. So this is tools. So this is tools. So this is a kind of set or a kind of set or a kind of set or a combination of grades that a combination of grades that a combination of grades that can provide can provide can provide correct signal correct signal correct signal to understand that to understand that to understand that what is happening with what is happening with what is happening with our agent. OK our agent. OK , let's move on , let's move on , let's move on to the heart of the matter to the heart of the matter . The idea is that . The idea is that . The idea is that what can we do now what can we do now what can we do now create a certain type create a certain type create a certain type signal within our signal within our signal within our logs. Of course, now in logs. Of course, now in logs. Of course, now in We have one log, and we We have one log, and we We have one log, and we need to generate need to generate need to generate data and enter it into data and enter it into data and enter it into Braintrust. But before that Braintrust. But before that Braintrust. But before that let's use let's use let's use these series of assessments and these series of assessments and these series of assessments and let's upload them to let's upload them to let's upload them to Braintrust, to Braintrust, to Braintrust, to to adjust to adjust to adjust automation for automation for automation for start-up at start-up at start-up at receiving new receiving new receiving new logs. So, again logs. So, again logs. So, again yes, I will use it yes, I will use it yes, I will use it make push scores command.

make push scores command. She will find a series She will find a series created by us created by us created by us ratings and download ratings and download ratings and download them at Braintrust. There is them at Braintrust. There is them at Braintrust. There is a way to identify them a way to identify them a way to identify them directly in directly in directly in interface interface interface user. You too user. You too user. You too you can, as you can see here, you can, as you can see here, you can, as you can see here, to determine your own to determine your own to determine your own evaluations in the code and evaluations in the code and evaluations in the code and download them to download them to download them to Braintrust. The idea is that Braintrust. The idea is that Braintrust. The idea is that you want to run you want to run you want to run them as part of the online them as part of the online automation, or, automation, or, automation, or, maybe we have maybe we have maybe we have a gaming environment where a gaming environment where a gaming environment where you can provide these you can provide these you can provide these evaluations to another evaluations to another evaluations to another to the user to to the user to to the user to he used he used he used them, keeping them, keeping them, keeping version control in version control in version control in your repository. your repository. your repository. Again, here it is. Again, here it is. Again, here it is. series of tools, series of tools, series of tools, what we just... what we just... sorry, series sorry, series sorry, series the estimates that we just the estimates that we just the estimates that we just uploaded to Braintrust. uploaded to Braintrust. uploaded to Braintrust. You should now see You should now see You should now see on the grades tab ( on the grades tab ( scores) various indicators, scores) various indicators, scores) various indicators, which we created. Therefore, which we created. Therefore, which we created. Therefore, I have, let's say, I have, let's say, I have, let's say, resolving requests in resolving requests in resolving requests in support, as an example support, as an example , right? This will be , right? This will be , right? This will be LLM-based evaluator. LLM-based evaluator. And here you are, again, And here you are, again, define the criteria, define the criteria, define the criteria, for which you want for which you want for which you want evaluate the interaction evaluate the interaction evaluate the interaction user with this user with this user with this agent. I want agent. I want agent. I want to note that there are different to note that there are different to note that there are different types of variables to which types of variables to which types of variables to which can be obtained can be obtained can be obtained access and transfer them access and transfer them access and transfer them in the LLM evaluator. This one " in the LLM evaluator. This one " thread—that's it thread—that's it thread—that's it what I described what I described what I described earlier. This is what you earlier. This is what you earlier. This is what you you will see if you will see if you will see if go to go to go to magazine page and magazine page and magazine page and click on the stream, click on the stream, click on the stream, that is, on this icon, — that is, on this icon, — that's just the whole story that's just the whole story that's just the whole story communication with communication with communication with agent, right? This is very agent, right? This is very agent, right? This is very useful, especially useful, especially useful, especially if you have if you have if you have multi-step multi-step multi-step conversation and you want conversation and you want conversation and you want to transmit the entire conversation

to transmit the entire conversation to transmit the entire conversation to LLM, you can to LLM, you can to LLM, you can use a variable use a variable use a variable called thread. There is also called thread. There is also called thread. There is also others that can be others that can be others that can be use, use, use, for example, for example, for example, message message message user or user or user or assistant, or simply assistant, or simply assistant, or simply incoming and outgoing incoming and outgoing incoming and outgoing specific data specific data specific data time interval (span), time interval (span), time interval (span), if you want if you want if you want create an estimate for create an estimate for create an estimate for policy analysis policy analysis policy analysis search, as an example. search, as an example. search, as an example. But there are several different But there are several different But there are several different ways to get started ways to get started ways to get started to think about the assessment to think about the assessment to think about the assessment the qualities of your agent the qualities of your agent the qualities of your agent in many different ways in many different ways in many different ways parameters. Now, parameters. Now, parameters. Now, when our grades when our grades when our grades uploaded to Braintrust, uploaded to Braintrust, uploaded to Braintrust, we need we need we need set them up set them up set them up launch during launch during launch during receipt receipt receipt tracing (traces) to tracing (traces) to tracing (traces) to Braintrust. From Braintrust's side, this Braintrust. From Braintrust's side, this Braintrust. From Braintrust's side, this implemented through implemented through implemented through automation. If automation. If automation. If you will look at you will look at you will look at magazine page, you magazine page, you magazine page, you you will see on the right you will see on the right you will see on the right above the point " above the point " automation". Exactly automation". Exactly automation". Exactly here you can create here you can create here you can create rules by which rules by which rules by which will be called will be called will be called your grades. If you your grades. If you your grades. If you look here at look here at look here at scope, I scope, I scope, I I think it's very I think it's very I think it's very important, especially important, especially important, especially when do you start when do you start when do you start to think about different things to think about different things to think about different things options options options using. You using. You using. You you might think: in you might think: in you might think: in me multi-step me multi-step me multi-step conversation, but my conversation, but my conversation, but my steps actually steps actually steps actually distributed between distributed between distributed between different tracings different tracings . I don't see every step, . I don't see every step, . I don't see every step, embedded in one embedded in one embedded in one tracing. Therefore, in tracing. Therefore, in tracing. Therefore, in in that case I can in that case I can in that case I can choose a scope choose a scope choose a scope groups and provide groups and provide groups and provide some ID, way some ID, way some ID, way understand how all this understand how all this understand how all this related. In mine related. In mine related. In mine in this case in this case in this case identifier identifier identifier conversations (conversation ID).

conversations (conversation ID). conversations (conversation ID). Another thing you Another thing you Another thing you you can do, is you can do, is you can do, is create an assessment that create an assessment that create an assessment that called for called for called for specific interval specific interval specific interval time (span). Again time (span). Again , you want to evaluate , you want to evaluate , you want to evaluate certain intermediate certain intermediate certain intermediate result or step and result or step and result or step and to aim for to aim for to aim for specific range. specific range. specific range. When you do this, When you do this, When you do this, pressing on pressing on pressing on range, it is for range, it is for range, it is for default will choose default will choose default will choose root element. You root element. You root element. You you can also add you can also add you can also add any filter, any filter, any filter, for example, for example, for example, recognize politics recognize politics recognize politics search, and specify search, and specify search, and specify something like this something like this something like this case. So, I want case. So, I want case. So, I want understand whether it is understand whether it is understand whether it is root element, root element, root element, or attribute name or attribute name or attribute name range is contained range is contained range is contained in search policy. in search policy. in search policy. There is also tracing, There is also tracing, There is also tracing, which one will we be which one will we be which one will we be use for use for use for this demonstration. It this demonstration. It this demonstration. It will allow us will allow us will allow us access to access to access to all different all different all different ranges within ranges within ranges within this trace, so that this trace, so that this trace, so that to get a certain to get a certain to get a certain indicator. So in this indicator. So in this indicator. So in this I will choose I will choose I will choose quality of communication, quality of communication, quality of communication, quality of solution quality of solution quality of solution support requests and support requests and support requests and quality of use quality of use quality of use tools. That's all. tools. That's all. tools. That's all. LLM-based appraisers. LLM-based appraisers. So if you follow So if you follow follow me, know: follow me, know: follow me, know: depending on frequency depending on frequency depending on frequency the sample you are here the sample you are here the sample you are here you will install you will install you will install will spend will spend will spend computational computational computational model resources when model resources when model resources when loading data into loading data into loading data into Braintrust through these Braintrust through these Braintrust through these appraisers. So, I appraisers. So, I appraisers. So, I just wanted to do this just wanted to do this just wanted to do this emphasize. Again emphasize. Again emphasize. Again yes, if you follow, yes, if you follow, yes, if you follow, maybe you want maybe you want maybe you want lower it a little lower it a little lower it a little value. I will install value. I will install value. I will install 100% so that you get 100% so that you get 100% so that you get as clear as possible as clear as possible as clear as possible signal or better signal or better signal or better idea of ​​how it is idea of ​​how it is idea of ​​how it is may look like. Therefore, may look like. Therefore, may look like. Therefore, what I just did—I what I just did—I what I just did—I said: "Hey, for whatever said: "Hey, for whatever -what new logs that -what new logs that -what new logs that received by Braintrust, I received by Braintrust, I received by Braintrust, I I want to run these

I want to run these I want to run these evaluate and check evaluate and check evaluate and check them based on them based on them based on tracing. OK. tracing. OK. tracing. OK. Let's go now. Let's go now. Let's go now. Let's move on to the topics. Let's move on to the topics. Let's move on to the topics. So, there is one side. So, there is one side. So, there is one side. our signal, which our signal, which our signal, which we can create. we can create. we can create. This is narrower This is narrower This is narrower way of understanding way of understanding way of understanding type of error, right? type of error, right? type of error, right? Yes? This is perhaps what Yes? This is perhaps what Yes? This is perhaps what that we are obviously that we are obviously that we are obviously we evaluate. We we evaluate. We we evaluate. We defined it as something that defined it as something that defined it as something that we need we need we need to understand, or not to understand, or not to understand, or not is happening is happening is happening regression when we regression when we regression when we we change something in we change something in we change something in our agent. Topics same our agent. Topics same our agent. Topics same more suitable for more suitable for more suitable for those "unknowns" those "unknowns" those "unknowns" unknowns. Those areas, unknowns. Those areas, unknowns. Those areas, which we are not testing yet which we are not testing yet which we are not testing yet or which we haven't talked about yet or which we haven't talked about yet or which we haven't talked about yet we know. You too we know. You too we know. You too notice right next to it notice right next to it notice right next to it with ours with ours with ours automations automations automations the option to enable themes. the option to enable themes. the option to enable themes. Actually, let me Actually, let me Actually, let me I have to go back to I have to go back to I have to go back to this. I think the first thing is, this. I think the first thing is, this. I think the first thing is, what are we going to do here— what are we going to do here— Let's turn on the themes. You Let's turn on the themes. You Let's turn on the themes. You you will see a screen that you will see a screen that you will see a screen that looks like looks like looks like Yes. And you will see these Yes. And you will see these Yes. And you will see these built-in functions. AND built-in functions. AND built-in functions. AND namely: task, namely: task, namely: task, tonality and tonality and tonality and problems. So, this is it, problems. So, this is it, problems. So, this is it, what I was talking about: what I was talking about: what I was talking about: what exactly are people or what exactly are people or what exactly are people or users do users do users do with my agent? Which with my agent? Which with my agent? Which their mood? Or their mood? Or their mood? Or Are they satisfied? Or Are they satisfied? Or Are they satisfied? Or Are they upset? AND Are they upset? AND Are they upset? AND also, what are the general also, what are the general also, what are the general they are problems they are problems they are problems feel during feel during feel during interactions with this interactions with this interactions with this agent? So I agent? So I agent? So I I'll turn it on. It I'll turn it on. It I'll turn it on. It applies to applies to applies to existing records.

existing records. existing records. Of course, there is not much here. Of course, there is not much here. Of course, there is not much here. data, so later we data, so later we data, so later we Let's see how they are here. Let's see how they are here. Let's see how they are here. will appear. That which will appear. That which will appear. That which again allows you again allows you again allows you apply your own apply your own apply your own way of understanding way of understanding way of understanding patterns in patterns in patterns in data, is data, is data, is the opportunity to create the opportunity to create the opportunity to create own aspect. You own aspect. You own aspect. You you will also notice that you will also notice that you will also notice that here you can determine here you can determine here you can determine own preprocessor. own preprocessor. own preprocessor. Today we just Today we just Today we just we will use we will use we will use standard standard standard streaming, but be aware streaming, but be aware that if you want that if you want that if you want add your own add your own add your own function in the Brain Trust for function in the Brain Trust for function in the Brain Trust for analysis of the record, you, analysis of the record, you, analysis of the record, you, of course you can do that of course you can do that of course you can do that make. So, if you make. So, if you make. So, if you look, in look, in look, in repositories are a file repositories are a file repositories are a file called "support workflow" called "support workflow" called "support workflow" issue facet". This is a file in issue facet". This is a file in issue facet". This is a file in markdown format. It markdown format. It markdown format. It a hint (prompt) that a hint (prompt) that a hint (prompt) that we we we we will use we will use we will use for our for our for our special aspect special aspect . So let's go back to . So let's go back to . So let's go back to Brain Trust, I'll take the name Brain Trust, I'll take the name Brain Trust, I'll take the name "support workflow issue". This will be "support workflow issue". This will be "support workflow issue". This will be our special our special our special aspect. And then I want aspect. And then I want aspect. And then I want copy here copy here copy here hint. Another thing, hint. Another thing, hint. Another thing, which you most likely which you most likely which you most likely you want to do, — you want to do, — this excludes certain this excludes certain this excludes certain results of aspects.

results of aspects. results of aspects. As you can see, I gave As you can see, I gave As you can see, I gave output LLM instruction " output LLM instruction " none", if followed none", if followed none", if followed the following criteria. the following criteria. the following criteria. I don't necessarily I don't necessarily I don't necessarily I want to see aspects, I want to see aspects, I want to see aspects, which are generated when which are generated when which are generated when they say, "Hey, here they say, "Hey, here they say, "Hey, here there is none there is none there is none problems". Me problems". Me problems". Me are actually interested are actually interested are actually interested only those things that go only those things that go only those things that go not so. Therefore, I also not so. Therefore, I also not so. Therefore, I also I'll take this one. I'll take this one. I'll take this one. regular expression regular expression regular expression from above. And I will insert from above. And I will insert from above. And I will insert him here. You too him here. You too him here. You too you can choose you can choose you can choose application to application to application to existing records. IN existing records. IN existing records. IN in this case it is not in this case it is not in this case it is not very important, very important, very important, because we have them because we have them because we have them little. So, regarding little. So, regarding little. So, regarding that's all you need that's all you need that's all you need need to be done to need to be done to need to be done to begin to understand whether begin to understand whether begin to understand whether Do you see certain Do you see certain Do you see certain patterns in patterns in patterns in your data. If you your data. If you your data. If you will return to will return to will return to our repository and our repository and our repository and look for another way. look for another way. I will not go into detail I will not go into detail to sort it out, but to sort it out, but to sort it out, but I just wanted one more time. I just wanted one more time. I just wanted one more time. emphasize emphasize emphasize possibility possibility possibility using using using interface interface interface command line command line command line BrainTrust for many BrainTrust for many BrainTrust for many interesting things. So interesting things. So interesting things. So workflow that workflow that workflow that I just passed by.

I just passed by. I just passed by. graphic graphic graphic interface, I also interface, I also interface, I also I can do it here at BT. I can do it here at BT. . I actually can. . I actually can. . I actually can. customize these themes customize these themes customize these themes to launch. I can to launch. I can to launch. I can set these up set these up set these up parameters parameters parameters automation. Therefore, automation. Therefore, automation. Therefore, here I have determined here I have determined here I have determined problems with problems with problems with the tone of the tasks the tone of the tasks the tone of the tasks and problems with the workplace and problems with the workplace and problems with the workplace support process. support process. support process. Another thing I Another thing I Another thing I did, you too did, you too did, you too you can see there you can see there you can see there different configurations. THERE ARE different configurations. THERE ARE different configurations. THERE ARE different ways different ways different ways settings settings settings parameters, right? parameters, right? parameters, right? So, you can So, you can So, you can to control how to control how to control how the conveyor is running, which the conveyor is running, which the conveyor is running, which is performed in is performed in is performed in real time, or real time, or real time, or what is he on what is he on what is he on is launched. I have on is launched. I have on is launched. I have on meaning that you may meaning that you may meaning that you may you want to add you want to add you want to add filter. You can filter. You can filter. You can to say: "I want to say: "I want to say: "I want launch your launch your launch your topic pipeline only on topic pipeline only on topic pipeline only on routes that routes that routes that correspond to these correspond to these correspond to these criteria". You too criteria". You too criteria". You too you can apply you can apply you can apply a certain type of frequency a certain type of frequency a certain type of frequency based on samples based on samples based on samples traffic indicators traffic indicators traffic indicators and possible costs and possible costs and possible costs etc. You too etc. You too etc. You too need to understand need to understand need to understand topics window For example, topics window For example, topics window For example, when I create a map when I create a map when I create a map what is the range of topics what is the range of topics what is the range of topics I must I must I must use on use on use on based on a specific window? based on a specific window? This is it, I think, This is it, I think, maybe it was yours maybe it was yours maybe it was yours question, sir question, sir question, sir green shirt, regarding green shirt, regarding green shirt, regarding how to set up how to set up how to set up this is for long term this is for long term this is for long term conversations? This is partly conversations? This is partly conversations? This is partly consists of grouping consists of grouping consists of grouping by area of ​​operation, so that by area of ​​operation, so that by area of ​​operation, so that to be able to to be able to to be able to involve other routes, involve other routes, involve other routes, which exist in other which exist in other which exist in other on the tracks. This is another one on the tracks. This is another one on the tracks. This is another one a parameter that can be a parameter that can be a parameter that can be set up where you are set up where you are set up where you are you can specify, you can specify, you can specify, for example, day. It for example, day. It for example, day. It maybe two days. You maybe two days. You maybe two days. You You all understand that You all understand that You all understand that your conversations your conversations your conversations agents from agents from agents from users can users can users can last a very long time.

last a very long time. So you can So you can set when set when set when is just starting is just starting is just starting topic conveyor. I will do it. topic conveyor. I will do it. topic conveyor. I will do it. here is something very short here is something very short . But then again, you . But then again, you . But then again, you you can customize you can customize you can customize this is based on your this is based on your this is based on your understanding interaction understanding interaction understanding interaction users from users from users from your data. your data. your data. Sorry, Sorry, Sorry, interactions interactions interactions users with your users with your users with your agent. Again, agent. Again, agent. Again, here is the same type here is the same type here is the same type the reach that I the reach that I the reach that I allocated for evaluations. allocated for evaluations. allocated for evaluations. You can run You can run You can run this is for a separate track. this is for a separate track. this is for a separate track. Or, if you have Or, if you have Or, if you have multi-way conversation multi-way conversation , which exists in several , which exists in several , which exists in several on the tracks, you on the tracks, you on the tracks, you use this use this use this group, and then group, and then group, and then joint joint joint identifier. IN identifier. IN identifier. IN in my case I have in my case I have in my case I have identifier identifier identifier conversation, which conversation, which conversation, which is written as is written as is written as metadata. So I metadata. So I metadata. So I I'll click "save". I'll click "save". So, now we have So, now we have customized themes. We customized themes. We customized themes. We turned on the standard turned on the standard turned on the standard functions, created functions, created functions, created own special own special own special edge, as well as edge, as well as edge, as well as set up set up set up parameters parameters parameters automation for automation for automation for start-up at start-up at start-up at data entry in data entry in data entry in BrainTrust. Besides, BrainTrust. Besides, BrainTrust. Besides, now we can now we can now we can see how it is see how it is see how it is looks like in practice.

looks like in practice. looks like in practice. There is a team here, There is a team here, There is a team here, with which you can with which you can with which you can launch...I have launch...I have launch...I have trace data that trace data that trace data that stored in S3- stored in S3- buckets. We are going buckets. We are going buckets. We are going them simply them simply them simply import. Again import. Again import. Again yes, under the hood here yes, under the hood here yes, under the hood here you will see BT sync. Therefore you will see BT sync. Therefore , we can take these , we can take these , we can take these data and download it data and download it data and download it at Braintrust. Regarding your at Braintrust. Regarding your at Braintrust. Regarding your question about LangSmith, question about LangSmith, question about LangSmith, which you put before which you put before —I helped many people —I helped many people —I helped many people customers to import customers to import customers to import tracing from others tracing from others tracing from others services, because they services, because they services, because they wanted to keep these wanted to keep these wanted to keep these data. But I just data. But I just data. But I just I want to emphasize that I want to emphasize that I want to emphasize that this is a great way this is a great way this is a great way do it on a large scale. do it on a large scale. You see, we just You see, we just downloaded 968 downloaded 968 downloaded 968 tracings. Or 1052 tracings. Or 1052 tracings. Or 1052 Sorry, this is Sorry, this is Sorry, this is speed. 5260 spans. speed. 5260 spans. speed. 5260 spans. Pretty fast Pretty fast Pretty fast way to download way to download way to download data in Braintrust. But data in Braintrust. But data in Braintrust. But again, the idea in again, the idea in again, the idea in in order to start in order to start in order to start generate signals and generate signals and generate signals and understand where we are understand where we are understand where we are we can improve we can improve we can improve our agent. You our agent. You our agent. You you will start seeing here you will start seeing here you will start seeing here a few examples a few examples a few examples tracings. You too tracings. You too tracings. You too you will see what is here you will see what is here you will see what is here actually actually actually two are displayed two are displayed two are displayed things. Every step things. Every step things. Every step actually actually actually presented as presented as presented as separate tracing. It separate tracing. It separate tracing. It latest question latest question latest question user, but here user, but here user, but here the answer to this the answer to this the answer to this question. Also question. Also question. Also it can be seen that it can be seen that it can be seen that here are 11 previous ones here are 11 previous ones here are 11 previous ones messages. So I messages. So I messages. So I I can make it so that I can make it so that I can make it so that tighten your tighten your tighten your identifier identifier identifier conversations. Now I can

conversations. Now I can conversations. Now I can to pull others up to pull others up to pull others up tracings that have tracings that have tracings that have same metadata. same metadata. same metadata. OpenAI agents give you OpenAI agents give you OpenAI agents give you a kind of "code" a kind of "code" a kind of "code" access", because access", because access", because they automatically they automatically they automatically everyone adds everyone adds everyone adds previous previous previous message. So we message. So we message. So we not necessarily not necessarily not necessarily go through this step. go through this step. But I wanted to highlight But I wanted to highlight another Braintrust feature another Braintrust feature , which allows , which allows , which allows group traces group traces group traces by arbitrary by arbitrary by arbitrary metadata, metadata, metadata, defined for them. defined for them. You can combine You can combine all relevant all relevant all relevant tree tracing tree tracing tree tracing here. Very quickly, right? here. Very quickly, right? here. Very quickly, right? wrong? In my opinion, wrong? In my opinion, wrong? In my opinion, viewing a branch is viewing a branch is viewing a branch is great way great way great way understand the interaction understand the interaction understand the interaction your agent. Also your agent. Also your agent. Also there is a time review there is a time review there is a time review scale, which allows scale, which allows scale, which allows understand where understand where understand where narrow appear narrow appear narrow appear places, or even places, or even places, or even to see something interesting to see something interesting to see something interesting information about information about information about LLM call caching LLM call caching . I think that this . I think that this . I think that this information can be information can be information can be useful for useful for useful for many types many types many types work processes. Yet work processes. Yet work processes. Yet one moment at which one moment at which one moment at which I want to emphasize, and I want to emphasize, and I want to emphasize, and I haven't talked about this yet. I haven't talked about this yet. , this is a small blue , this is a small blue , this is a small blue icon below icon below icon below on the right—our agent on the right—our agent on the right—our agent Loop. Everyone has an agent, Loop. Everyone has an agent, Loop. Everyone has an agent, truth? So, you truth? So, you truth? So, you actually you can actually you can actually you can use Loop use Loop use Loop for many different for many different for many different tasks on the platform.

tasks on the platform. tasks on the platform. You can You can You can use Loop, use Loop, use Loop, to create your own to create your own to create your own interface on top of these interface on top of these interface on top of these trace data. AND trace data. AND trace data. AND this, in my opinion, this, in my opinion, this, in my opinion, a great way, about a great way, about a great way, about which I mentioned in which I mentioned in which I mentioned in in the context of another in the context of another in the context of another evaluation method: evaluation method: evaluation method: to allow people to allow people to allow people check check check work results work results work results agent or themselves agent or themselves agent or themselves judges-LLM and judges-LLM and judges-LLM and provide provide provide calibration. This is what calibration. This is what calibration. This is what what can you do what can you do what can you do use a hint use a hint use a hint in the Loop, and he in the Loop, and he in the Loop, and he will automatically generate will automatically generate will automatically generate interface for these interface for these interface for these trace data. So trace data. So trace data. So this is just another one this is just another one this is just another one a way to attract another a way to attract another a way to attract another person for person for person for improving this improving this improving this mechanism. You too mechanism. You too mechanism. You too you will see that we you will see that we you will see that we we begin to notice we begin to notice we begin to notice launch of a certain launch of a certain launch of a certain automation. We have automation. We have automation. We have there is my automation there is my automation there is my automation evaluation: quality evaluation: quality evaluation: quality communion, communion, communion, resolving requests resolving requests resolving requests support, etc. You support, etc. You support, etc. You should also should also should also to see that to see that to see that begin begin begin generate certain generate certain generate certain topics. So, in this topics. So, in this topics. So, in this case, right? We have case, right? We have case, right? We have user who wants user who wants user who wants check status check status check status refund for refund for refund for goods, etc. We have goods, etc. We have goods, etc. We have negative feedback. negative feedback. negative feedback. This user This user This user expressed expressed expressed disappointment and disappointment and disappointment and dissatisfaction with the situation dissatisfaction with the situation dissatisfaction with the situation unfinished unfinished unfinished transactions. Here is mine. transactions. Here is mine. transactions. Here is mine. workflow workflow workflow support, my support, my support, my own aspect.

own aspect. own aspect. An error occurred under An error occurred under An error occurred under time of receipt of the record time of receipt of the record time of receipt of the record about the order. No about the order. No about the order. No search for specific search for specific search for specific identifier identifier identifier order, no wider order, no wider order, no wider customer search is not customer search is not customer search is not made it possible to find made it possible to find made it possible to find order. I just order. I just order. I just wanted to show two wanted to show two wanted to show two different things, right? We different things, right? We different things, right? We downloaded downloaded downloaded tracing in Braintrust and, tracing in Braintrust and, tracing in Braintrust and, based on based on based on configured by us configured by us configured by us automation and automation and automation and topics, we start topics, we start topics, we start to receive this to receive this to receive this signal in mode signal in mode signal in mode real time. Yes? real time. Yes? real time. Yes? So, that complicated one So, that complicated one So, that complicated one a pipeline that generates a pipeline that generates a pipeline that generates patterns in your data patterns in your data , appears immediately. , appears immediately. , appears immediately. Yes? He appears. Yes? He appears. Yes? He appears. depending on how depending on how depending on how you set this up you set this up you set this up conveyor, of course. But conveyor, of course. But conveyor, of course. But this is the signal that we this is the signal that we this is the signal that we we will be able we will be able we will be able use. use. While it's starting While it's starting to fill, I to fill, I to fill, I I will move on to another one. I will move on to another one. I will move on to another one. project where I already project where I already project where I already launched. I used to launched. I used to launched. I used to showed you showed you showed you illustrations of how illustrations of how illustrations of how this may look like this may look like this may look like Braintrust. So, up here Braintrust. So, up here Braintrust. So, up here pages you see pages you see pages you see a snapshot of your topics in a snapshot of your topics in a snapshot of your topics in time and frequency, with time and frequency, with time and frequency, with which tracing, in which tracing, in which tracing, in which we recognized which we recognized which we recognized certain patterns, certain patterns, certain patterns, are grouped together. are grouped together.

So, in the first So, in the first I will expand on the example. I will expand on the example. I will expand on the example. it. I can understand. it. I can understand. it. I can understand. that users that users that users are trying to do with are trying to do with are trying to do with my agent. Yes? I my agent. Yes? I my agent. Yes? I I see above that I see above that I see above that resolution of issues resolution of issues resolution of issues regarding damage under regarding damage under regarding damage under transportation time, transportation time, transportation time, seems to be the most common seems to be the most common seems to be the most common request with which request with which request with which users users users are addressing my are addressing my are addressing my agent. And this, I think, agent. And this, I think, agent. And this, I think, takes us back to takes us back to takes us back to what I described earlier what I described earlier what I described earlier in slides: I can in slides: I can in slides: I can use this use this use this information to information to information to understand where mine is understand where mine is understand where mine is the agent can the agent can the agent can to develop further, or to develop further, or to develop further, or What features do I not yet have? What features do I not yet have? What features do I not yet have? implemented, and implemented, and implemented, and users already users already users already they are trying they are trying they are trying use. All of this use. All of this use. All of this will be together will be together will be together used for used for used for to fuel this to fuel this to fuel this development mechanism. development mechanism. A little lower I have A little lower I have mood theme. My mood theme. My mood theme. My there were no problems there were no problems there were no problems generated. Reason, generated. Reason, generated. Reason, why not why not why not generated, generated, generated, is that we is that we is that we looking for at least 100 looking for at least 100 looking for at least 100 matches or 100 matches or 100 matches or 100 aspects created in aspects created in aspects created in within this conveyor. within this conveyor. This type of agent that I This type of agent that I created, and created, and created, and standard request, standard request, standard request, which we have for which we have for which we have for problems, not really problems, not really problems, not really agree, so we agree, so we agree, so we we do not create we do not create we do not create aspects, but I aspects, but I aspects, but I I think that even I think that even I think that even good. We don't want to.

good. We don't want to. good. We don't want to. generate bad generate bad generate bad data. We don't want data. We don't want data. We don't want generate a signal, with generate a signal, with generate a signal, with which is impossible which is impossible which is impossible nothing to do. Ago nothing to do. Ago nothing to do. Ago I created my own I created my own I created my own problems with problems with problems with support of the worker support of the worker support of the worker process. And in this process. And in this process. And in this case I start case I start case I start understand: oh, it arose understand: oh, it arose understand: oh, it arose search failure search failure search failure recording. Something recording. Something recording. Something what is happening to me what is happening to me what is happening to me agent, what we don't agent, what we don't agent, what we don't discovered during discovered during discovered during evaluation, but we evaluation, but we evaluation, but we we understand that during we understand that during we understand that during agent's work agent's work agent's work an error occurs an error occurs an error occurs search. Yes? This is it search. Yes? This is it search. Yes? This is it information that, if information that, if information that, if to put it together... to put it together... along with context along with context along with context our online reviews, our online reviews, our online reviews, allows me to use allows me to use allows me to use measures and certain measures and certain measures and certain way to improve my way to improve my way to improve my agent. Have a question? agent. Have a question? Perfectly. OK. I want Perfectly. OK. I want show you the working one show you the working one show you the working one a process that, as a process that, as a process that, as me, I definitely me, I definitely me, I definitely used when used when used when just started in just started in just started in BrainTrust, and I think that BrainTrust, and I think that BrainTrust, and I think that many people also do this many people also do this many people also do this did, right? We did, right? We did, right? We generated this generated this generated this signal. This is great.

signal. This is great. signal. This is great. Now I need Now I need Now I need do something about it. AND do something about it. AND do something about it. AND so I think that, so I think that, so I think that, of course, in 在 of course, in 在 of course, in 在 The BrainTrust interface is The BrainTrust interface is The BrainTrust interface is many different many different many different tools that I tools that I tools that I I can use it to I can use it to I can use it to to begin to understand or to begin to understand or to begin to understand or filter this noise. filter this noise. filter this noise. Hmm, maybe I'll want to. Hmm, maybe I'll want to. Hmm, maybe I'll want to. to review, you know, to review, you know, to review, you know, various classifiers various classifiers various classifiers or the labels that were or the labels that were or the labels that were applied to these applied to these applied to these tracing, right? Which tracing, right? Which tracing, right? Which correspond to those correspond to those correspond to those topics you saw on topics you saw on topics you saw on another illustration. It another illustration. It another illustration. It I can use I can use I can use isn't that right? That isn't that right? That isn't that right? That filter it out. Ahem filter it out. Ahem I'll probably increase this. I'll probably increase this. . No. Um, theoretically I . No. Um, theoretically I . No. Um, theoretically I I can use this. I can use this. I can use this. so that somehow so that somehow so that somehow filter data, filter data, filter data, isn't that right? Based on isn't that right? Based on isn't that right? Based on reviews that we reviews that we reviews that we we get, and we get, and we get, and understand what I can do with understand what I can do with understand what I can do with to do this, right? Therefore to do this, right? Therefore , I filtered , I filtered , I filtered your set your set your set results. I results. I results. I I delve into I delve into I delve into tracing and, tracing and, tracing and, analyzing it and analyzing it and analyzing it and covered by it covered by it covered by it segments, I understand that segments, I understand that segments, I understand that It went wrong. Perhaps, It went wrong. Perhaps, It went wrong. Perhaps, I will use it again. I will use it again. I will use it again. your online ratings, your online ratings, your online ratings, so that even more so that even more so that even more filter this one filter this one filter this one set of results.

set of results. set of results. But I think that But I think that But I think that workflow, through workflow, through workflow, through which you can pass which you can pass , consists of , consists of , consists of using this using this using this interface to interface to interface to begin to understand everything begin to understand everything begin to understand everything various things, for example various things, for example , what exactly is going wrong. I , what exactly is going wrong. I , what exactly is going wrong. I I will take my assignment. I will take my assignment. I will take my assignment. X, I'll combine it with this X, I'll combine it with this X, I'll combine it with this I'll take a look and see, I'll take a look and see, I'll take a look and see, What conclusions can I draw? What conclusions can I draw? What conclusions can I draw? make. Ahem, make. Ahem, make. Ahem, signal generation signal generation —this is, of course, very —this is, of course, very —this is, of course, very good. We now have good. We now have good. We now have a lot of data, thanks a lot of data, thanks a lot of data, thanks how can we how can we how can we accept accept accept reasoned decisions reasoned decisions . But I think you all, . But I think you all, . But I think you all, You probably see it too. You probably see it too. You probably see it too. problem of such problem of such problem of such approach. It's like I'm acting. approach. It's like I'm acting. approach. It's like I'm acting. very reactive. very reactive. very reactive. Responsibility Responsibility Responsibility It falls on me like a It falls on me like a It falls on me like a Brain Trust user: Brain Trust user: Brain Trust user: need it yourself need it yourself need it yourself move, move, move, press buttons, press buttons, press buttons, filter, and then filter, and then filter, and then analyze these analyze these analyze these tracing. It has tracing. It has tracing. It has be the best way. be the best way. be the best way. Hmm, maybe I'll want to. Hmm, maybe I'll want to. Hmm, maybe I'll want to. use Loop. Loop— use Loop. Loop— this, again, this, again, this, again, agent built into agent built into agent built into platform that can platform that can platform that can to perform various to perform various to perform various task. Um, in this task. Um, in this task. Um, in this I asked for it. I asked for it. I asked for it. him: help me him: help me him: help me understand what is worth understand what is worth understand what is worth improve in my improve in my improve in my support agents on support agents on support agents on based on the latest based on the latest based on the latest logs. So, one of logs. So, one of logs. So, one of things you will start things you will start things you will start to notice and which, in my opinion to notice and which, in my opinion to notice and which, in my opinion opinion is very interesting opinion is very interesting opinion is very interesting from the flywheel's point of view from the flywheel's point of view —this is how we can —this is how we can —this is how we can attract relevant attract relevant attract relevant data for its operation.

data for its operation. data for its operation. Um, one more thing, which, Um, one more thing, which, Um, one more thing, which, in my opinion, again in my opinion, again in my opinion, again really interesting in Brain really interesting in Brain really interesting in Brain Trust and, in particular, in Brain Trust and, in particular, in Brain Trust and, in particular, in Brain Store, backend, what is it Store, backend, what is it Store, backend, what is it provides, is that, provides, is that, provides, is that, that you can write SQL that you can write SQL -requests -requests -requests directly to directly to directly to him. All this data him. All this data him. All this data tracing, all these tracing, all these tracing, all these segments can be segments can be segments can be to receive by writing to receive by writing to receive by writing SQL query. Hmm, maybe SQL query. Hmm, maybe SQL query. Hmm, maybe you won't want to you won't want to you won't want to write this SQL, but your write this SQL, but your write this SQL, but your coding agent coding agent coding agent it can be very easy it can be very easy it can be very easy make. So what you make. So what you make. So what you You see here—this is Loop. Yet You see here—this is Loop. Yet You see here—this is Loop. Yet once, Loop is our agent once, Loop is our agent once, Loop is our agent in the platform. I in the platform. I in the platform. I gave him gave him gave him question, and Loop started question, and Loop started question, and Loop started execute a bunch of SQL- execute a bunch of SQL- requests. He pulls out requests. He pulls out requests. He pulls out context that context that context that responds to responds to responds to the question I asked him the question I asked him the question I asked him put. What are you, put. What are you, put. What are you, you probably won't be able to you probably won't be able to you probably won't be able to to do—again to do—again , on a system scale , on a system scale —is to pull out each —is to pull out each —is to pull out each a trace in search a trace in search a trace in search signal using signal using signal using your agents, your your agents, your your agents, your writing agents writing agents writing agents code. You need, code. You need, code. You need, so that this signal is so that this signal is so that this signal is in a certain way in a certain way in a certain way tied to yours tied to yours tied to yours traces, and so that this traces, and so that this traces, and so that this the signal was real. the signal was real. the signal was real. And then you need And then you need And then you need very flexible way very flexible way very flexible way start making requests start making requests start making requests to all this data, so that to all this data, so that to all this data, so that extract only extract only extract only the relevant part the relevant part the relevant part information. Uh, information. Uh, information. Uh, the last thing you did here the last thing you did here the last thing you did here you don't see, but you don't see, but you don't see, but you will see a little you will see a little you will see a little later: as soon as I later: as soon as I later: as soon as I I find these interesting.

I find these interesting. I find these interesting. examples based on examples based on examples based on generated generated generated signal, I want signal, I want signal, I want go to this. I go to this. I go to this. I I want to explore this. I want to explore this. I want to explore this. trace. I want to click. trace. I want to click. trace. I want to click. for this range. I for this range. I for this range. I I want to look at I want to look at I want to look at input data. I want input data. I want input data. I want look at look at look at source data. source data. source data. That's right, this is the one. That's right, this is the one. That's right, this is the one. analysis that I probably analysis that I probably , I would do it, but I can't , I would do it, but I can't , I would do it, but I can't to do in large to do in large to do in large scale. But if I scale. But if I scale. But if I I can translate. I can translate. I can translate. part of this on Loop, part of this on Loop, part of this on Loop, to find out what to find out what to find out what me to change, me to change, me to change, correct or add, correct or add, correct or add, or translate it to or translate it to or translate it to your agent with your agent with your agent with writing code, then I writing code, then I writing code, then I I think that's it. I think that's it. I think that's it. effectively increases effectively increases effectively increases the quality of the agent's work. the quality of the agent's work. Uh, so I'll skip it. Uh, so I'll skip it. something, but the fact that I'm here something, but the fact that I'm here something, but the fact that I'm here described, —this described, —this described, —this use of themes and use of themes and use of themes and grades for understanding grades for understanding grades for understanding and factual analysis and factual analysis and factual analysis this data based on this data based on this data based on signal. Uh, what are you? signal. Uh, what are you? signal. Uh, what are you? you can also do you can also do you can also do now, use this now, use this now, use this this filtered this filtered this filtered data set. You data set. You data set. You found interesting found interesting found interesting example and you want example and you want example and you want add it to the set add it to the set add it to the set data, right? This is it. data, right? This is it. which allows us which allows us which allows us now improve now improve now improve agent's work. I want agent's work. I want agent's work. I want return this return this return this example to mine example to mine example to mine offline assessments. Yes, I am.

offline assessments. Yes, I am. offline assessments. Yes, I am. I can do it here, in I can do it here, in I can do it here, in interface. I interface. I interface. I going to add this to going to add this to going to add this to your data set. your data set. your data set. Now I have one. Now I have one. Now I have one. extreme case, extreme case, extreme case, thing that I thing that I thing that I need to improve, need to improve, need to improve, or a new feature that or a new feature that or a new feature that must be added, must be added, must be added, based on the fact that based on the fact that based on the fact that I saw during I saw during I saw during interactions interactions interactions user with mine user with mine user with mine agent. I can agent. I can agent. I can add this back to add this back to add this back to your data set. your data set. your data set. But, again, But, again, But, again, it is important that it is important that it is important that add to the set add to the set add to the set the right things the right things the right things based on the signal, based on the signal, based on the signal, created in our created in our created in our logs. And then, of course logs. And then, of course , you would have left, but you didn't , you would have left, but you didn't , you would have left, but you didn't Do you want to see me? Do you want to see me? Do you want to see me? I code, and try I code, and try I code, and try understand, can I understand, can I understand, can I improve the agent. improve the agent. improve the agent. But as soon as I do it But as soon as I do it But as soon as I do it I will do it again. I will do it again. I will do it again. I will start the evaluation. I I will start the evaluation. I I will start the evaluation. I I'll compare this to how I'll compare this to how I'll compare this to how I was launching I was launching I was launching evaluation last time. evaluation last time. evaluation last time. Is there any improvement? Is there Is there any improvement? Is there Is there any improvement? Is there regression? Did I do it? regression? Did I do it? regression? Did I do it? better by adding this better by adding this better by adding this new example? Or new example? Or new example? Or I really improved. I really improved. I really improved. results that results that results that received in received in received in online mode? online mode? online mode? One more thing you, One more thing you, One more thing you, maybe you want maybe you want maybe you want to make—is to create to make—is to create to make—is to create assessment when you already assessment when you already assessment when you already discovered these new discovered these new discovered these new edge cases. Yes? edge cases. Yes? edge cases. Yes? I have a search error. I have a search error. I have a search error. recording. It seems that this recording. It seems that this recording. It seems that this there was a problem with which there was a problem with which there was a problem with which I encountered I encountered I encountered most often. Apparently, most often. Apparently, most often. Apparently, I should create I should create I should create indicator that indicator that indicator that will allow me will allow me will allow me to fix it before to fix it before to fix it before How do I release mine?

How do I release mine? How do I release mine? agent in production. agent in production. agent in production. So there are all these steps, So there are all these steps, So there are all these steps, isn't that right? These are things that isn't that right? These are things that isn't that right? These are things that you probably you probably you probably you want to do so you want to do so you want to do so create this flywheel. create this flywheel. create this flywheel. And what, in my opinion, And what, in my opinion, And what, in my opinion, it's getting real here it's getting real here it's getting real here powerful, it is powerful, it is powerful, it is ability to launch ability to launch ability to launch all this is automatic. all this is automatic. all this is automatic. When can I When can I When can I connect skill connect skill connect skill or some type or some type or some type workflow, workflow, workflow, using all using all using all described by me described by me described by me primitives for primitives for primitives for automation of everything automation of everything automation of everything this. So if you this. So if you this. So if you look here, at look here, at look here, at our markup file for our markup file for our markup file for workshop, there is workshop, there is workshop, there is repository on GitHub with repository on GitHub with repository on GitHub with skills that we skills that we skills that we we have. Currently there is we have. Currently there is we have. Currently there is just one skill. just one skill. just one skill. The idea is that The idea is that The idea is that to give your to give your to give your coding agents coding agents coding agents ability to launch ability to launch ability to launch this automated this automated this automated flywheel. Yes? We flywheel. Yes? We flywheel. Yes? We we want to go through we want to go through we want to go through the same process. I the same process. I the same process. I I want Codex, Cloud Coder I want Codex, Cloud Coder I want Codex, Cloud Coder or anyone wrote or anyone wrote or anyone wrote SQL. I want him to SQL. I want him to SQL. I want him to viewed viewed viewed tracing. I want, tracing. I want, tracing. I want, for him to review for him to review for him to review sleep. I want him to sleep. I want him to sleep. I want him to created new ones created new ones created new ones evaluations, launched evaluations, launched evaluations, launched evaluation and evaluation and evaluation and changed something in me changed something in me changed something in me agents, and I want agents, and I want agents, and I want to do this, you know, to do this, you know, to do this, you know, watching watching watching with your screen. Yes, with your screen. Yes, with your screen. Yes, forward. Yes, so if forward. Yes, so if forward. Yes, so if look at mine look at mine If you look at If you look at my assessment, which I my assessment, which I my assessment, which I created So, here it is created So, here it is created So, here it is about like this about like this about like this looks like an assessment

looks like an assessment looks like an assessment at Braintrust. Um, your set at Braintrust. Um, your set at Braintrust. Um, your set there is actually data here there is actually data here there is actually data here part of the teams that part of the teams that part of the teams that we launched on we launched on we launched on beginning, so that beginning, so that beginning, so that download our download our download our sample data, such as sample data, such as sample data, such as our test series our test series our test series cases, at Braintrust. But cases, at Braintrust. But cases, at Braintrust. But you can also you can also you can also imagine that on the very imagine that on the very imagine that on the very This is my beginning. This is my beginning. This is my beginning. limited view of limited view of limited view of the world in which it works the world in which it works the world in which it works my agent, you know? my agent, you know? my agent, you know? These are test cases, These are test cases, These are test cases, which I want to give him, which I want to give him, which I want to give him, to understand whether to understand whether to understand whether Should I let it out? Should I let it out? Should I let it out? this thing in production. But this thing in production. But this thing in production. But the point of me being here the point of me being here the point of me being here I describe, consists of I describe, consists of I describe, consists of because the most because the most because the most quality improvement quality improvement quality improvement happens then, happens then, happens then, when you can when you can when you can select traces, select traces, select traces, examples from examples from examples from productive productive productive environment and environment and environment and return them to their return them to their return them to their data set. So, this is data set. So, this is data set. So, this is this is very this is very this is very simplified working simplified working simplified working the process I'm talking about the process I'm talking about the process I'm talking about told. This entry told. This entry , this line, this trace, , this line, this trace, , this line, this trace, perhaps indicates perhaps indicates perhaps indicates some type of failure. And I some type of failure. And I some type of failure. And I I want it back. I want it back. I want it back. to the existing or to the existing or to the existing or new data set, new data set, new data set, for which I want for which I want for which I want run the assessment.

run the assessment. Right. So, I Right. So, I I think that's the climax. I think that's the climax. I think that's the climax. of all this—this is what of all this—this is what of all this—this is what how can we how can we how can we use all use all use all those primitives that I those primitives that I those primitives that I described earlier. I think described earlier. I think described earlier. I think conveyor belt of topics that conveyor belt of topics that conveyor belt of topics that running in the background running in the background running in the background mode, is the main mode, is the main mode, is the main star of the show. But star of the show. But star of the show. But importance importance importance replenishment of the set replenishment of the set replenishment of the set data is that, data is that, data is that, What are your grades? What are your grades? What are your grades? must develop must develop must develop together with your agent together with your agent . And when I think about . And when I think about . And when I think about evaluations, I think about evaluations, I think about evaluations, I think about data set. I think data set. I think data set. I think about your agent, about your agent, about your agent, True? That is, this True? That is, this True? That is, this function that calls function that calls function that calls my agent. And then I my agent. And then I my agent. And then I I'm thinking about a series of points. I'm thinking about a series of points. that allow me that allow me that allow me to understand whether it is good or not to understand whether it is good or not to understand whether it is good or not These are going badly. These are going badly. These are going badly. test cases for test cases for test cases for my agent. But when my agent. But when my agent. But when I am creating my own set. I am creating my own set. I am creating my own set. estimates, this is a way estimates, this is a way estimates, this is a way set basic set basic set basic level. He, of course, level. He, of course, level. He, of course, not completely not completely not completely reflects everything that reflects everything that reflects everything that users can users can users can to do with my agent to do with my agent . Therefore, adding . Therefore, adding . Therefore, adding new entries to new entries to new entries to this dataset this dataset this dataset allows me over time allows me over time allows me over time record different types record different types record different types failures and make changes failures and make changes failures and make changes to an agent to...We to an agent to...We to an agent to...We no longer encountered no longer encountered no longer encountered these problems in these problems in these problems in future. But future. But future. But it's more about it's more about it's more about evolution of your evolution of your evolution of your agent and in order to agent and in order to agent and in order to everything that accompanies everything that accompanies everything that accompanies your grades, your grades, your grades, developed together with

developed together with developed together with him. So, you can him. So, you can him. So, you can not only to record not only to record not only to record new things, but also, new things, but also, new things, but also, changing the code, not changing the code, not changing the code, not to make work worse to make work worse to make work worse agent compared to agent compared to agent compared to previous previous previous interactions that you interactions that you interactions that you selected for this selected for this selected for this data set. THERE ARE data set. THERE ARE data set. THERE ARE several different several different several different ways to install ways to install ways to install it. I will choose simply it. I will choose simply it. I will choose simply first option. If first option. If first option. If you have GitHub installed you have GitHub installed you have GitHub installed CLI, you can CLI, you can CLI, you can to use it. to use it. to use it. There is also NPX. And you can There is also NPX. And you can There is also NPX. And you can just clone just clone just clone repository locally repository locally repository locally and follow and follow and follow instructions with instructions with instructions with installation depending installation depending installation depending from which from which from which agents you agents you agents you use. use. Good. So, we have Good. So, we have skill file. We are it skill file. We are it skill file. We are it established. Therefore, established. Therefore, established. Therefore, now I may now I may now I may I just want to I just want to I just want to run this run this run this mechanism. I want mechanism. I want mechanism. I want see it in action. So see it in action. So , I'll do two here , I'll do two here , I'll do two here various things. I various things. I various things. I I'm going to do this. I'm going to do this. I'm going to do this. launch. And then I launch. And then I launch. And then I I will show you.

I will show you. I will show you. previous example, previous example, previous example, so that you don't sit here, so that you don't sit here, so that you don't sit here, watching how watching how watching how Codex is working. So, we Codex is working. So, we Codex is working. So, we We will do that. We We will do that. We We will do that. We let's allow this let's allow this let's allow this work. I am going work. I am going work. I am going move on to another move on to another move on to another repository. Good. repository. Good. repository. Good. So, the same So, the same So, the same request. And in general, request. And in general, request. And in general, Let me, I think. Let me, I think. , I missed one thing , I missed one thing , which is probably important , which is probably important , which is probably important to highlight. This, in fact, to highlight. This, in fact, to highlight. This, in fact, magazines that were magazines that were magazines that were created, or topics, created, or topics, created, or topics, which were generated. which were generated. which were generated. So now we have So now we have So now we have be all this signal, be all this signal, be all this signal, isn't that right? This is what for isn't that right? This is what for isn't that right? This is what for What did I just say? What did I just say? What did I just say? applied Codex to applied Codex to applied Codex to of this project. Now of this project. Now of this project. Now he must be able to he must be able to he must be able to get all these different get all these different get all these different things, right? These things, right? These things, right? These tasks, these moods, tasks, these moods, tasks, these moods, these problems of the worker these problems of the worker these problems of the worker support process. support process. support process. All this and these ratings All this and these ratings All this and these ratings attached to those attached to those attached to those tracings and now there are tracings and now there are tracings and now there are part of this part of this part of this workflow. So workflow. So , let it work in , let it work in , let it work in this terminal here this terminal here this terminal here here. Okay, we here. Okay, we here. Okay, we let's get back to this, let's get back to this, let's get back to this, while it works.

while it works. So, just like that, So, just like that, What did you see in the session? What did you see in the session? What did you see in the session? cycle, you will start cycle, you will start cycle, you will start see how Codex writes SQL- see how Codex writes SQL- teams. So, SQL, and we teams. So, SQL, and we teams. So, SQL, and we we are going to get we are going to get we are going to get all this information or all this information or all this information or relevant data in relevant data in relevant data in session. So we can session. So we can session. So we can make requests flexibly make requests flexibly make requests flexibly to this project and to this project and to this project and to these tracings, and we to these tracings, and we to these tracings, and we must be able to must be able to must be able to recognize and recognize and recognize and understand, or Codex has understand, or Codex has understand, or Codex has to be able to to be able to to be able to recognize and recognize and recognize and understand these topics and understand these topics and understand these topics and these estimates that were these estimates that were these estimates that were launched. So, you launched. So, you launched. So, you you see here we have 968 you see here we have 968 you see here we have 968 support traces, support traces, support traces, this is exactly what we this is exactly what we this is exactly what we downloaded. THERE ARE downloaded. THERE ARE downloaded. THERE ARE estimated failures, estimated failures, estimated failures, noticeable cluster noticeable cluster noticeable cluster tool errors tool errors tool errors transmission to a person. I transmission to a person. I transmission to a person. I I'm going to check. I'm going to check. I'm going to check. results results results classifier and classifier and classifier and representative representative representative failed tracings and failed tracings and failed tracings and turn it into turn it into turn it into a specific resume. a specific resume. a specific resume. So, again, So, again, So, again, possibility of flexibility possibility of flexibility possibility of flexibility to make inquiries for to make inquiries for to make inquiries for using this SQL- using this SQL- the tool becomes the tool becomes the tool becomes quite impressive. quite impressive. quite impressive. It seems he was able to do something. It seems he was able to do something. It seems he was able to do something. to recognize by to recognize by to recognize by with the help of our with the help of our with the help of our grades. And then he, grades. And then he, grades. And then he, you know, constantly you know, constantly you know, constantly makes requests. And that's it.

makes requests. And that's it. what I was talking about what I was talking about what I was talking about earlier, —possibility earlier, —possibility earlier, —possibility to study in detail to study in detail to study in detail separate tracing. separate tracing. separate tracing. So, he is something So, he is something So, he is something found. He found found. He found found. He found example, which example, which example, which will allow us again will allow us again will allow us again improve our improve our improve our agent certain agent certain agent certain in a measured way in a measured way . We want . We want . We want view this trace, view this trace, view this trace, write some more SQL, write some more SQL, write some more SQL, but that's what allows but that's what allows but that's what allows we should be very we should be very we should be very exhaustive and exhaustive and exhaustive and actually receive actually receive actually receive all this rich data, all this rich data, all this rich data, but choose only that, but choose only that, but choose only that, which will really lead to which will really lead to which will really lead to changes we can make changes we can make changes we can make to contribute to our to contribute to our to contribute to our agent. Okay, so agent. Okay, so agent. Okay, so let's see, let's see, let's see, what else did we just do what else did we just do what else did we just do done. It seems we done. It seems we done. It seems we we consider how we consider how we consider how run assessments. We run assessments. We run assessments. We we review the available we review the available we review the available datasets. We datasets. We datasets. We just created a new one just created a new one just created a new one data set. data set. data set. In fact, we are in In fact, we are in In fact, we are in help team. But help team. But help team. But I hope that you I hope that you I hope that you you can see here, you can see here, This is it. This is what we This is it. This is what we we are trying to create we are trying to create . This is what...this is . This is what...this is . This is what...this is primitives in BrainTrust, primitives in BrainTrust, primitives in BrainTrust, our topics, our BrainTrust our topics, our BrainTrust our topics, our BrainTrust CLI, backend, which CLI, backend, which CLI, backend, which allows you to write SQL, allows you to write SQL, allows you to write SQL, having a platform that having a platform that having a platform that performs both assessments and performs both assessments and performs both assessments and observability.

observability. All these things taken All these things taken together, allow you together, allow you together, allow you to create again to create again to create again this flywheel, and this flywheel, and this flywheel, and do it for real do it for real do it for real in an effective way. in an effective way. I would be happy. I would be happy. respond to respond to respond to direct question direct question direct question now. We wait a little, now. We wait a little, now. We wait a little, until Codex finishes until Codex finishes until Codex finishes work, but I wonder if work, but I wonder if work, but I wonder if Does anyone have...Yes, please Does anyone have...Yes, please Does anyone have...Yes, please affection. Yes, we have affection. Yes, we have affection. Yes, we have hybrid deployment hybrid deployment . This means that . This means that . This means that data plane, where data plane, where data plane, where all the data is there, all the data is there, all the data is there, that is, your traces, that is, your traces, that is, your traces, experiments, sets experiments, sets experiments, sets data, data, data, will remain in will remain in will remain in your infrastructure your infrastructure , and the control plane, , and the control plane, , and the control plane, i.e. a web application, i.e. a web application, i.e. a web application, which you see, which you see, which you see, will stay with us. will stay with us. Hybrid deployment Hybrid deployment available only for available only for available only for corporate plan corporate plan . Yes. Moreover, if . Yes. Moreover, if . Yes. Moreover, if Are you curious, there are two? Are you curious, there are two? Are you curious, there are two? different options for this. different options for this. different options for this. There is BYOC, where again There is BYOC, where again There is BYOC, where again data plane data plane data plane unfolds in unfolds in unfolds in your infrastructure your infrastructure , but we manage it , but we manage it , but we manage it for you. Then there is more for you. Then there is more for you. Then there is more hybrid version: hybrid version: hybrid version: unfolds in unfolds in unfolds in your data plane, your data plane, your data plane, and your team is in charge and your team is in charge and your team is in charge by her. We have a provider by her. We have a provider by her. We have a provider Terraform or Helm charts Terraform or Helm charts Terraform or Helm charts depending on whether depending on whether depending on whether Which cloud are you?

Which cloud are you? Which cloud are you? working. Someone in working. Someone in working. Someone in end of the hall? You don't end of the hall? You don't end of the hall? You don't be sure to shout. be sure to shout. I think, for the most part, I think, for the most part, and it seems like it's a bit and it seems like it's a bit and it seems like it's a bit changing, people changing, people changing, people use use use more models from more models from more models from big big big suppliers suppliers suppliers models. This is generally models. This is generally models. This is generally what I see. I think what I see. I think what I see. I think that the situation is beginning that the situation is beginning that the situation is beginning change a little. I change a little. I change a little. I I know that at the beginning I know that at the beginning I know that at the beginning everything was very strong everything was very strong everything was very strong tilted to the side tilted to the side tilted to the side main main main suppliers suppliers suppliers models. Hmm, I don't know. models. Hmm, I don't know. models. Hmm, I don't know. exactly, I didn't see exactly, I didn't see exactly, I didn't see small models as small models as small models as such, but I saw how such, but I saw how such, but I saw how people start, um, people start, um, people start, um, perhaps, perhaps, perhaps, to fine-tune to fine-tune to fine-tune own models and own models and own models and to see if it can to see if it can to see if it can to push it, to push it, to push it, you know, the idea itself, you know, the idea itself, you know, the idea itself, again, behind this, again, behind this, again, behind this, for example, for example, for example, evaluation, so that evaluation, so that evaluation, so that understand, can I understand, can I understand, can I replace models replace models replace models without input without input without input regressions or, um, regressions or, um, regressions or, um, quality degradation quality degradation quality degradation agent. But, yes, I would agent. But, yes, I would agent. But, yes, I would said, mostly said, mostly said, mostly these are big these are big these are big suppliers suppliers suppliers models, perhaps, models, perhaps, models, perhaps, are changing a little.

are changing a little. Sorry, this is for LLM- judges? A. Yes, it is important. judges? A. Yes, it is important. judges? A. Yes, it is important. note, it is possible note, it is possible note, it is possible to see, probably, on to see, probably, on to see, probably, on this screen when you this screen when you this screen when you look at look at look at settings settings settings automation of topics. and automation of topics. and automation of topics. and additional. Only additional. Only additional. Only the option you have the option you have the option you have for stages for stages for stages generalization and generalization and generalization and vector vector vector embedding—this embedding—this embedding—this models placed on models placed on models placed on Braintrust. Um, there are a few. Braintrust. Um, there are a few. Braintrust. Um, there are a few. reasons for this. On- reasons for this. On- first, we taught these first, we taught these first, we taught these models do just that models do just that . And secondly, . And secondly, . And secondly, conveyor complexity, conveyor complexity, conveyor complexity, essentially does this, essentially does this, essentially does this, that you can't that you can't that you can't just connect just connect just connect any model and any model and any model and to hope for similar to hope for similar to hope for similar results. We results. We results. We started with the fact that started with the fact that started with the fact that allowed to connect allowed to connect allowed to connect different models, but different models, but different models, but topic or theme pipeline, topic or theme pipeline, topic or theme pipeline, which were generated in which were generated in which were generated in as a result, were as a result, were as a result, were simply poor quality.

simply poor quality. So we established that So we established that for this step for this step for this step only models available, only models available, only models available, hosted on Braintrust. hosted on Braintrust. Yes, you can. Um, I Yes, you can. Um, I I move on to So, I move on to So, I move on to So, because all this data because all this data because all this data exist as metadata exist as metadata exist as metadata on top of the tracing. You on top of the tracing. You on top of the tracing. You you can take them out for you can take them out for you can take them out for various visualizations. various visualizations. So, I think this is So, I think this is I have a case. I have a case. I have a case. saved view. Ahem saved view. Ahem , and just, you know, , and just, you know, , and just, you know, some things from Braintrust some things from Braintrust some things from Braintrust here. If you wanted here. If you wanted here. If you wanted create your own create your own create your own your own dashboard or your own dashboard or your own dashboard or your own appearance, your own appearance, your own appearance, this is how you would do it this is how you would do it this is how you would do it done. All these done. All these done. All these looks can be looks can be looks can be save, you can save, you can save, you can create your own create your own create your own graphics, but from above graphics, but from above graphics, but from above you can see you can see you can see mood dynamics, mood dynamics, mood dynamics, task dynamics, task dynamics, task dynamics, best wishes, best wishes, best wishes, the best tasks, the best tasks, the best tasks, that is, things that you that is, things that you that is, things that you created by hand, created by hand, created by hand, can be extracted here.

can be extracted here. can be extracted here. You see what this is about. You see what this is about. You see what this is about. I am looking for a case. I am looking for a case. I am looking for a case. classifications, where classifications, where classifications, where sentiment is not empty, and sentiment is not empty, and sentiment is not empty, and I group by this I group by this I group by this label. So label. So label. So set it up set it up set it up quite simple. Yes, I am. quite simple. Yes, I am. quite simple. Yes, I am. I think it's that again. I think it's that again. , treat with a certain , treat with a certain , treat with a certain a touch of skepticism. a touch of skepticism. a touch of skepticism. In general, I noticed that In general, I noticed that In general, I noticed that newer agents or newer agents or newer agents or new things that just happened new things that just happened new things that just happened are produced in are produced in are produced in production, have production, have production, have higher sampling rate. higher sampling rate. higher sampling rate. Over time, these "famous" Over time, these "famous" Over time, these "famous" unknowns" or things, unknowns" or things, unknowns" or things, What are your grades? What are your grades? What are your grades? usually found, usually found, usually found, are becoming less and less are becoming less and less are becoming less and less frequent, and the signal, frequent, and the signal, frequent, and the signal, Which one are you? Which one are you? Which one are you? you get, too you get, too you get, too decreases. Therefore you decreases. Therefore you decreases. Therefore you start gradually start gradually start gradually reduce this frequency reduce this frequency reduce this frequency sampling. Me too sampling. Me too sampling. Me too I think that if you I think that if you I think that if you use use use LLM-based evaluator, LLM-based evaluator, LLM-based evaluator, which one is bigger which one is bigger which one is bigger oriented towards oriented towards oriented towards classification or classification or classification or taxonomy, you can taxonomy, you can taxonomy, you can try to replace try to replace try to replace it on topics. it on topics. Dropbox is a client that Dropbox is a client that which I work for, and they which I work for, and they which I work for, and they have a very complex have a very complex have a very complex series series series classification classification classification the grades they the grades they the grades they are launched to are launched to are launched to to understand a certain to understand a certain to understand a certain taxonomy regarding taxonomy regarding taxonomy regarding of their tracings. AND of their tracings. AND of their tracings. AND they they they experimented with experimented with experimented with using themes, using themes, using themes, to do this. On- to do this. On- first, because first, because first, because the most important reason the most important reason the most important reason this is that this this is that this this is that this lot lot lot more economical way more economical way more economical way processing of the majority processing of the majority processing of the majority or even 100% of them or even 100% of them or even 100% of them tracings, where LLM- tracings, where LLM- the appraiser simply does not the appraiser simply does not the appraiser simply does not could scale.

could scale. could scale. You are able to generate You are able to generate You are able to generate the same amount of signal. the same amount of signal. the same amount of signal. So, to sum up, I So, to sum up, I So, to sum up, I I think that maturity I think that maturity I think that maturity system determines, system determines, system determines, what should the frequency be what should the frequency be what should the frequency be sampling. She sampling. She sampling. She usually decreases usually decreases usually decreases as it grows as it grows as it grows maturity of this maturity of this maturity of this agent. And there are also types agent. And there are also types agent. And there are also types things that can things that can things that can create your LLM- create your LLM- appraiser. If this appraiser. If this appraiser. If this more about more about more about classification or classification or classification or taxonomies, you can taxonomies, you can taxonomies, you can replace it with themes. replace it with themes. replace it with themes. Yes, not directly Yes, not directly Yes, not directly in the interface, but you in the interface, but you in the interface, but you you can create you can create you can create code-based assessment code-based assessment . Isn't that right? You can . Isn't that right? You can . Isn't that right? You can imagine what I'm causing imagine what I'm causing imagine what I'm causing rating one. On rating one. On rating one. On result-based result-based result-based I am the only one making assessments. I am the only one making assessments. I am the only one making assessments. I am causing something. I am causing something. I am causing something. rating two. rating two. rating two. The result of this The result of this The result of this functions are array functions are array functions are array grades. Isn't that right? It grades. Isn't that right? It grades. Isn't that right? It array of evaluation objects array of evaluation objects . But yes, it is . But yes, it is . But yes, it is certainly possible, certainly possible, certainly possible, however you determine however you determine however you determine it's in the code. At Braintrust it's in the code. At Braintrust it's in the code. At Braintrust there is nothing that there is nothing that there is nothing that would allow me to say: would allow me to say: would allow me to say: "Hey, I want to "Hey, I want to "Hey, I want to run the A grade and run the A grade and run the A grade and based on it based on it based on it to transfer the result to transfer the result to transfer the result given a grade of B or C", given a grade of B or C", given a grade of B or C", but it is quite possible but it is quite possible but it is quite possible implement in code. AT, implement in code. AT, implement in code. AT, Yes. You can also Yes. You can also Yes. You can also customize yours customize yours customize yours filter so that it filter so that it filter so that it specifically searched specifically searched specifically searched the presence of this the presence of this the presence of this the estimation interval, and the estimation interval, and the estimation interval, and it would work it would work it would work only when he is there.

only when he is there. only when he is there. Yes. I forgot the name. Yes. I forgot the name. Yes. I forgot the name. client, but this client, but this client, but this the question was raised the question was raised the question was raised the other day, and it was the other day, and it was the other day, and it was more on the task side more on the task side . Hmm, I don't remember. . Hmm, I don't remember. . Hmm, I don't remember. exactly what it was, but exactly what it was, but exactly what it was, but that was actually what that was actually what that was actually what what they don't even talk about what they don't even talk about what they don't even talk about guessed that guessed that guessed that users do users do users do with their agent. So with their agent. So with their agent. So it was more of a coincidence, it was more of a coincidence, it was more of a coincidence, such as a request for such as a request for such as a request for function or, you know, " function or, you know, " unknown unknown" unknown unknown" unknown unknown" regarding that regarding that regarding that users do users do users do with an agent, not with an agent, not with an agent, not necessarily a failure in necessarily a failure in necessarily a failure in work. And I think that work. And I think that work. And I think that this is probably the most this is probably the most this is probably the most amazing thing for amazing thing for amazing thing for me—what people me—what people me—what people use this, use this, use this, to reveal such to reveal such to reveal such things. I think when I things. I think when I things. I think when I started started started to experiment with to experiment with to experiment with this function and this function and this function and communicate with communicate with communicate with customers, most customers, most customers, most from what I am from what I am from what I am concentrated, concentrated, concentrated, related to types related to types related to types problems that can problems that can problems that can reveal. But I think reveal. But I think reveal. But I think that exists so much that exists so much that exists so much wide range wide range wide range patterns that are now patterns that are now patterns that are now can be attached to can be attached to can be attached to your traces and your traces and your traces and use them use them use them together to find out together to find out together to find out something new about yourself something new about yourself something new about yourself agent, what are you doing before agent, what are you doing before agent, what are you doing before didn't know. So yes, in didn't know. So yes, in didn't know. So yes, in I don't have exact numbers.

I don't have exact numbers. I don't have exact numbers. data about what exactly data about what exactly data about what exactly reveal these topics, reveal these topics, reveal these topics, but it was more in but it was more in but it was more in style: "Hey, I don't style: "Hey, I don't style: "Hey, I don't knew that users knew that users knew that users use ours use ours use ours agents or agents or agents or are trying are trying are trying use mine use mine use mine agent for this." agent for this." Now, I would say, Now, I would say, they are oriented towards they are oriented towards they are oriented towards scenarios scenarios scenarios use, or use, or use, or can you even imagine can you even imagine can you even imagine their specific to their specific to their specific to each agent. Therefore, each agent. Therefore, each agent. Therefore, in my case, in in my case, in in my case, in I have an agent. I have an agent. I have an agent. support, and in his support, and in his support, and in his I am writing a project. I am writing a project. I am writing a project. my agent logs my agent logs my agent logs support for the project support for the project support for the project this agent. Me too this agent. Me too this agent. Me too I keep the sets. I keep the sets. I keep the sets. data for which data for which data for which I am starting the assessment. I am starting the assessment. I am starting the assessment. your agent your agent your agent support. Me too support. Me too support. Me too I keep grades for I keep grades for I keep grades for this agent this agent this agent support in the same support in the same support in the same project. So, they, project. So, they, project. So, they, as a rule, there is as a rule, there is as a rule, there is container for container for container for relevant resources relevant resources relevant resources for a specific scenario for a specific scenario for a specific scenario use or use or use or agent. Soon we agent. Soon we agent. Soon we we will implement in the project we will implement in the project we will implement in the project agent object as agent object as agent object as complete element complete element complete element first class, so first class, so first class, so can you imagine that can you imagine that can you imagine that there is a multi-agent there is a multi-agent there is a multi-agent system, and you want, system, and you want, system, and you want, so that all agent logs so that all agent logs so that all agent logs fell into one fell into one fell into one project, but at the same time project, but at the same time project, but at the same time was kept was kept was kept distinction between distinction between distinction between agents A, B and C. It is possible agents A, B and C. It is possible agents A, B and C. It is possible Imagine that.

Imagine that. So yes, the page So yes, the page monitoring, which I monitoring, which I monitoring, which I showed earlier, showed earlier, showed earlier, limited by boundaries limited by boundaries limited by boundaries project. If you project. If you project. If you want to create want to create want to create broader representation broader representation broader representation at the level of the whole at the level of the whole at the level of the whole organizations, you organizations, you organizations, you will have to will have to will have to create queries and create queries and create queries and extract data from extract data from extract data from BrainTrust. This over time BrainTrust. This over time BrainTrust. This over time will change. Question in will change. Question in will change. Question in because we because we because we allow our allow our allow our customers to do it in customers to do it in customers to do it in on a large scale? You on a large scale? You on a large scale? You can you imagine that can you imagine that can you imagine that attempt to make requests attempt to make requests attempt to make requests to many different to many different to many different projects and construction projects and construction projects and construction of this data on one of this data on one of this data on one dashboard dashboard dashboard it gets a little complicated it gets a little complicated it gets a little complicated task. So, yes, task. So, yes, task. So, yes, at the moment we at the moment we at the moment we we have a project we have a project we have a project approach, but you can approach, but you can approach, but you can to assume that with to assume that with to assume that with sometimes it will start sometimes it will start sometimes it will start change. I have on change. I have on change. I have on meaning that this is probably meaning that this is probably meaning that this is probably rightly. Um, I rightly. Um, I rightly. Um, I I think you are potentially I think you are potentially I think you are potentially you will see that your you will see that your you will see that your assessments assessments assessments will change like this will change like this will change like this as often as myself as often as myself as often as myself agent. Maybe this agent. Maybe this agent. Maybe this this is not happening this is not happening this is not happening often, but I think often, but I think often, but I think What is the prospect of change?

What is the prospect of change? What is the prospect of change? based on what you based on what you based on what you you can learn from you can learn from you can learn from interactions interactions interactions users with this users with this users with this agent, right? It agent, right? It agent, right? It there may be a clue, there may be a clue, there may be a clue, this could be this could be this could be tool, it can tool, it can tool, it can to be new to be new to be new a tool, or something a tool, or something a tool, or something in the environment in the environment in the environment execution or execution or execution or architecture that architecture that architecture that needs changes. As for needs changes. As for needs changes. As for I saw me more often, I saw me more often, I saw me more often, that assessments usually that assessments usually that assessments usually change in change in change in as a result of something that as a result of something that as a result of something that you change it yourself. you change it yourself. Of course, if it is Of course, if it is simple change simple change simple change tips, you are unlikely tips, you are unlikely tips, you are unlikely Do you change a lot in Do you change a lot in Do you change a lot in estimates, except, estimates, except, estimates, except, maybe adding maybe adding maybe adding new examples to new examples to new examples to data set. But I data set. But I data set. But I really saw that really saw that really saw that agents have agents have agents have tendency to tendency to tendency to development, and if you development, and if you development, and if you rely on rely on rely on real interaction with real interaction with real interaction with users, they users, they users, they may change may change may change a little more often than you a little more often than you a little more often than you expect. But yes, I am expect. But yes, I am expect. But yes, I am I think that this I think that this I think that this fair review, fair review, fair review, if the agent is not very if the agent is not very if the agent is not very complex. He doesn't have complex. He doesn't have complex. He doesn't have or, you know, not or, you know, not or, you know, not needs many needs many needs many changes, based on the fact that changes, based on the fact that changes, based on the fact that What exactly does the agent do?

What exactly does the agent do? . So the estimates may not . So the estimates may not . So the estimates may not too much change in too much change in too much change in as a result of this. I as a result of this. I as a result of this. I I still think that I still think that I still think that the value lies in the value lies in the value lies in in order to be able to in order to be able to in order to be able to generate this generate this generate this work signal work signal work signal systems in real systems in real systems in real conditions. Especially if conditions. Especially if conditions. Especially if it can be done it can be done it can be done cost-effective cost-effective , then yes, I still , then yes, I still , then yes, I still I see this I see this I see this value. Yes, I have. value. Yes, I have. value. Yes, I have. there are customers who there are customers who there are customers who They do both. They do both. They do both. I think that's partly it. I think that's partly it. I think that's partly it. a matter of preference. AND a matter of preference. AND a matter of preference. AND sometimes it's simple sometimes it's simple sometimes it's simple misunderstanding of what misunderstanding of what misunderstanding of what what Braintrust is capable of. what Braintrust is capable of. For example, estimates For example, estimates codebase—this is how codebase—this is how codebase—this is how to me, wonderful to me, wonderful to me, wonderful example where you can example where you can example where you can to be very difficult to be very difficult to be very difficult assessment that assessment that assessment that uses different uses different uses different Python dependencies or Python dependencies or Python dependencies or TypeScript. And you need TypeScript. And you need TypeScript. And you need run this, or you run this, or you run this, or you What do you think? What do you think? What do you think? need to run need to run need to run it's in your own it's in your own it's in your own infrastructure, infrastructure, infrastructure, because she because she because she supported supported supported many complex many complex many complex dependencies. At Braintrust dependencies. At Braintrust when you send when you send when you send assessment, even assessment assessment, even assessment assessment, even assessment code base, you code base, you code base, you you can add all you can add all you can add all required dependencies required dependencies required dependencies directly to directly to directly to this request. Therefore, this request. Therefore, this request. Therefore, things that things that things that this one relies on this one relies on this one relies on indicator, still indicator, still indicator, still can can can use. I use. I use. I I know that some I know that some I know that some clients I work with clients I work with clients I work with worked, just not worked, just not worked, just not understood that they understood that they understood that they could do it. For could do it. For could do it. For I value I value I value is to not is to not is to not have parts in the code have parts in the code have parts in the code that asynchronous that asynchronous that asynchronous code that looks for these code that looks for these code that looks for these triggers or triggers or triggers or checks whether it was checks whether it was checks whether it was created a trace, created a trace, created a trace, and waits 60 seconds and waits 60 seconds and waits 60 seconds after the last one after the last one after the last one events to launch.

events to launch. So for me the value So for me the value consists of consists of consists of moving parts moving parts moving parts this complexity in this complexity in this complexity in Braintrust. Although you all Braintrust. Although you all Braintrust. Although you all you can do it anyway you can do it anyway . One of the unique, . One of the unique, . One of the unique, in my opinion, rice in my opinion, rice in my opinion, rice Braintrust data models are Braintrust data models are Braintrust data models are what you can what you can what you can update spans: update spans: update spans: if span already exists in if span already exists in if span already exists in Braintrust, it is variable. Braintrust, it is variable. You can add You can add or change or change or change metadata, metadata, metadata, attach grades attach grades attach grades and various metrics. So, and various metrics. So, and various metrics. So, if you wanted if you wanted if you wanted do it in code, in do it in code, in do it in code, in you would have been there already you would have been there already you would have been there already prepared prepared prepared necessary necessary necessary infrastructure with infrastructure with infrastructure with your side. This becomes your side. This becomes your side. This becomes quite simple quite simple quite simple tasks, such as: tasks, such as: tasks, such as: "Hey, I have this ID "Hey, I have this ID "Hey, I have this ID sleep. I want to add sleep. I want to add sleep. I want to add to him these assessments and to him these assessments and to him these assessments and metrics". I think, metrics". I think, metrics". I think, partly it's simple partly it's simple partly it's simple a matter of preference. a matter of preference. a matter of preference. Some people want Some people want Some people want to have a little more to have a little more to have a little more control. This is quite control. This is quite control. This is quite reasonably. You, of course, reasonably. You, of course, reasonably. You, of course, You can do it. You can do it. You can do it. For me, it's probably For me, it's probably For me, it's probably management issues management issues management issues access based on access based on access based on roles (RBAC). Do you want roles (RBAC). Do you want roles (RBAC). Do you want to be able to to be able to to be able to record this data record this data record this data tracing and tracing and tracing and to limit the circle of those to limit the circle of those to limit the circle of those who can see them. I who can see them. I who can see them. I I believe that the described I believe that the described I believe that the described my workflow— my workflow— this automated this automated this automated the mechanism is wonderful the mechanism is wonderful the mechanism is wonderful way to do it, way to do it, way to do it, because I can because I can because I can to provide people to provide people to provide people insights. I can insights. I can insights. I can create PR based on create PR based on create PR based on all these insights, all these insights, all these insights, but not to give them but not to give them but not to give them access to themselves access to themselves access to themselves tracings. I give them tracings. I give them tracings. I give them access to access to access to analysis results analysis results analysis results these traces. Here is the SQL- these traces. Here is the SQL- requests that I have completed.

requests that I have completed. requests that I have completed. Here is the aggregated data, Here is the aggregated data, Here is the aggregated data, which I received. If which I received. If which I received. If no engineer no engineer no engineer must see these must see these must see these data, write them down in data, write them down in data, write them down in project to which project to which project to which no one has access. no one has access. no one has access. Restrict access from this Restrict access from this Restrict access from this points of view. But points of view. But points of view. But create a service create a service create a service account that account that account that will be able to run this will be able to run this will be able to run this mechanism on top of these mechanism on top of these mechanism on top of these data. Thus, you data. Thus, you data. Thus, you don't lose access to don't lose access to don't lose access to all these insights. all these insights. Yes. Most Yes. Most undervalued functions. undervalued functions. undervalued functions. Ahem. I suppose that Ahem. I suppose that Ahem. I suppose that Braintrust CLI is rated Braintrust CLI is rated Braintrust CLI is rated quite adequately. I quite adequately. I quite adequately. I I think I advertised. I think I advertised. I think I advertised. would have it at any cost would have it at any cost would have it at any cost occasion or occasion or occasion or would tell about would tell about would tell about him to anyone him to anyone him to anyone to the engineer. I say this. to the engineer. I say this. to the engineer. I say this. because we seem to because we seem to because we seem to they released him they released him they released him at the end of February. IN at the end of February. IN at the end of February. IN we had a conference we had a conference we had a conference users. Um, me too users. Um, me too users. Um, me too still, communicating with still, communicating with still, communicating with customers today, customers today, customers today, I ask: whether I ask: whether I ask: whether do you use do you use do you use Braintrust CLI? Or Braintrust CLI? Or Braintrust CLI? Or Do you use it? Do you use it? Do you use it? And some of them still haven't. And some of them still haven't. And some of them still haven't. Um, but as for me, I Um, but as for me, I Um, but as for me, I ready to bet, ready to bet, ready to bet, that 100% of you that 100% of you that 100% of you use use use some agent for some agent for some agent for coding, right?

coding, right? coding, right? It's the way you write. It's the way you write. It's the way you write. code today. Um, if code today. Um, if code today. Um, if CLI is part of this, I CLI is part of this, I CLI is part of this, I I hope you were able to I hope you were able to I hope you were able to to see what types to see what types to see what types workflows workflows workflows can be created. As for can be created. As for can be created. As for me, it becomes me, it becomes me, it becomes incredibly incredibly incredibly convincing. You already convincing. You already convincing. You already in the terminal, you are already in in the terminal, you are already in in the terminal, you are already in in your IDE, or elsewhere. in your IDE, or elsewhere. in your IDE, or elsewhere. Use the CLI, Use the CLI, Use the CLI, add it or add it or add it or train your agent train your agent train your agent for coding, as it for coding, as it for coding, as it use. AND use. AND use. AND then use then use then use this is to improve this is to improve this is to improve your work process. your work process. your work process. Ahem. Another thing that... Ahem. Another thing that... Another thing that can Another thing that can to be quite interesting, to be quite interesting, to be quite interesting, depends, probably, depends, probably, depends, probably, a little bit of yours a little bit of yours a little bit of yours organizations and organizations and organizations and different people who different people who different people who contribute to this contribute to this contribute to this flywheel. We have flywheel. We have flywheel. We have function that function that function that called " called " remote assessment"(remote remote assessment"(remote remote assessment"(remote eval). This is a way to endure eval). This is a way to endure eval). This is a way to endure this rating for the game this rating for the game this rating for the game playground. AND playground. AND playground. AND it's more about how, it's more about how, it's more about how, maybe in this maybe in this maybe in this scenarios, there are experts scenarios, there are experts scenarios, there are experts from the subject area, from the subject area, from the subject area, managers from managers from managers from products, people who products, people who products, people who in a certain way in a certain way in a certain way contribute to the development contribute to the development contribute to the development this flywheel. this flywheel. this flywheel. For example, they For example, they For example, they change the prompts change the prompts change the prompts or are engaged in or are engaged in or are engaged in optimization. And we optimization. And we optimization. And we created all these created all these created all these programs for them to programs for them to programs for them to they could they could they could connect via connect via connect via using low-code using low-code using low-code approach. Remote approach. Remote approach. Remote the assessment allows you to the assessment allows you to the assessment allows you to to give that assessment, to give that assessment, to give that assessment, which I was showing which I was showing which I was showing earlier. This assessment is from earlier. This assessment is from earlier. This assessment is from task and set task and set task and set data can be data can be data can be to put on the game board to put on the game board to put on the game board playground with playground with playground with parameters to parameters to parameters to the user could the user could the user could start changing start changing start changing system prompt, system prompt, system prompt, description of the instrument or description of the instrument or description of the instrument or something like that. Why is this?

something like that. Why is this? something like that. Why is this? may be interesting for may be interesting for may be interesting for AI engineer: you don't AI engineer: you don't AI engineer: you don't must be " must be " "bottleneck" in "bottleneck" in "bottleneck" in creating applications creating applications creating applications or preparation or preparation or preparation tools to tools to tools to product manager product manager product manager could change it myself could change it myself could change it myself hint or do hint or do hint or do contribution to this flywheel. contribution to this flywheel. So maybe not. So maybe not. influential as CLI, but influential as CLI, but influential as CLI, but potentially could be, potentially could be, potentially could be, according to according to according to your structure your structure your structure organizations. organizations. organizations. I think I caught it. I think I caught it. I think I caught it. essence. Tell me where essence. Tell me where essence. Tell me where I am wrong. I think so. I am wrong. I think so. what I started doing what I started doing what I started doing here, that's it. THERE ARE, here, that's it. THERE ARE, here, that's it. THERE ARE, Braintrust as such, Braintrust as such, Braintrust as such, at least for now at least for now at least for now moment, does not have moment, does not have moment, does not have access to your access to your access to your code base, right? code base, right? code base, right? Yes? He doesn't understand. Yes? He doesn't understand. Yes? He doesn't understand. of what you wrote, of what you wrote, of what you wrote, and, based on the existing and, based on the existing and, based on the existing information, cannot information, cannot information, cannot necessarily necessarily necessarily recommend changes recommend changes recommend changes to specific to specific to specific functions or functions or functions or agent architecture. agent architecture. agent architecture. But when you But when you But when you implement Braintrust CLI implement Braintrust CLI implement Braintrust CLI at your agent for at your agent for at your agent for coding and coding and coding and connect it, connect it, connect it, of course, being in of course, being in of course, being in to my repository, I to my repository, I to my repository, I now I can now I can now I can try to see, try to see, try to see, Did we achieve anything? Did we achieve anything? Did we achieve anything? get. So, in my get. So, in my get. So, in my sessions, based on sessions, based on sessions, based on what I was able to do what I was able to do what I was able to do understand, we understand, we understand, we we create new ones we create new ones we create new ones evaluations. So, there is a change.

evaluations. So, there is a change. evaluations. So, there is a change. evaluations. Well, we can. evaluations. Well, we can. evaluations. Well, we can. skip tests. skip tests. It seems that they appeared It seems that they appeared new tools that new tools that new tools that we were able to add to we were able to add to we were able to add to our agent, again our agent, again our agent, again anyway, based on these anyway, based on these anyway, based on these conclusions. So I conclusions. So I conclusions. So I I think the answer is I think the answer is I think the answer is to your question: this to your question: this to your question: this quite possible. That which quite possible. That which quite possible. That which I just I just I just demonstrated, —this demonstrated, —this demonstrated, —this reverse mechanism reverse mechanism reverse mechanism communication: "Hey, communication: "Hey, communication: "Hey, find out what find out what find out what takes place in takes place in takes place in production, production, production, use mine use mine use mine grades or topics, so grades or topics, so grades or topics, so that I added information that I added information that I added information about problems, moods about problems, moods about problems, moods or something else." This becomes or something else." This becomes or something else." This becomes extremely interesting extremely interesting extremely interesting for the agent to for the agent to for the agent to could get it, but could get it, but could get it, but then modify then modify then modify your agent certain your agent certain your agent certain in a way. I think later, in a way. I think later, in a way. I think later, moving forward, you moving forward, you moving forward, you you will see that now in you will see that now in you will see that now in we have new examples in we have new examples in we have new examples in our dataset. our dataset. We seem to have We seem to have a new set has appeared a new set has appeared a new set has appeared data with these data with these data with these examples. We examples. We examples. We we are going to launch we are going to launch we are going to launch our assessment, probably our assessment, probably , a little lower. You , a little lower. You , a little lower. You you also see that we you also see that we you also see that we we make a request we make a request we make a request directly from that directly from that directly from that experiment, which experiment, which experiment, which was launched. Therefore, was launched. Therefore, was launched. Therefore, this whole mechanism this whole mechanism this whole mechanism displayed here in displayed here in displayed here in within this skill within this skill within this skill agent improvement agent improvement . And I hope that . And I hope that . And I hope that to some extent to some extent to some extent answered yours answered yours answered yours question. Class. It, question. Class. It, question. Class. It, in my opinion, one of in my opinion, one of in my opinion, one of the most interesting parts the most interesting parts the most interesting parts Braintrust at the moment Braintrust at the moment . We are all one way or another . We are all one way or another . We are all one way or another we are developing we are developing we are developing our agents for our agents for our agents for using an agent for using an agent for using an agent for coding. If I coding. If I coding. If I I can provide I can provide I can provide correct context

correct context correct context and get access to and get access to and get access to required primitives required primitives , which attach , which attach , which attach correct data to mine correct data to mine correct data to mine tracings, then this tracings, then this tracings, then this automatic automatic automatic the mechanism becomes quite the mechanism becomes quite the mechanism becomes quite interesting. So I interesting. So I interesting. So I absolutely sure, absolutely sure, absolutely sure, that it is possible. If I that it is possible. If I that it is possible. If I could show, I could show, I could show, I I will show you this one. I will show you this one. I will show you this one. example. Actually, if example. Actually, if example. Actually, if I will come back here...U I will come back here...U I will come back here...U within the platform within the platform within the platform there is no such thing now there is no such thing now there is no such thing now button that can be button that can be button that can be press to press to press to to say: "Hey, to say: "Hey, to say: "Hey, start this mechanism start this mechanism ". But what I tell you ". But what I tell you ". But what I tell you showed, this is what you showed, this is what you showed, this is what you you can very easily you can very easily you can very easily connect to connect to connect to this, this, this, using all using all using all available to you available to you available to you tools. What you tools. What you tools. What you now you can do now you can do now you can do based on this, right? based on this, right? based on this, right? Can you imagine? Can you imagine? Can you imagine? for example, for example, for example, connecting this to connecting this to connecting this to GitHub action, which GitHub action, which GitHub action, which starts with starts with starts with at a certain frequency at a certain frequency , is activated on , is activated on , is activated on the basis of something, and we the basis of something, and we the basis of something, and we we open PR. Or you we open PR. Or you we open PR. Or you can you imagine what this is can you imagine what this is can you imagine what this is activated activated activated by message in Slack, by message in Slack, by message in Slack, Yes? I think there is Yes? I think there is Yes? I think there is many different many different many different ways in which it ways in which it ways in which it can be activated.

can be activated. can be activated. Inside all Inside all Inside all tools available tools available tools available so that you can so that you can so that you can join this. join this. join this. Here is one example. Here Here is one example. Here Here is one example. Here example GitHub action, which example GitHub action, which example GitHub action, which I created. So, if I created. So, if I created. So, if look, there is YAML- look, there is YAML- file that is actually file that is actually file that is actually works through Cloud Code. works through Cloud Code. works through Cloud Code. It pulls up Braintrust CLI. It pulls up Braintrust CLI. It pulls up Braintrust CLI. He is improving his skills. He is improving his skills. He is improving his skills. But then again, that But then again, that But then again, that What are we still doing here? What are we still doing here? What are we still doing here? need, is need, is need, is understanding that understanding that understanding that what has changed in what has changed in what has changed in in this case? And this in this case? And this in this case? And this my very, very my very, very my very, very simple example simple example simple example agent supervisor. agent supervisor. agent supervisor. What has changed? Why these What has changed? Why these What has changed? Why these Have things changed? Why Have things changed? Why Have things changed? Why in this case Cloud Code in this case Cloud Code in this case Cloud Code recommended these recommended these recommended these changes? Here is the actual changes? Here is the actual changes? Here is the actual influence, right? Based on influence, right? Based on influence, right? Based on assessments that I conducted. assessments that I conducted. assessments that I conducted. Here are some links Here are some links Here are some links on Braintrust, where you can on Braintrust, where you can on Braintrust, where you can check everything. Here check everything. Here check everything. Here regressions that I regressions that I regressions that I discovered based on discovered based on discovered based on of this analysis. And here it is of this analysis. And here it is of this analysis. And here it is your PR. You are the one who your PR. You are the one who your PR. You are the one who is conducting an inspection. is conducting an inspection. is conducting an inspection. Decide if this is something Decide if this is something Decide if this is something it is valuable to us it is valuable to us it is valuable to us worth implementing. worth implementing. worth implementing. So, I am completely So, I am completely So, I am completely convinced that this is what convinced that this is what convinced that this is what what can you do what can you do what can you do join now join now join now today. We have today. We have today. We have customers who are customers who are customers who are do. One of them, do. One of them, do. One of them, the one who was on the first the one who was on the first the one who was on the first slide that I slide that I slide that I showed, showed, showed, uses this for uses this for uses this for creation of 5–10 PRs per creation of 5–10 PRs per creation of 5–10 PRs per a day more than a day more than a day more than earlier. If you have earlier. If you have earlier. If you have again there is all this again there is all this again there is all this information if you information if you information if you you are the super smart one you are the super smart one you are the super smart one Codex Cloud Code or something Codex Cloud Code or something Codex Cloud Code or something similar, flexible similar, flexible similar, flexible asks your workers asks your workers asks your workers data and extracts data and extracts data and extracts interesting, I think, this interesting, I think, this interesting, I think, this it's getting enough it's getting enough it's getting enough attractive for attractive for attractive for many different many different many different cases cases cases using. Yes.

using. Yes. using. Yes. Yes, that repository Yes, that repository Yes, that repository Braintrust skills? He has Braintrust skills? He has Braintrust skills? He has open weekend open weekend open weekend code. Yes. There's also YAML- code. Yes. There's also YAML- file that passed file that passed file that passed through, actually, GitHub, through, actually, GitHub, through, actually, GitHub, that is, he worked that is, he worked that is, he worked inside a GitHub action, and inside a GitHub action, and inside a GitHub action, and it is also open it is also open it is also open source code. Yes. source code. Yes. Yes. Yes, that's how it is. Yes. Yes, that's how it is. looks like: Um, so I looks like: Um, so I looks like: Um, so I previously showed how previously showed how previously showed how we write assessments, or we write assessments, or we write assessments, or one of the ways one of the ways one of the ways writing grades in writing grades in writing grades in BrainTrust is BrainTrust is BrainTrust is using this using this using this evaluation code. Here is mine. evaluation code. Here is mine. evaluation code. Here is mine. assessment for another assessment for another assessment for another project, beyond project, beyond project, beyond which I am taking you out of. which I am taking you out of. which I am taking you out of. The only thing I The only thing I The only thing I I draw attention here, I draw attention here, I draw attention here, which is different and which is different and which is different and which you haven't seen in which you haven't seen in which you haven't seen in previous code, — previous code, — these are parameters. Um, under these are parameters. Um, under these are parameters. Um, under I'm basically a hood I'm basically a hood I'm basically a hood exhibited various exhibited various exhibited various parameters that parameters that parameters that relate to my relate to my relate to my agent system. And in agent system. And in agent system. And in in this case it is, in this case it is, in this case it is, you know, something is enough you know, something is enough you know, something is enough trivial, right?

trivial, right? trivial, right? This is a supervisory agent, This is a supervisory agent, This is a supervisory agent, but if you look but if you look but if you look to the playground itself, um, to the playground itself, um, to the playground itself, um, Come here. Therefore, Come here. Therefore, Come here. Therefore, you have different things, you have different things, you have different things, which can be added to which can be added to which can be added to BrainTrust, right? I think BrainTrust, right? I think BrainTrust, right? I think You know, I don't think so. You know, I don't think so. You know, I don't think so. many people now many people now many people now use this use this use this for tips, except for tips, except for tips, except basic things, but basic things, but basic things, but for remote for remote for remote I can evaluate I can evaluate I can evaluate start the server start the server start the server development, so I can development, so I can development, so I can to be local, I can to be local, I can to be local, I can write eval write eval write eval filename--dev. He filename--dev. He filename--dev. He starts the server and starts the server and starts the server and automatically searches automatically searches automatically searches it on localhost 8300. You it on localhost 8300. You it on localhost 8300. You you can also you can also you can also expose this server expose this server expose this server in your own in your own in your own infrastructure. You infrastructure. You infrastructure. You as if they created this as if they created this as if they created this assessment, they gave it, assessment, they gave it, assessment, they gave it, and also exhibited and also exhibited and also exhibited different parameters. So different parameters. So different parameters. So in this case I in this case I in this case I exposed the system exposed the system exposed the system hint and model hint and model hint and model for this system for this system for this system tips. And if I tips. And if I tips. And if I I'll scroll a little further, I'll scroll a little further, I'll scroll a little further, there is also a hint there there is also a hint there there is also a hint there and a model for the agent and a model for the agent and a model for the agent in mathematics, in mathematics, in mathematics, hint and model hint and model hint and model for an agent with for an agent with for an agent with research. But I research. But I research. But I I can interact with I can interact with I can interact with all this all this all this complexity, with all complexity, with all complexity, with all tools that tools that tools that has each of these has each of these has each of these agents, directly from agents, directly from agents, directly from playground. I can playground. I can playground. I can just press " just press " "run", he "run", he "run", he the whole process will pass the whole process will pass the whole process will pass the code will be executed the code will be executed the code will be executed on the server, and on the server, and on the server, and assessment results assessment results assessment results will be broadcast will be broadcast will be broadcast back to the playground.

back to the playground. Yes, BrainTrust initiates POST- request to the end request to the end request to the end the score you get the score you get the score you get exhibited on this exhibited on this exhibited on this server. So here it is, I server. So here it is, I server. So here it is, I I can show you. I can show you. I can show you. another example. Where are you? another example. Where are you? another example. Where are you? you can use you can use you can use create app function or create app function or create app function or you can put things away, you can put things away, you can put things away, which you will need which you will need which you will need to start the server. to start the server. But this server But this server remote assessment remote assessment remote assessment now deployed on now deployed on now deployed on modal. I use modal. I use modal. I use This is a create app. He This is a create app. He This is a create app. He gives all the grades, gives all the grades, gives all the grades, at least the way I am at least the way I am at least the way I am set up, in my set up, in my set up, in my evals directories. AND evals directories. AND evals directories. AND they are now available they are now available they are now available on the site for on the site for on the site for users to users to users to they could them they could them they could them use. One of use. One of use. One of things that we are doing now things that we are doing now things that we are doing now we do it internally, — we do it internally, — this is what we really this is what we really this is what we really in many ways in many ways in many ways we are guided by interaction we are guided by interaction we are guided by interaction with our clients.

with our clients. I think I I think I mentioned this on mentioned this on mentioned this on beginning, what now beginning, what now beginning, what now there are a lot of people there are a lot of people there are a lot of people are thinking about are thinking about are thinking about how effective how effective how effective they work with they work with they work with their agents for their agents for their agents for coding. Part I coding. Part I coding. Part I this—this is actually this—this is actually this—this is actually carrying out carrying out carrying out skills assessment, skills assessment, skills assessment, what are they what are they what are they are used. It are used. It are used. It understanding the sessions that understanding the sessions that understanding the sessions that they have with cloudy they have with cloudy they have with cloudy code, codex, etc. code, codex, etc. I say this because we I say this because we we put in a lot of effort we put in a lot of effort we put in a lot of effort behind-the-scenes efforts behind-the-scenes efforts behind-the-scenes efforts to create different to create different to create different primitives and primitives and primitives and mechanisms that our mechanisms that our mechanisms that our customers can customers can customers can connect or connect or connect or to use to to use to to use to begin to understand such begin to understand such begin to understand such things. Yes, what you things. Yes, what you things. Yes, what you you only see two you only see two you only see two commits and one asterisk commits and one asterisk , does not indicate that , does not indicate that , does not indicate that what do we think it is what do we think it is what do we think it is very important very important very important a problem for a problem for a problem for engineers today. engineers today. Yes. Steeply. Perfectly. Yes. Steeply. Perfectly. Thank you all. Thank you for that.

Thank you all. Thank you for that. Thank you all. Thank you for that. came.