The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

Let's start. My name is Let's start. My name is Yoni Michael. I am one of Yoni Michael. I am one of Yoni Michael. I am one of co-founders co-founders co-founders a startup called a startup called a startup called Typedef. Together with our Typedef. Together with our Typedef. Together with our lead engineer lead engineer lead engineer by Brandon Calendar by Brandon Calendar by Brandon Calendar we will spend this we will spend this we will spend this workshop. We have workshop. We have workshop. We have two-hour slot. We two-hour slot. We two-hour slot. We we will try to do we will try to do we will try to do its possible its possible its possible more interesting and more interesting and more interesting and more fun. I know that more fun. I know that more fun. I know that now it is very now it is very now it is very tempting. You all tempting. You all tempting. You all got laptops, and got laptops, and got laptops, and Germany loses 1-0 Germany loses 1-0 Germany loses 1-0 at the World Championship. at the World Championship. at the World Championship. But we promise But we promise But we promise to do everything possible to do everything possible to do everything possible more interesting that you have more interesting that you have more interesting that you have there was no desire there was no desire there was no desire it is broadcast. Therefore, it is broadcast. Therefore, it is broadcast. Therefore, before we before we before we Let's move on to the interesting one, Let's move on to the interesting one, Let's move on to the interesting one, practical technical practical technical practical technical parts of the workshop. I parts of the workshop. I parts of the workshop. I I would like some I would like some I would like some tell about the circle tell about the circle tell about the circle problems that we problems that we problems that we trying to solve trying to solve trying to solve in Typedef. Prepare in Typedef. Prepare in Typedef. Prepare soil and provide soil and provide soil and provide context for context for context for concepts and concepts and concepts and principles that principles that principles that will be later will be later will be later applied to applied to applied to workshop under workshop under workshop under leadership leadership leadership Brandon. Main Brandon. Main Brandon. Main the idea of ​​this conversation is the idea of ​​this conversation is the idea of ​​this conversation is Typedef background. We Typedef background. We Typedef background. We create a level create a level create a level trust for agents trust for agents trust for agents data so that they can data so that they can data so that they can access and access and access and to perform complex tasks to perform complex tasks to perform complex tasks task on task on task on platforms and in platforms and in platforms and in data systems. Name data systems. Name data systems. Name my report—" my report—" context level context level context level data", this is the key data", this is the key data", this is the key element element element infrastructure, which infrastructure, which infrastructure, which we built for we built for we built for the fulfillment of this mission.

the fulfillment of this mission. In many ways this In many ways this it follows from the fact that, it follows from the fact that, it follows from the fact that, maybe you don't maybe you don't maybe you don't so deep so deep so deep into the data realm. But into the data realm. But into the data realm. But when you try when you try when you try attract agents- attract agents- coders or agents- coders or agents- developers developers developers software software software provision for provision for provision for working with data, working with data, working with data, the environment becomes the environment becomes the environment becomes much more difficult much more difficult much more difficult and more nuanced, and and more nuanced, and and more nuanced, and need to consider need to consider need to consider many things, building many things, building many things, building the correct frame and the correct frame and the correct frame and doing doing doing contextual contextual contextual engineering, so that these engineering, so that these engineering, so that these the agents were really the agents were really the agents were really effective. About this effective. About this effective. About this We'll talk. We'll talk. We'll talk. The main conclusion in The main conclusion in The main conclusion in because it is not a model because it is not a model because it is not a model a bottleneck. a bottleneck. a bottleneck. The model can The model can The model can continue continue continue to increase their to increase their to increase their opportunities, but opportunities, but opportunities, but no context, no context, no context, who makes these who makes these who makes these agents effective agents effective —this is the most important —this is the most important —this is the most important part. So, this is it, part. So, this is it, part. So, this is it, what do we call what do we call what do we call context level context level context level data. Okay, here it is. data. Okay, here it is. data. Okay, here it is. what are we betting what are we betting what are we betting as a startup called as a startup called as a startup called Typedef. We all spent Typedef. We all spent Typedef. We all spent the last two years in the last two years in the last two years in industry, thinking about industry, thinking about industry, thinking about context engineering context engineering context engineering for coder agents, for coder agents, for coder agents, isn't that right? So, we have isn't that right? So, we have isn't that right? So, we have there is Anthropic, Codecs, Cursor, where you there is Anthropic, Codecs, Cursor, where you there is Anthropic, Codecs, Cursor, where you can you understand how can you understand how can you understand how effectively effectively effectively to structure to structure to structure repositories, repositories, repositories, to search and to search and to search and design design design tools so that these tools so that these tools so that these encoding agents encoding agents encoding agents were really were really were really effective. Where effective. Where effective. Where we haven't yet we haven't yet we haven't yet we reach and where we reach and where we reach and where we begin to see we begin to see we begin to see greater need, this is what greater need, this is what greater need, this is what that it should be the same that it should be the same that it should be the same made for agents, made for agents, made for agents, which can which can which can to access to access to access data and work with data and work with data and work with them. And it is with the data them. And it is with the data them. And it is with the data a situation arises: a situation arises: a situation arises: if the agent is confident if the agent is confident if the agent is confident is wrong, and this is wrong, and this is wrong, and this It happens often, this It happens often, this It happens often, this can cost a lot can cost a lot can cost a lot expensive, isn't it?

expensive, isn't it? expensive, isn't it? Because many of these Because many of these Because many of these metrics and calculations, metrics and calculations, metrics and calculations, which we ask which we ask which we ask perform perform perform analytical agents, analytical agents, analytical agents, fall into fall into fall into presentations for the board presentations for the board presentations for the board directors, or is it directors, or is it directors, or is it metrics in the model, or metrics in the model, or metrics in the model, or conveyors that are quiet conveyors that are quiet conveyors that are quiet are out of order, are out of order, are out of order, having a critical having a critical having a critical important signs, important signs, important signs, metrics and indicators. metrics and indicators. So our main So our main the bet is that the bet is that the bet is that agents will become agents will become agents will become key key key operators of these operators of these operators of these data systems. That data systems. That data systems. That they were really they were really they were really effective, we effective, we effective, we must guarantee, must guarantee, must guarantee, that during execution that during execution that during execution read operations and read operations and read operations and record they operate record they operate record they operate safely and accurately, and safely and accurately, and safely and accurately, and we can trust we can trust we can trust these results in these results in these results in scale. And it works. scale. And it works. scale. And it works. only when you only when you only when you provide the agent provide the agent provide the agent level of reasoning to level of reasoning to level of reasoning to he could understand he could understand he could understand data system from data system from data system from from beginning to end, from beginning to end, from beginning to end, similar to how similar to how similar to how data command- data command- engineers who engineers who engineers who created this system, created this system, created this system, able to think about able to think about able to think about her work. This is the goal. her work. This is the goal. her work. This is the goal. context level context level context level data. So, I will say this. data. So, I will say this. : in the last year and a half : in the last year and a half : in the last year and a half year of work in this year of work in this year of work in this in the field we found three in the field we found three in the field we found three specific industries where specific industries where specific industries where modern agents for modern agents for modern agents for encoding fails, encoding fails, encoding fails, when you try when you try when you try to make them to make them to make them effective in work effective in work effective in work with data. The first is with data. The first is with data. The first is impact analysis. That is, impact analysis. That is, impact analysis. That is, essentially, that essentially, that essentially, that will break if I will break if I will break if I Will I make this change?

Will I make this change? Will I make this change? Next is this Next is this Next is this recommendations regarding recommendations regarding recommendations regarding table join ( table join ( join). Is it safe for me? join). Is it safe for me? join). Is it safe for me? to do this union to do this union to do this union in my data system? in my data system? And generally safe And generally safe code changes. Safe code changes. Safe code changes. Safe logic changes logic changes logic changes transformation. Or transformation. Or transformation. Or does this mean does this mean does this mean editing the same, editing the same, editing the same, what about before? Or what about before? Or what about before? Or stored stored stored the meaning of this the meaning of this the meaning of this metrics, or now metrics, or now metrics, or now metric definition metric definition metric definition will change because of will change because of will change because of that I am making this change in that I am making this change in that I am making this change in data? This is not data? This is not data? This is not secondary tasks, secondary tasks, secondary tasks, isn't that right? It isn't that right? It isn't that right? It everyday tasks everyday tasks everyday tasks and workflows and workflows and workflows teams working with teams working with teams working with data that data that data that are trying are trying are trying understand how to force understand how to force understand how to force corporate data corporate data corporate data work with AI. And if work with AI. And if work with AI. And if your agent cannot your agent cannot your agent cannot to think and to think and to think and to do these to do these to do these task reliably, task reliably, task reliably, it is very difficult for him it is very difficult for him it is very difficult for him to actually trust. to actually trust. to actually trust. Very difficult actually Very difficult actually Very difficult actually to entrust him with access to entrust him with access to entrust him with access to the record, and sometimes to the record, and sometimes to the record, and sometimes even before reading even before reading even before reading data. And this is exactly it. data. And this is exactly it. data. And this is exactly it. important: that they important: that they important: that they became effective became effective became effective main main main our operators our operators our operators data systems, not data systems, not data systems, not just agents for just agents for just agents for questions and questions and questions and answers. So, here it is. answers. So, here it is. answers. So, here it is. gap, main gap, main gap, main the gap that we the gap that we the gap that we were able to determine and were able to determine and were able to determine and really discover really discover really discover thanks to our work.

thanks to our work. thanks to our work. This context, This context, This context, required for data, required for data, required for data, distributed between distributed between distributed between many different many different many different systems, right? systems, right? systems, right? Your storage can Your storage can Your storage can save rows and save rows and save rows and the actual data itself. the actual data itself. the actual data itself. Your code Your code Your code transformations transformations transformations contains the actual contains the actual contains the actual business logic that is business logic that is business logic that is very important. Your BI- very important. Your BI- the tool saves the tool saves the tool saves metrics, measurements and metrics, measurements and metrics, measurements and certain business certain business definition in it. AND definition in it. AND definition in it. AND still have yours still have yours still have yours documentation, channels documentation, channels documentation, channels Slack, discussion threads Slack, discussion threads Slack, discussion threads and "tribal knowledge", and "tribal knowledge", and "tribal knowledge", who often live in who often live in who often live in people's heads. True? people's heads. True? people's heads. True? But the agent actually But the agent actually But the agent actually need something that need something that need something that unites all of this. He unites all of this. He unites all of this. He must understand that must understand that must understand that means a model, as means a model, as means a model, as it is related to it is related to it is related to others that will break others that will break others that will break after some changes and after some changes and after some changes and what operations what operations what operations actually safe, actually safe, actually safe, Yes? For Yes? For Yes? For the next, say, 15 the next, say, 15 the next, say, 15 minutes while I'm talking about it minutes while I'm talking about it minutes while I'm talking about it I will tell you, we I will tell you, we I will tell you, we we will use we will use we will use the term "graininess" the term "graininess" (grain). So, for those who (grain). So, for those who (grain). So, for those who who doesn't work with who doesn't work with who doesn't work with data: graininess data: graininess data: graininess from the team's point of view from the team's point of view from the team's point of view data means that data means that data means that actually means actually means actually means each row of the table.

each row of the table. What actually What actually represents each represents each represents each line? If line? If line? If graininess—this graininess—this graininess—this client per month, and I client per month, and I client per month, and I accidentally change it to accidentally change it to accidentally change it to customer per week, customer per week, customer per week, then each subsequent then each subsequent then each subsequent the amount can be the amount can be the amount can be wrong, even wrong, even wrong, even if SQL compiles if SQL compiles if SQL compiles and everything works correctly and everything works correctly , right? Here's what I have on , right? Here's what I have on , right? Here's what I have on mind when I say mind when I say mind when I say about the change about the change about the change graininess. Let's graininess. Let's graininess. Let's let's do this a little bit let's do this a little bit let's do this a little bit more specific. WITH more specific. WITH more specific. WITH from a reliability perspective, from a reliability perspective, from a reliability perspective, what is it really what is it really what is it really means? If someone means? If someone means? If someone asks the agent why asks the agent why asks the agent why number of rows in number of rows in number of rows in pipeline suddenly pipeline suddenly pipeline suddenly increased by 40% overnight. increased by 40% overnight. increased by 40% overnight. The agent can write The agent can write The agent can write valid SQL, execute valid SQL, execute valid SQL, execute him, build him, build him, build great schedule, but great schedule, but great schedule, but after all he after all he after all he will inform will inform will inform the wrong reason, the wrong reason, the wrong reason, a false analysis of what a false analysis of what a false analysis of what What happened. He will say: What happened. He will say: What happened. He will say: "It seems to have arrived." "It seems to have arrived." "It seems to have arrived." more data." But more data." But more data." But actually happened actually happened actually happened changing the column above changing the column above changing the column above by the flow that by the flow that by the flow that transformed the relationship transformed the relationship transformed the relationship "one-to-one" on " "one-to-one" on " one-to-many", one-to-many", one-to-many", Yes? So, this means, Yes? So, this means, Yes? So, this means, what level what level what level data detailing data detailing data detailing really changed. Uh, really changed. Uh, really changed. Uh, and the agent literally does not and the agent literally does not and the agent literally does not can distinguish can distinguish can distinguish a real jump, a real jump, a real jump, for example, real for example, real for example, real data burst, and data burst, and data burst, and actual level change actual level change actual level change details, details, details, because this level because this level because this level not contained in themselves not contained in themselves not contained in themselves data, right? Um, and data, right? Um, and data, right? Um, and so the agent can so the agent can so the agent can execute the request, execute the request, execute the request, get a few get a few get a few sample lines, sample lines, sample lines, analyze data, analyze data, analyze data, he can do great he can do great he can do great read SQL to read SQL to read SQL to understand the structure, understand the structure, understand the structure, but he didn't know that but he didn't know that but he didn't know that meant one line meant one line meant one line before and now he before and now he before and now he means something completely different, means something completely different, means something completely different, truth? So, exactly truth? So, exactly truth? So, exactly so just so just so just valid SQL valid SQL valid SQL not enough, right?

not enough, right? not enough, right? Yes? Semantics Yes? Semantics Yes? Semantics really, really really, really really, really important when you important when you important when you trying to do trying to do trying to do agent effective. agent effective. So let's suppose if So let's suppose if to think, exist, to think, exist, to think, exist, probably two obvious ones probably two obvious ones probably two obvious ones solutions solutions solutions for such a scenario. for such a scenario. And that is why none of And that is why none of these obvious these obvious these obvious ways actually ways actually ways actually doesn't work, right? Therefore, doesn't work, right? Therefore, doesn't work, right? Therefore, you can provide you can provide you can provide agent code, correct agent code, correct ? But the problem is, ? But the problem is, ? But the problem is, what if you give it to him, what if you give it to him, what if you give it to him, let's say your dbt project let's say your dbt project let's say your dbt project or Airflow, it every time or Airflow, it every time or Airflow, it every time will output context will output context will output context from scratch, right? On from scratch, right? On from scratch, right? On each one each one each one a clue what it means a clue what it means a clue what it means probabilistic approach, probabilistic approach, probabilistic approach, it is limited it is limited it is limited context window, context window, context window, which is not always good which is not always good which is not always good works, and every time works, and every time works, and every time you are starting a new one you are starting a new one you are starting a new one session, you have to session, you have to session, you have to rebuild this rebuild this rebuild this context from scratch. Then context from scratch. Then context from scratch. Then let's say: "Okay, let's say: "Okay, let's say: "Okay, let's try to provide let's try to provide let's try to provide he has access to the database he has access to the database he has access to the database data". So, I data". So, I data". So, I I'm going to connect. I'm going to connect. I'm going to connect. to him his Snowflake to him his Snowflake to him his Snowflake or Databricks MCP. Um, and or Databricks MCP. Um, and or Databricks MCP. Um, and instead he instead he instead he sees lines, maybe sees lines, maybe sees lines, maybe analyze them and analyze them and analyze them and see sample see sample see sample data, but he doesn't have data, but he doesn't have data, but he doesn't have real real real understanding that understanding that understanding that actually means actually means actually means each line. That is each line. That is each line. That is he can read he can read he can read numbers, but not numbers, but not numbers, but not be sure to explain be sure to explain be sure to explain them, right? Therefore, them, right? Therefore, them, right? Therefore, the first situation—he the first situation—he the first situation—he can read the code can read the code can read the code but has no memory, but has no memory, but has no memory, to preserve knowledge to preserve knowledge to preserve knowledge context, on context, on context, on the creation of which you the creation of which you the creation of which you spent a lot spent a lot spent a lot time, exchanging time, exchanging time, exchanging requests. And the other one requests. And the other one requests. And the other one sees the data, but not sees the data, but not sees the data, but not connects them with connects them with connects them with hidden meaning.

hidden meaning. hidden meaning. So none of them So none of them So none of them has a constant pattern has a constant pattern has a constant pattern of how it really is of how it really is of how it really is The system is working. AND The system is working. AND The system is working. AND This is exactly the task we This is exactly the task we This is exactly the task we trying to solve trying to solve . So let's do it. . So let's do it. . So let's do it. step back and for sure step back and for sure step back and for sure Let's define what exactly Let's define what exactly Let's define what exactly missing in context, missing in context, missing in context, which must be provided which must be provided which must be provided to the agent so that he can to the agent so that he can to the agent so that he can to reason effectively to reason effectively to reason effectively about these complex changes about these complex changes about these complex changes in the data. There are four in the data. There are four in the data. There are four facts, let's say, facts, let's say, facts, let's say, which determine whether there is which determine whether there is which determine whether there is the answer is correct the answer is correct , and none of them , and none of them , and none of them not really not really not really recorded nowhere, where recorded nowhere, where recorded nowhere, where the agent could him the agent could him the agent could him read or read or read or to withdraw. So, first— to withdraw. So, first— this is what it really is this is what it really is this is what it really is represents one represents one represents one row in the table. This is it. row in the table. This is it. what we talked about— what we talked about— granularity. granularity. granularity. Next—is there Next—is there Next—is there safe association safe association safe association or not. And then—what? or not. And then—what? or not. And then—what? actually means actually means actually means indicator and what goes with it indicator and what goes with it indicator and what goes with it related, right? related, right? related, right? Dependencies are very Dependencies are very Dependencies are very important in systems important in systems important in systems data. So, the agent can data. So, the agent can data. So, the agent can easy to read your easy to read your easy to read your SQL, but it can't SQL, but it can't SQL, but it can't it's easy to see these it's easy to see these it's easy to see these four things during four things during four things during request execution, request execution, request execution, when is this when is this when is this the most important moment the most important moment the most important moment to receive to receive to receive precise context. precise context. precise context. Therefore, our goal Therefore, our goal Therefore, our goal consisted in the fact that consisted in the fact that consisted in the fact that to make them to make them to make them calculable and calculable and calculable and available for available for available for requests that the agent requests that the agent requests that the agent could effectively could effectively could effectively get the one you need get the one you need get the one you need the context is then, the context is then, the context is then, when he him when he him when he him needed, and not needed, and not needed, and not recreate it recreate it recreate it every time. Therefore, every time. Therefore, every time. Therefore, the next few the next few the next few slides about how we slides about how we slides about how we this was implemented. Well, this was implemented. Well, this was implemented. Well, let's take, let's take, let's take, for example, an idea for example, an idea for example, an idea association. This is the second one association. This is the second one association. This is the second one the fact that we are talking about the fact that we are talking about the fact that we are talking about spoke in spoke in spoke in previous slide.

previous slide. previous slide. This is a great example, This is a great example, This is a great example, because he is the brightest because he is the brightest because he is the brightest shows why the most shows why the most shows why the most syntax syntax syntax not enough for not enough for not enough for effective work effective work effective work agent. So, there is one agent. So, there is one agent. So, there is one association, we association, we association, we we call it "equi- we call it "equi- join", which is recorded join", which is recorded join", which is recorded in a way, but in a way, but in a way, but it can have four it can have four it can have four completely different completely different completely different results depending results depending results depending from the data, right? from the data, right? from the data, right? It can save It can save It can save your lines, find your lines, find your lines, find measurement or imperceptibly measurement or imperceptibly measurement or imperceptibly duplicate each duplicate each duplicate each line when transitioning line when transitioning line when transitioning from one-to-one communication from one-to-one communication from one-to-one communication many." The same many." The same many." The same concerns the operator concerns the operator concerns the operator "between", which is often "between", which is often "between", which is often found in SQL. It found in SQL. It found in SQL. It may be safe may be safe may be safe search by time or search by time or search by time or timeline that timeline that timeline that actually expands actually expands actually expands your granularity, your granularity, your granularity, isn't that right? Agent isn't that right? Agent isn't that right? Agent sees this syntax, sees this syntax, sees this syntax, but can't understand but can't understand , which of these , which of these , which of these scenarios will arise in scenarios will arise in scenarios will arise in as a result of it as a result of it as a result of it using. This is not using. This is not using. This is not the problem of the model or its the problem of the model or its the problem of the model or its opportunities, this opportunities, this opportunities, this the problem of context, the problem of context, the problem of context, which we are trying to which we are trying to which we are trying to arm the agent. arm the agent. Fourth fact, one Fourth fact, one one of the most interesting, which one of the most interesting, which one of the most interesting, which I love I love I love to demonstrate, is to demonstrate, is to demonstrate, is understanding of what understanding of what understanding of what models and assets models and assets models and assets data are related between data are related between data are related between yourself. For example, which yourself. For example, which yourself. For example, which models belong to models belong to models belong to income domain, and which income domain, and which —to the product domain.

—to the product domain. —to the product domain. And they often And they often And they often correlate with correlate with correlate with the roles of people who the roles of people who the roles of people who really trying really trying really trying access to access to access to data or data or data or use them use them use them to work with to work with to work with key metrics key metrics key metrics or indicators in these or indicators in these or indicators in these domains. Today domains. Today domains. Today teams often teams often teams often organize data organize data organize data by folder names, and by folder names, and by folder names, and mostly mostly mostly understanding of what understanding of what understanding of what models relate to models relate to models relate to which business which business divisions, divisions, divisions, based on based on based on internal knowledge internal knowledge internal knowledge team. But you team. But you team. But you actually you can actually you can actually you can to find out this, you to find out this, you to find out this, you can you do can you do can you do more and discover it more and discover it more and discover it from the structure of themselves from the structure of themselves from the structure of themselves data. For example, data. For example, data. For example, you can see how you can see how you can see how models actually models actually models actually are interconnected are interconnected , having examined their , having examined their , having examined their linear dependence. linear dependence. linear dependence. Linear dependence Linear dependence Linear dependence at the table level, at at the table level, at at the table level, at column levels, column levels, column levels, consideration of common consideration of common consideration of common keys and shared keys and shared keys and shared names Therefore, the system names Therefore, the system names Therefore, the system automatically groups automatically groups automatically groups them for you in domains. them for you in domains. So when you So when you please contact please contact please contact agent: "Okay, what agent: "Okay, what agent: "Okay, what models belong to models belong to models belong to my domain income my domain income my domain income for the RevOps team? It for the RevOps team? It for the RevOps team? It becomes a query that becomes a query that becomes a query that the agent can perform the agent can perform the agent can perform independently, not independently, not independently, not a question that a question that a question that needs constant needs constant needs constant clarifications in humans, clarifications in humans, clarifications in humans, to understand what to understand what to understand what it belongs to it belongs to it belongs to one domain. It one domain. It one domain. It same principle same principle same principle designing designing designing the context you the context you the context you you see again and again.

you see again and again. you see again and again. You take what is. You take what is. You take what is. internal knowledge, internal knowledge, internal knowledge, which lives in someone which lives in someone which lives in someone heads or in some heads or in some heads or in some documentation, documentation, documentation, trying trying trying calculate this, and then calculate this, and then calculate this, and then make available make available make available for agent requests. for agent requests. So how are you really? So how are you really? Are you creating these facts? Are you creating these facts? Are you creating these facts? The key principle, about The key principle, about The key principle, about which we are many which we are many which we are many we say, consists in we say, consists in we say, consists in in order to always be in order to always be in order to always be maximum maximum maximum deterministic, and LLM deterministic, and LLM deterministic, and LLM use only use only use only when needed when needed when needed eliminate eliminate eliminate ambiguity or ambiguity or ambiguity or to complete the task, in to complete the task, in to complete the task, in which models really which models really which models really good. Yes? You can good. Yes? You can good. Yes? You can calculate calculate calculate granularity and granularity and granularity and linear dependence linear dependence linear dependence directly from directly from directly from SQL itself. So we SQL itself. So we SQL itself. So we we take all the logic we take all the logic we take all the logic transformations and transformations and transformations and we analyze it all the way to AST we analyze it all the way to AST to understand how to understand how to understand how actually actually actually structured data. structured data. Thus, the agent does not Thus, the agent does not deviates from deviates from deviates from actual implementation actual implementation , since the basis , since the basis , since the basis graph and context, graph and context, graph and context, which we provide, which we provide, which we provide, come from come from come from directly from directly from directly from the implementation itself. AND the implementation itself. AND the implementation itself. AND then you can then you can then you can perform the following actions, perform the following actions, perform the following actions, as a security check as a security check as a security check connections on real connections on real connections on real data. So, is there a key?

data. So, is there a key? data. So, is there a key? connection really connection really connection really unique? This is what unique? This is what unique? This is what can be done by can be done by can be done by table profiling table profiling . So you can come in. . So you can come in. . So you can come in. and profile and profile and profile tables that are in tables that are in tables that are in your storage. It your storage. It your storage. It allows the agent not to allows the agent not to allows the agent not to to guess such things, to guess such things, to guess such things, as cardinality. AND as cardinality. AND as cardinality. AND then you can then you can then you can use LLM use LLM use LLM for what should be for what should be for what should be understandable to people, understandable to people, understandable to people, Yes? So, if you Yes? So, if you Yes? So, if you are you trying are you trying are you trying understand the names understand the names understand the names business objects, business objects, business objects, to summarize to summarize to summarize usefulness of models usefulness of models usefulness of models or eliminate or eliminate or eliminate ambiguity, ambiguity, ambiguity, for example, from the first for example, from the first for example, from the first two levels. Right here two levels. Right here two levels. Right here LLM can come into play LLM can come into play . These facts . These facts . These facts are calculated, and then are calculated, and then are calculated, and then check with yours check with yours check with yours data so that the LLM simply data so that the LLM simply data so that the LLM simply called these assets called these assets called these assets data on the graph, which data on the graph, which data on the graph, which we are building. And this we are building. And this we are building. And this allows us allows us allows us make sure that we make sure that we make sure that we not really not really not really deviate from deviate from deviate from basic implementation. basic implementation. basic implementation. And this separation And this separation And this separation important because it important because it important because it maintains the main maintains the main maintains the main facts tied facts tied facts tied to the real to the real to the real implementations, as we implementations, as we implementations, as we already talked about, and to already talked about, and to already talked about, and to the data itself. So, this is the data itself. So, this is the data itself. So, this is everything is fine and good. Where everything is fine and good. Where everything is fine and good. Where all this is preserved all this is preserved ? We call it ? We call it ? We call it context level context level context level data. Ultimately, this data. Ultimately, this data. Ultimately, this Count, right? And he Count, right? And he Count, right? And he one is calculated one is calculated one is calculated once, and then once, and then once, and then updated atomically updated atomically updated atomically at each committee, at each committee, at each committee, True? We are gradually True? We are gradually True? We are gradually let's change this let's change this let's change this context graph, context graph, context graph, because the teams because the teams because the teams data developers data developers data developers they make new ones for him they make new ones for him they make new ones for him commits. So he commits. So he commits. So he always displays always displays always displays the most recent state the most recent state the most recent state your production your production your production corporate data.

corporate data. The reason why this is a graph, The reason why this is a graph, not just a bunch not just a bunch not just a bunch text or set text or set text or set markdown files markdown files , consists of a pattern , consists of a pattern , consists of a pattern access, right? Agent access, right? Agent access, right? Agent can pull it out can pull it out can pull it out that subgraph which that subgraph which that subgraph which he needs, for he needs, for he needs, for one pass. It one pass. It one pass. It allows you to avoid allows you to avoid allows you to avoid overload overload overload prompts or prompts or prompts or context, and this context, and this context, and this scaled to scaled to scaled to over 500 models, over 500 models, over 500 models, because you don't because you don't because you don't needed every time needed every time needed every time reread the project, reread the project, reread the project, to build to build to build context level, or context level, or context level, or wrong? For the last wrong? For the last wrong? For the last a few years ago we became a few years ago we became a few years ago we became witnesses of the revival witnesses of the revival witnesses of the revival graph databases, and graph databases, and graph databases, and I always, even before I always, even before I always, even before appearance of AI, I was surprised, in appearance of AI, I was surprised, in appearance of AI, I was surprised, in why is their main why is their main why is their main utility. But utility. But utility. But it turned out that the models it turned out that the models it turned out that the models and AI is very good and AI is very good and AI is very good know how to receive know how to receive know how to receive context, write context, write context, write Cypher queries, Cypher queries, Cypher queries, to carry them out and to carry them out and to carry them out and to bypass to bypass to bypass column. Nature column. Nature column. Nature data assets very data assets very data assets very dependent on connections. dependent on connections. That's why they are great That's why they are great suitable for suitable for suitable for creation creation creation typed objects, typed objects, typed objects, nodes and edges in nodes and edges in nodes and edges in this graph. Ahem. Oh. You, this graph. Ahem. Oh. You, this graph. Ahem. Oh. You, Guys, don't you see?

Guys, don't you see? Guys, don't you see? slides? A. Can you slides? A. Can you slides? A. Can you Did you turn off the light? Did you turn off the light? Did you turn off the light? I don't know. No? Oh, okay. I don't know. No? Oh, okay. Um, yes. Oh, that's better. Um, yes. Oh, that's better. It's getting a little better. It's getting a little better. It's getting a little better. Let's turn it down Let's turn it down Let's turn it down a little more light. a little more light. Sorry, friends. Me Sorry, friends. Me should have thought should have thought should have thought about the black background and about the black background and about the black background and different color...But I'm with different color...But I'm with different color...But I'm with I will be happy to send this I will be happy to send this I will be happy to send this presentation after presentation after presentation after performance. Good. Please performance. Good. Please performance. Good. Please Sorry about that. Ahem. Sorry about that. Ahem. So, um, yeah, let's So, um, yeah, let's Let's see. So, you Let's see. So, you Let's see. So, you you see, or maybe you see, or maybe you see, or maybe You don't see what I'm thinking. You don't see what I'm thinking. You don't see what I'm thinking. Hmm, but here's a sample Hmm, but here's a sample Hmm, but here's a sample subgraph, and what you subgraph, and what you subgraph, and what you you see, actually you see, actually you see, actually demonstrates that the edges demonstrates that the edges demonstrates that the edges carry a lot carry a lot carry a lot information about information about information about origin, right? origin, right? origin, right? So, the ribs So, the ribs So, the ribs define different define different define different types of relationships between types of relationships between types of relationships between the assets themselves the assets themselves the assets themselves data. We have, data. We have, data. We have, for example, " for example, " materialized materialized materialized through", "has" through", "has" through", "has" semantic semantic semantic definition", and that's all definition", and that's all definition", and that's all nodes that nodes that nodes that represent different represent different represent different types of data assets in types of data assets in types of data assets in the graph itself. So the graph itself. So the graph itself. So there will be a node that there will be a node that there will be a node that represents represents represents semantic semantic semantic representation, representation, representation, for example. And then for example. And then for example. And then there will be a node that there will be a node that there will be a node that represents the physical represents the physical represents the physical the table, which is the table, which is the table, which is actual physical actual physical actual physical reflection reflection reflection logical code itself logical code itself logical code itself tables. I will send this.

tables. I will send this. tables. I will send this. you later, so that you you later, so that you you later, so that you could see could see could see closer, or we closer, or we closer, or we Let's figure out how it is for you. Let's figure out how it is for you. Let's figure out how it is for you. pass. So yes, by pass. So yes, by pass. So yes, by Basically, here's how Basically, here's how Basically, here's how the subgraph will look like. the subgraph will look like. If we return If we return to the scenarios to the scenarios to the scenarios uses that uses that uses that the agent can now the agent can now the agent can now to decide thanks to to decide thanks to to decide thanks to this level this level this level context, then the first context, then the first context, then the first of which is impact analysis, of which is impact analysis, of which is impact analysis, isn't that right? Agent isn't that right? Agent isn't that right? Agent proposes a change, proposes a change, proposes a change, for example, for for example, for for example, for modifications modifications modifications model granularity model granularity , isn't it? Before , isn't it? Before , isn't it? Before do this before do this before do this before it will write a SQL string it will write a SQL string it will write a SQL string for this change, he for this change, he for this change, he can go down can go down can go down graph and ask graph and ask graph and ask every node that is every node that is every node that is downward dependency downward dependency downward dependency from the node that from the node that from the node that he is trying he is trying he is trying change: "Does it violate change: "Does it violate change: "Does it violate this is the granularity that this is the granularity that this is the granularity that you have in this you have in this you have in this knots?" So here he is knots?" So here he is knots?" So here he is may find that may find that may find that monthly ARR indicator monthly ARR indicator monthly ARR indicator will be repeated in will be repeated in will be repeated in weekly lines, if weekly lines, if weekly lines, if I am trying to change. I am trying to change. I am trying to change. monthly monthly monthly granularity on granularity on granularity on weekly, and what is below weekly, and what is below weekly, and what is below the trend is the ARR indicator, the trend is the ARR indicator, the trend is the ARR indicator, which summarizes which summarizes which summarizes different meanings different meanings different meanings monthly, which will result in monthly, which will result in monthly, which will result in to the fact that the metric to the fact that the metric to the fact that the metric will be four times will be four times will be four times greater than the actual greater than the actual greater than the actual value. And he can value. And he can value. And he can to reach everything that to reach everything that to reach everything that is below is below is below current, for example, to current, for example, to current, for example, to information panel information panel information panel council level council level council level directors, because directors, because directors, because context graph context graph context graph covers all your covers all your covers all your data system, from data system, from data system, from output systems to output systems to output systems to BI level and dashboards BI level and dashboards BI level and dashboards monitoring. Therefore, monitoring. Therefore, monitoring. Therefore, the agent has visibility and the agent has visibility and the agent has visibility and can go all over can go all over can go all over chain of life chain of life chain of life data cycle if data cycle if data cycle if can you do this can you do this can you do this express.

express. express. The key point in The key point in The key point in because all this because all this because all this occurs during occurs during occurs during previous previous previous checks, right? checks, right? checks, right? This is not during assembly, This is not during assembly, This is not during assembly, and this is not during and this is not during and this is not during incident at work incident at work incident at work environment. He knows environment. He knows environment. He knows the affected area before the affected area before the affected area before before it starts working. before it starts working. before it starts working. And the second case And the second case And the second case use—this use—this use—this recommendations regarding recommendations regarding recommendations regarding data merging, right? data merging, right? data merging, right? So, this is the second fact. So, this is the second fact. Well, the agent wants Well, the agent wants distribute ARR by distribute ARR by distribute ARR by marketing marketing marketing channels, you know? channels, you know? He does not choose the path. He does not choose the path. just by name just by name just by name columns that can columns that can columns that can understand; he understand; he understand; he checks whether everyone checks whether everyone checks whether everyone the union key is the union key is the union key is unique. It unique. It unique. It is done because we is done because we is done because we we profile ourselves we profile ourselves we profile ourselves tables. And this tables. And this tables. And this checked for checked for checked for based on the data itself, based on the data itself, based on the data itself, Yes? So he proposes Yes? So he proposes Yes? So he proposes safe way safe way safe way association for association for association for doing this doing this doing this task, at the same time task, at the same time task, at the same time indicating those that indicating those that indicating those that will cause will cause will cause branching or 7- branching or 7- multiple increase multiple increase multiple increase lines, or those that lines, or those that lines, or those that suddenly, imperceptibly suddenly, imperceptibly suddenly, imperceptibly will reject 12% of the data will reject 12% of the data will reject 12% of the data further along the chain. We further along the chain. We further along the chain. We we provide recommendations we provide recommendations we provide recommendations with justification and with justification and with justification and evidence that evidence that evidence that are based on themselves are based on themselves are based on themselves data is key data is key data is key moment. Further- moment. Further- secure code changes, secure code changes, secure code changes, Yes? This is the third fact.

Yes? This is the third fact. Yes? This is the third fact. It can often be noticed that It can often be noticed that It can often be noticed that what is the business definition what is the business definition what is the business definition laid down laid down laid down directly in directly in directly in SQL itself, right? So SQL itself, right? So SQL itself, right? So if you ask if you ask if you ask agent, let's say, agent, let's say, agent, let's say, to organize something to organize something to organize something measure or metric, and he measure or metric, and he measure or metric, and he removes the predicate— removes the predicate— syntactically it is syntactically it is syntactically it is It will work, right? SQL It will work, right? SQL It will work, right? SQL compiles, but compiles, but compiles, but actually it can actually it can actually it can to look innocent, to look innocent, to look innocent, although the predicate was although the predicate was although the predicate was an important part an important part an important part the business itself the business itself definition, right? IN definition, right? IN definition, right? IN in this example he in this example he in this example he removes the definition of " removes the definition of " payment", right? He payment", right? He payment", right? He silently redefines silently redefines silently redefines metrics, including metrics, including metrics, including trial outflow trial outflow trial outflow users at all users at all users at all three dashboards, where three dashboards, where three dashboards, where this is displayed this is displayed this is displayed metrics. This metrics. This metrics. This context layer context layer context layer reveals such a change in reveals such a change in reveals such a change in business definitions and business definitions and business definitions and metrics before metrics before metrics before she will be she will be she will be implemented. If you implemented. If you implemented. If you ask the agent these ask the agent these ask the agent these restrictions as restrictions as restrictions as first priority, it's not first priority, it's not first priority, it's not just a comment in YAML- just a comment in YAML- file or something like that file or something like that . This is the main thing, and this is the limit . This is the main thing, and this is the limit . This is the main thing, and this is the limit between syntactic between syntactic between syntactic refactoring and refactoring and refactoring and the change that a person the change that a person the change that a person can trust. That's all.

can trust. That's all. can trust. That's all. great, but is it great, but is it great, but is it does it actually work? You does it actually work? You does it actually work? You need this need this need this to measure. When you to measure. When you to measure. When you create something create something create something agent, you must agent, you must agent, you must be able to measure it. be able to measure it. be able to measure it. So, we practice So, we practice So, we practice development that development that development that is guided is guided is guided evaluation (eval-driven evaluation (eval-driven evaluation (eval-driven development), and it is very development), and it is very development), and it is very an important part of it. an important part of it. an important part of it. We are essentially creating We are essentially creating We are essentially creating a bunch of real ones a bunch of real ones a bunch of real ones production cases, production cases, production cases, tasks and work tasks and work tasks and work engineering processes engineering processes engineering processes data, complexity data, complexity data, complexity which varies from which varies from which varies from easy to easy to easy to extremely complex, and each extremely complex, and each extremely complex, and each of them is launched of them is launched of them is launched several times on several times on several times on isolated clones isolated clones isolated clones real environment real environment real environment data that is very data that is very data that is very important to avoid important to avoid important to avoid information leakage information leakage information leakage between them. In addition, in between them. In addition, in between them. In addition, in we have a system we have a system we have a system success checks success checks , which evaluates , which evaluates , which evaluates actual result. actual result. actual result. We also We also We also we use LLM as we use LLM as we use LLM as a judge to conduct a judge to conduct a judge to conduct impact analysis. But impact analysis. But impact analysis. But key conclusion key conclusion key conclusion here is that although here is that although here is that although the models are the same and the models are the same and the models are the same and the tasks are the same, the tasks are the same, the tasks are the same, the only variable that the only variable that the only variable that is changing, it is is changing, it is is changing, it is availability of access availability of access availability of access agent to level agent to level agent to level data context. Therefore data context. Therefore , you see the essential , you see the essential , you see the essential improvement when there is improvement when there is improvement when there is the right context. the right context. the right context. He is capable He is capable He is capable read information read information read information through these very difficult through these very difficult through these very difficult changes, and the indicator changes, and the indicator changes, and the indicator successful execution successful execution successful execution tasks significantly tasks significantly tasks significantly is growing. He is almost there. is growing. He is almost there. is growing. He is almost there. 35 points higher when 35 points higher when 35 points higher when agent armed agent armed agent armed context level context level context level data. This benchmark, data. This benchmark, data. This benchmark, of course, it will grow of course, it will grow of course, it will grow and develop, and develop, and develop, because we work because we work because we work with increasing with increasing with increasing number of teams to number of teams to number of teams to to discover their to discover their to discover their specific options specific options specific options using. But using. But using. But Is the signal stable?

Is the signal stable? Is the signal stable? wrong? What is better and wrong? What is better and wrong? What is better and richer context you richer context you richer context you you provide to the agent, then you provide to the agent, then you provide to the agent, then more radically more radically more radically the result changes the result changes than just from than just from than just from replacement models or replacement models or replacement models or adding general adding general adding general tools like MCP tools like MCP , to the model itself. So , to the model itself. So , to the model itself. So I am summing up, I am summing up, I am summing up, but before that but before that but before that complete the five-point survey complete the five-point survey complete the five-point survey patterns and transfer patterns and transfer patterns and transfer Brendan's word, Brendan's word, Brendan's word, I note: even if I note: even if I note: even if you don't build like that you don't build like that you don't build like that the system itself, it the system itself, it the system itself, it key findings that key findings that key findings that we learned on we learned on we learned on from my own experience from my own experience from my own experience last year of work. last year of work. last year of work. First rule: always First rule: always First rule: always choose choose choose determinism, determinism, determinism, when possible. when possible. when possible. Never Never Never just try just try just try take all your take all your take all your DBT project or logic DBT project or logic DBT project or logic Airflow and give them away Airflow and give them away Airflow and give them away models so that she models so that she models so that she completed these difficult completed these difficult completed these difficult task. This is not task. This is not task. This is not will work. What do you want? will work. What do you want? will work. What do you want? actually need, actually need, actually need, so get it so get it so get it deep and detailed deep and detailed deep and detailed understanding of logic understanding of logic understanding of logic transformation into transformation into transformation into deterministic deterministic deterministic way. Therefore, the analysis way. Therefore, the analysis way. Therefore, the analysis up to the abstract up to the abstract up to the abstract syntax trees ( syntax trees ( AST) to understand AST) to understand AST) to understand structure, connections and structure, connections and structure, connections and dependencies, there are very dependencies, there are very dependencies, there are very important. Also such important. Also such important. Also such things like ability things like ability things like ability calculate calculate calculate granularity. This is what granularity. This is what granularity. This is what what can you do what can you do what can you do use. We use. We use. We spent a lot spent a lot spent a lot time for time for time for improvement and improvement and improvement and using using using relational algebra relational algebra relational algebra for calculations on for calculations on for calculations on at each stage at each stage at each stage life cycle life cycle life cycle data. And then data. And then data. And then use use use model when it model when it model when it really necessary.

really necessary. really necessary. When it comes down to it When it comes down to it When it comes down to it to generalize to generalize to generalize models, different models, different models, different datasets, their datasets, their datasets, their categorization or categorization or categorization or classifications, models classifications, models classifications, models are dealing with this are dealing with this are dealing with this just great. On- just great. On- second: keep second: keep second: keep evidence, not conclusions. evidence, not conclusions. evidence, not conclusions. It's often better to just It's often better to just It's often better to just store signals and store signals and store signals and allow the agent allow the agent allow the agent to think, because these to think, because these to think, because these the models are very capable and the models are very capable and the models are very capable and smart—they can smart—they can smart—they can to do effectively to do effectively to do effectively conclusions conclusions conclusions directly under directly under directly under request time. Thirdly, request time. Thirdly, request time. Thirdly, graph traversal always graph traversal always graph traversal always better than better than better than overload overload overload prompts or prompts or prompts or context, right? context, right? context, right? It is worth letting him It is worth letting him It is worth letting him to pull out exactly that one to pull out exactly that one to pull out exactly that one subgraph that he subgraph that he subgraph that he needed. Not needed. Not needed. Not download the whole download the whole download the whole repository during repository during repository during execution: agent himself execution: agent himself execution: agent himself will get the one you need will get the one you need will get the one you need subgraph and can subgraph and can subgraph and can to walk along it, to walk along it, to walk along it, to decide to decide to decide task, because task, because task, because all dependencies are there all dependencies are there all dependencies are there clearly displayed. It clearly displayed. It clearly displayed. It will allow to work will allow to work will allow to work with AI more efficiently, with AI more efficiently, with AI more efficiently, because the context because the context because the context will be much smaller will be much smaller will be much smaller every time when every time when every time when agents agents agents will try will try will try to complete the task. to complete the task. Fourth— control the recruitment control the recruitment control the recruitment tools. We tools. We tools. We found out that found out that found out that carefully selected carefully selected carefully selected tool kit tool kit tool kit always surpasses " always surpasses " buffet table with buffet table with buffet table with ready-made MCPs. Even for ready-made MCPs. Even for ready-made MCPs. Even for those MCPs to which we belong those MCPs to which we belong those MCPs to which we belong we give access to our we give access to our we give access to our agent like Snowflake or agent like Snowflake or agent like Snowflake or Databricks, we had to Databricks, we had to Databricks, we had to rewrite a lot, rewrite a lot, rewrite a lot, to reconcile them with to reconcile them with to reconcile them with opportunities and opportunities and opportunities and functional, which functional, which functional, which we strive to achieve.

we strive to achieve. And fifthly—we are already And fifthly—we are already talked about talked about talked about evaluation, evaluation, evaluation, evaluation and again evaluation and again evaluation and again evaluation, right? evaluation, right? evaluation, right? This is the most important thing. You This is the most important thing. You This is the most important thing. You you won't be able to you won't be able to you won't be able to build a reliable build a reliable build a reliable agent, regardless of agent, regardless of agent, regardless of his tasks, if not his tasks, if not his tasks, if not you can measure them. you can measure them. you can measure them. You can't study. You can't study. You can't study. on what is impossible on what is impossible on what is impossible measure. So measure. So measure. So invest in invest in invest in infrastructure infrastructure infrastructure evaluation as much as possible evaluation as much as possible evaluation as much as possible earlier and more often; it earlier and more often; it earlier and more often; it alive, constantly alive, constantly alive, constantly variable part variable part variable part your software, which your software, which your software, which need need need support, support, support, update and update and update and run constantly. run constantly. At the end of this At the end of this parts of the workshop: parts of the workshop: parts of the workshop: so far it hasn't been so far it hasn't been so far it hasn't been quite a workshop, but quite a workshop, but quite a workshop, but rather my flow rather my flow rather my flow thoughts, but now we thoughts, but now we thoughts, but now we let's move on to let's move on to let's move on to the most interesting. We the most interesting. We the most interesting. We found that agents do not found that agents do not found that agents do not are failing in are failing in are failing in data development or data development or data development or data tasks data tasks data tasks because they don't know how because they don't know how because they don't know how read or write SQL. read or write SQL. read or write SQL. They do it, and They do it, and They do it, and are dealing with this are dealing with this are dealing with this very good They very good They very good They are failing because are failing because are failing because do not have or do not do not have or do not do not have or do not understand that understand that understand that actually mean actually mean actually mean data as they are data as they are data as they are circulating or what circulating or what circulating or what breaks when they breaks when they breaks when they are changing, aren't they? are changing, aren't they? are changing, aren't they? So, if you do So, if you do So, if you do this context this context this context complete complete complete component: component: component: remove it, remove it, remove it, to process, to store, to process, to store, to process, to store, and then allow and then allow and then allow agent effectively agent effectively agent effectively to ask him/her to ask him/her to ask him/her through the graph, you through the graph, you through the graph, you you can go from you can go from you can go from an agent who only an agent who only an agent who only reads your code, to reads your code, to reads your code, to such that really such that really such that really ready to lead ready to lead ready to lead your data system.

your data system. your data system. And that's exactly what we And that's exactly what we And that's exactly what we created by created by created by with this with this with this context level context level context level data. OK. Thank you data. OK. Thank you data. OK. Thank you for this small for this small for this small the presentation that we the presentation that we the presentation that we spent here. I want spent here. I want spent here. I want to pass the word to pass the word to pass the word Brandon, who Brandon, who Brandon, who will start practical will start practical will start practical seminar where we seminar where we seminar where we let's apply some of let's apply some of let's apply some of concepts about which concepts about which concepts about which just talked. I am not just talked. I am not just talked. I am not I will answer I will answer I will answer question now, because question now, because question now, because I want to make sure that I want to make sure that I want to make sure that Brandon will have Brandon will have Brandon will have enough time for enough time for enough time for his seminar, but I his seminar, but I his seminar, but I I will be there after. I will be there after. I will be there after. this, therefore this, therefore this, therefore contact me at contact me at contact me at any time. I will any time. I will any time. I will happy to chat happy to chat happy to chat more about this. more about this. more about this. Thank you. Thank you. Good job. Okay. As Good job. Okay. As everyone is busy everyone is busy everyone is busy today? Is everything okay? today? Is everything okay? Fantasy. Good, Fantasy. Good, good. Sorry, one good. Sorry, one good. Sorry, one a minute. OK, a minute. OK, a minute. OK, the topic I'm talking about the topic I'm talking about the topic I'm talking about Today I will tell you, Today I will tell you, Today I will tell you, definitely related definitely related definitely related with what he was talking about with what he was talking about with what he was talking about Yoni. We will try.

Yoni. We will try. Yoni. We will try. combine these combine these combine these concepts in the format, concepts in the format, concepts in the format, which one will we have time for? which one will we have time for? which one will we have time for? to process in the time that to process in the time that to process in the time that remained. So, me remained. So, me remained. So, me My name is Brandon. I am from Type My name is Brandon. I am from Type My name is Brandon. I am from Type Dev, and on this Dev, and on this Dev, and on this conferences you conferences you conferences you you will hear a lot you will hear a lot you will hear a lot reports that reports that reports that agent effectiveness agent effectiveness agent effectiveness depends on depends on depends on given to him given to him given to him context, and what context, and what context, and what building this building this building this context—this is, in fact, context—this is, in fact, , and there is the main job. , and there is the main job. , and there is the main job. So this is, essentially, So this is, essentially, So this is, essentially, practical version practical version practical version the same lesson. the same lesson. the same lesson. Over the next 90 Over the next 90 Over the next 90 minutes we are together minutes we are together minutes we are together let's build a level let's build a level let's build a level context for the agent context for the agent context for the agent programming, programming, programming, true level for true level for true level for real code real code real code bases. Today we bases. Today we bases. Today we we will we will we will use use use Qdrant codebase. AND Qdrant codebase. AND Qdrant codebase. AND then I will show you then I will show you then I will show you the last part: the last part: the last part: even if you even if you even if you create these create these create these amazing amazing amazing tools, agents tools, agents tools, agents mostly them mostly them mostly them will be ignored if will be ignored if will be ignored if you will not be them you will not be them you will not be them help. Real help. Real help. Real the work consists of the work consists of to force them to force them to force them really really really use this use this use this tool. Therefore, tool. Therefore, before we begin, before we begin, if you want if you want if you want monitor the process, monitor the process, monitor the process, this QR code leads to my this QR code leads to my this QR code leads to my GitHub repository, where there is GitHub repository, where there is GitHub repository, where there is all relevant code.

all relevant code. That repository has That repository has multiple Jupyter notebooks multiple Jupyter notebooks , by which you can , by which you can , by which you can to work together with to work together with to work together with by me. Again, no by me. Again, no by me. Again, no worry if you worry if you worry if you don't want don't want don't want set it all up set it all up set it all up independently. For this independently. For this independently. For this requires Docker Compose or requires Docker Compose or requires Docker Compose or Docker to run Neo4j Docker to run Neo4j . And if you want . And if you want . And if you want try LLM parts try LLM parts try LLM parts in real time, in real time, in real time, you can use you can use you can use OpenRouter key. But OpenRouter key. But OpenRouter key. But this is not necessary. this is not necessary. this is not necessary. If you wish, everything If you wish, everything If you wish, everything will work offline. will work offline. will work offline. Or you can just Or you can just Or you can just watch me watch me watch me I'm creating this here. So, I'm creating this here. So, I'm creating this here. So, I'll leave it for a little while longer, I'll leave it for a little while longer, I'll leave it for a little while longer, and then I'll start and then I'll start and then I'll start to tell you to tell you to tell you we managed everything we managed everything we managed everything adjust if adjust if adjust if you want, and we you want, and we you want, and we let's move on to the let's move on to the let's move on to the laptops. OK? laptops. OK? laptops. OK? Everyone made it. Everyone made it. Everyone made it. scan? Ready? scan? Ready? Let's give it another minute. Good. Perfectly. Therefore, Good. Perfectly. Therefore, Let's start with what Let's start with what Let's start with what each agent starts each agent starts each agent starts encoding—with encoding—with encoding—with text search. I text search. I text search. I I will ask about this type.

I will ask about this type. I will ask about this type. or structure under or structure under or structure under called posting list. It called posting list. It called posting list. It real type in Qdrant. real type in Qdrant. real type in Qdrant. So when the agent So when the agent So when the agent works, it launches works, it launches works, it launches Ripgrep, and you see: Ripgrep, and you see: Ripgrep, and you see: great, he found it great, he found it great, he found it two different answers, two different answers, two different answers, which are exact matches which are exact matches . Both of them are the same. . Both of them are the same. . Both of them are the same. posting list. But we see posting list. But we see posting list. But we see a dozen more similar ones a dozen more similar ones a dozen more similar ones options. You know, in options. You know, in options. You know, in we have posting list view, posting we have posting list view, posting we have posting list view, posting list iterator etc. Your list iterator etc. Your list iterator etc. Your the agent will do that the agent will do that the agent will do that exactly. It will execute grep, exactly. It will execute grep, exactly. It will execute grep, will get a bunch will get a bunch will get a bunch results and choose results and choose results and choose one, right? He can one, right? He can one, right? He can choose the right one, and choose the right one, and choose the right one, and could be wrong. could be wrong. could be wrong. He first He first He first will investigate the chosen one will investigate the chosen one will investigate the chosen one version. Perhaps, version. Perhaps, version. Perhaps, he will have to he will have to he will have to to go back, to go back, to go back, roll back to roll back to roll back to next, so that next, so that next, so that to understand what exactly to understand what exactly to understand what exactly you need it, right? But you need it, right? But you need it, right? But he doesn't really have he doesn't really have he doesn't really have opportunities to learn opportunities to learn , which posting list do you need? , which posting list do you need? , which posting list do you need? needed, right? needed, right? needed, right? He...he...he can't. He...he...he can't. He...he...he can't. The text itself has no The text itself has no The text itself has no concept of identity concept of identity concept of identity or which one exactly or which one exactly or which one exactly The posting list is meaningful. The posting list is meaningful. Things that make you Things that make you interested in—which interested in—which interested in—which methods you methods you methods you use where use where use where it is used in it is used in it is used in what a craze what a craze what a craze is located—it is not is located—it is not is located—it is not must be in must be in must be in in the text, it is in the very in the text, it is in the very in the text, it is in the very code structure. We code structure. We code structure. We just saw Yoni just saw Yoni just saw Yoni very similar very similar very similar example for data, and example for data, and example for data, and this is a problem when this is a problem when this is a problem when agent every time agent every time agent every time recreates the whole recreates the whole recreates the whole system again without system again without system again without memory. Because, after- memory. Because, after- first, it will be expensive, first, it will be expensive, first, it will be expensive, truth? He truth? He truth? He will pass through will pass through will pass through everything and will have to every time everything and will have to every time everything and will have to every time reread that one reread that one reread that one the same set of files.

the same set of files. the same set of files. If you put If you put If you put the same to him the same to him the same to him question five times question five times , he every time , he every time , he every time will return and will return and will return and will read all these files will read all these files , right? But the main thing is, , right? But the main thing is, , right? But the main thing is, That's unreliable, right? That's unreliable, right? That's unreliable, right? When he When he When he returns again and again returns again and again returns again and again reads these five reads these five reads these five files, next files, next files, next once he can once he can once he can read them in another read them in another read them in another order, or order, or order, or read first read first read first one file that will give one file that will give one file that will give wrong answer and wrong answer and wrong answer and will make you search in vain will make you search in vain will make you search in vain in another, right? in another, right? in another, right? So, this one So, this one So, this one problem problem problem determinism and determinism and determinism and reduction reduction reduction vulnerabilities we and vulnerabilities we and vulnerabilities we and trying to solve trying to solve when we design when we design when we design context for such context for such context for such agents, you know? agents, you know? agents, you know? The goal is not The goal is not The goal is not necessarily in that, necessarily in that, necessarily in that, to make an agent to make an agent to make an agent much more accurate or much more accurate or much more accurate or more effective, although more effective, although more effective, although sometimes that's important too. sometimes that's important too. sometimes that's important too. The real goal— The real goal— to make it to make it to make it much more much more much more deterministic, or not deterministic, or not deterministic, or not Yes? Because we Yes? Because we Yes? Because we trying to add trying to add trying to add as much as possible as much as possible as much as possible certainty of this certainty of this certainty of this very very very non-deterministic non-deterministic non-deterministic system. After all, without system. After all, without system. After all, without this constant and this constant and this constant and reliable code model, reliable code model, reliable code model, he will always be he will always be he will always be just guessing. AND just guessing. AND just guessing. AND we see that over time we see that over time we see that over time models are becoming more models are becoming more models are becoming more better and better in better and better in better and better in guessing, right? guessing, right? guessing, right? We all see the difference. We all see the difference. We all see the difference. while we while we while we launched Claude code year launched Claude code year launched Claude code year because, and because of how he because, and because of how he because, and because of how he is it working now or not is it working now or not is it working now or not Yes? This is completely different.

Yes? This is completely different. Yes? This is completely different. experience. So, if we experience. So, if we experience. So, if we we want to build we want to build we want to build contextual level contextual level contextual level for the code, we essentially for the code, we essentially for the code, we essentially we must adhere to we must adhere to we must adhere to the same recipe the same recipe the same recipe but on something different but on something different but on something different equal. We will reduce equal. We will reduce equal. We will reduce scale to conduct scale to conduct scale to conduct workshop where you workshop where you workshop where you you will be able to create you will be able to create you will be able to create something that will work something that will work something that will work today, instead of today, instead of today, instead of to try to try to try to recreate our entire to recreate our entire to recreate our entire the product is here. So we the product is here. So we the product is here. So we let's take this one let's take this one let's take this one concept—extraction concept—extraction , save and query , save and query —to start —to start —to start to get information to get information to get information from our codebase. from our codebase. We will pull out We will pull out information about information about information about symbols: each type, symbols: each type, symbols: each type, every trait, every every trait, every every trait, every function with their function with their function with their real real real identity and identity and identity and location. location. location. The relationship between these The relationship between these The relationship between these different symbols different symbols different symbols will become ribs in will become ribs in will become ribs in our graph, which our graph, which our graph, which we can we can we can move. They move. They move. They are calculated one are calculated one are calculated one once, and then once, and then once, and then repeatedly repeatedly repeatedly used for used for used for requests. This is the same one. requests. This is the same one. requests. This is the same one. general approach that general approach that general approach that and when working with and when working with and when working with large large large data architectures, data architectures, data architectures, just for example, just for example, just for example, small enough for small enough for small enough for launch in this room launch in this room . And, in fact, that's exactly what . And, in fact, that's exactly what . And, in fact, that's exactly what What was I just talking about?

What was I just talking about? What was I just talking about? spoke. So we spoke. So we spoke. So we let's run this let's run this let's run this today. It today. It today. It the concept you are talking about the concept you are talking about the concept you are talking about You will hear a lot more. You will hear a lot more. You will hear a lot more. during the conference during the conference . In my opinion, it was . In my opinion, it was . In my opinion, it was very cool I very cool I very cool I prepared a slide about prepared a slide about prepared a slide about three levels of context, three levels of context, three levels of context, and then looked at and then looked at and then looked at schedule and saw that schedule and saw that schedule and saw that tomorrow someone tomorrow someone tomorrow someone will tell about will tell about will tell about four levels four levels four levels context. So I context. So I context. So I I feel that I I feel that I I feel that I a little ahead of schedule, but a little ahead of schedule, but a little ahead of schedule, but we don't care we don't care we don't care let's go through this let's go through this let's go through this today, because I think today, because I think today, because I think this is quite useful this is quite useful this is quite useful for our context. for our context. for our context. So, you can So, you can So, you can consider the context consider the context consider the context encoding agent as encoding agent as encoding agent as a certain hierarchy a certain hierarchy a certain hierarchy levels. This is how I got used to it. levels. This is how I got used to it. levels. This is how I got used to it. think about it. First think about it. First think about it. First level—this is what you level—this is what you level—this is what you just saw. This is grep, just saw. This is grep, just saw. This is grep, this is glob, raw strings, this is glob, raw strings, this is glob, raw strings, no structure, no structure, no structure, True? Second level True? Second level —this is where —this is where —this is where are located are located are located most modern most modern most modern RAG solutions for code, RAG solutions for code, RAG solutions for code, True? You get True? You get True? You get semantic search, semantic search, semantic search, index the code, index the code, index the code, create embeddings, create embeddings, create embeddings, isn't that right? You do isn't that right? You do isn't that right? You do qualitative lexical and qualitative lexical and qualitative lexical and semantic search. You semantic search. You semantic search. You get ranked get ranked get ranked results, right? results, right? results, right? All the places where this thing is All the places where this thing is All the places where this thing is was mentioned. But was mentioned. But was mentioned. But after all, it's still after all, it's still after all, it's still there will be a flat list, there will be a flat list, there will be a flat list, isn't that right? This will be isn't that right? This will be isn't that right? This will be a flat list that a flat list that a flat list that simply provides more simply provides more simply provides more relevant relevant relevant results. This is still results. This is still results. This is still will remain flat will remain flat will remain flat list. He is not list. He is not list. He is not will show you the connections will show you the connections will show you the connections between different between different between different parts of code, right parts of code, right ? So the highest level, ? So the highest level, ? So the highest level, The third level is, on the The third level is, on the The third level is, on the in our opinion, the count, which in our opinion, the count, which in our opinion, the count, which you can move around.

you can move around. you can move around. It's something that connects It's something that connects It's something that connects symbols with real symbols with real symbols with real identity and identity and identity and adds typed adds typed adds typed ribs between them, which ribs between them, which ribs between them, which can you really can you really can you really to walk. I think so. to walk. I think so. to walk. I think so. next part next part next part this workshop we this workshop we this workshop we let's go through these let's go through these let's go through these different levels and different levels and different levels and let's build them. AND let's build them. AND let's build them. AND in the end we in the end we in the end we we will get an agent, we will get an agent, we will get an agent, who can who can who can to answer to answer to answer code question code question code question base using base using base using this graph. So, let's this graph. So, let's this graph. So, let's Let's start planning it. Let's start planning it. During development we During development we let's go through a few let's go through a few let's go through a few steps to reach steps to reach steps to reach this level this level this level understanding. Zero understanding. Zero understanding. Zero a step is a step is a step is syntactic level. syntactic level. syntactic level. First you need First you need First you need parse the code that parse the code that parse the code that you are trying you are trying you are trying understand. understand. The easiest way The easiest way to do this— to do this— to use something to use something to use something language-independent, language-independent, language-independent, for example tree-sitter, this for example tree-sitter, this for example tree-sitter, this will give a basic AST- will give a basic AST- parsing your code parsing your code parsing your code bases and understanding bases and understanding bases and understanding interaction of parts. interaction of parts. However, this still does not give However, this still does not give idea of ​​how idea of ​​how idea of ​​how to solve to solve to solve name conflicts, name conflicts, name conflicts, truth? Symbol ` posting truth? Symbol ` posting truth? Symbol ` posting list` in the file ` posting list.rs` list` in the file ` posting list.rs` list` in the file ` posting list.rs` and the ` posting list` symbol in and the ` posting list` symbol in and the ` posting list` symbol in in the `quant.rs` file—these are two in the `quant.rs` file—these are two in the `quant.rs` file—these are two various lists. On various lists. On various lists. On zero step zero step zero step they will still be they will still be they will still be conflict. Therefore, conflict. Therefore, conflict. Therefore, the name of our game— the name of our game— first level. Therefore, first level. Therefore, first level. Therefore, resolution resolution —this is what grep does —this is what grep does —this is what grep does can't really can't really can't really cope, right? It cope, right? It cope, right? It what transforms these what transforms these what transforms these two ambiguous two ambiguous two ambiguous lists of publications on lists of publications on lists of publications on two clearly two clearly two clearly identified types, identified types, identified types, Yes? So, with Rust it's Yes? So, with Rust it's Yes? So, with Rust it's a bit of a scam. You a bit of a scam. You a bit of a scam. You you get the first one you get the first one you get the first one level almost free, level almost free, level almost free, because the Rust compiler because the Rust compiler because the Rust compiler will do a lot of this will do a lot of this will do a lot of this work for you. And we, by work for you. And we, by work for you. And we, by essentially, essentially, essentially, we will use

we will use we will use rust-doc-json to get rust-doc-json to get rust-doc-json to get all this information all this information all this information practically "out of the box". practically "out of the box". practically "out of the box". Is this a scam? Yes. Is this a scam? Yes. Is this a scam? Yes. I chose Rust intentionally, I chose Rust intentionally, I chose Rust intentionally, because I needed to because I needed to because I needed to create this seminar, create this seminar, create this seminar, and the creation of a whole and the creation of a whole and the creation of a whole parser for another parser for another parser for another language or something language or something language or something something like that—it's a bit something like that—it's a bit something like that—it's a bit more difficult. But more difficult. But more difficult. But at the end I at the end I at the end I I will tell you about I will tell you about I will tell you about more useful case more useful case more useful case using this using this using this for Python, and in for Python, and in for Python, and in repositories are repositories are repositories are example of a notebook to example of a notebook to example of a notebook to you could view you could view you could view him, if not him, if not him, if not You'll have time today. You'll have time today. Next level, about Next level, about what is a notebook—I don't what is a notebook—I don't what is a notebook—I don't I am sure that we I am sure that we I am sure that we to him today to him today to him today let's get there—this is the second one let's get there—this is the second one let's get there—this is the second one level that is level that is level that is true meaning true meaning true meaning these different things that these different things that these different things that we pull out. So, these we pull out. So, these we pull out. So, these the first two levels are purely the first two levels are purely the first two levels are purely deterministic. We deterministic. We deterministic. We we are constantly trying we are constantly trying we are constantly trying base our base our base our understanding as much as possible understanding as much as possible understanding as much as possible more deterministic more deterministic more deterministic approach. So we approach. So we approach. So we let's build these first two let's build these first two let's build these first two equal, equal, equal, using using using tree-sitter and rust-doc-json. AND tree-sitter and rust-doc-json. AND tree-sitter and rust-doc-json. AND then, if we have then, if we have then, if we have time, we will explore how time, we will explore how time, we will explore how build that one build that one build that one second level. second level. second level. For example, how to take For example, how to take For example, how to take this information, submit this information, submit this information, submit her in LLM for her in LLM for her in LLM for classifications and classifications and classifications and generalization, so that generalization, so that generalization, so that to do what LLM is about to do what LLM is about to do what LLM is about good, instead of, good, instead of, good, instead of, to just throw to just throw to just throw all in a row and ask all in a row and ask all in a row and ask to figure it out. Perfectly.

to figure it out. Perfectly. So, now we So, now we let's move on to let's move on to let's move on to opening our opening our opening our notebook 01. And it will be notebook 01. And it will be confusion. Put confusion. Put this is here. Oh, actually, he is this is here. Oh, actually, he is this is here. Oh, actually, he is already here. I just already here. I just already here. I just need to go to need to go to need to go to him. Good. OK, him. Good. OK, him. Good. OK, Is everything okay? Is everything okay? Is everything okay? see? Do I see? Do I see? Do I zoom in from zoom in from zoom in from your side? Perfectly. your side? Perfectly. So, this notebook, which So, this notebook, which he will do—he will he will do—he will he will do—he will it will just pass it will just pass it will just pass through the process through the process through the process construction of these levels construction of these levels construction of these levels different contexts different contexts different contexts for our project for our project for our project quadrant. We will not quadrant. We will not quadrant. We will not index all index all index all codebase today.

codebase today. codebase today. She is a little She is a little She is a little too big. We too big. We too big. We consider a sample of, consider a sample of, consider a sample of, it seems like five it seems like five it seems like five packages (crates). But this packages (crates). But this packages (crates). But this gives us a good gives us a good gives us a good code body for code body for code body for Check here. Therefore, Check here. Therefore, Check here. Therefore, we can start we can start we can start launching these cells. launching these cells. launching these cells. To begin with, we simply To begin with, we simply To begin with, we simply let's do a bunch let's do a bunch let's do a bunch settings. And these settings. And these settings. And these settings are not settings are not settings are not ideal. We can ideal. We can ideal. We can start considering, um start considering, um , yes. Uh-huh. It's on GitHub. , yes. Uh-huh. It's on GitHub. , yes. Uh-huh. It's on GitHub. Yes, I can do it again. Yes, I can do it again. Yes, I can do it again. post a link post a link post a link on this. Quickly. on this. Quickly. I'll open it. Therefore, I'll open it. Therefore, link here. If link here. If link here. If you don't want you don't want you don't want work locally, there is work locally, there is work locally, there is link to Colab in link to Colab in link to Colab in each of these points each of these points . Yes. Regarding Colab for . Yes. Regarding Colab for . Yes. Regarding Colab for notepad if you notepad if you notepad if you do you want do you want do you want use graph use graph , you will need Neo4j , you will need Neo4j , you will need Neo4j or login, but this or login, but this or login, but this special requirement special requirement special requirement for him. Okay. As for him. Okay. As for him. Okay. As we see, there are many here we see, there are many here we see, there are many here additional code for additional code for additional code for doing everything doing everything doing everything analysis inside analysis inside analysis inside Rust himself. I am not Rust himself. I am not Rust himself. I am not I will definitely be there.

I will definitely be there. I will definitely be there. consider this consider this consider this today, but all this today, but all this today, but all this available in available in available in repositories for repositories for repositories for your acquaintance. your acquaintance. your acquaintance. Today we have Today we have Today we have magic function—extract magic function—extract magic function—extract Rust crate that will pull Rust crate that will pull Rust crate that will pull information, information, information, by running Tree-sitter on by running Tree-sitter on by running Tree-sitter on the entire Rust codebase. the entire Rust codebase. And one more function And one more function will do the same, but will do the same, but will do the same, but for information from Rustdoc. for information from Rustdoc. So, here we see, So, here we see, that of the first few that of the first few that of the first few We pulled out the packages We pulled out the packages We pulled out the packages a bunch of characters through a bunch of characters through a bunch of characters through Tree-sitter, as well as Tree-sitter, as well as Tree-sitter, as well as solved versions of these solved versions of these solved versions of these symbols via Rustdoc. symbols via Rustdoc. Now we can Now we can run the code that run the code that run the code that will help you understand, will help you understand, will help you understand, why do we distinguish why do we distinguish why do we distinguish level one and level level one and level level one and level zero. When we zero. When we zero. When we we ask about something we ask about something we ask about something like a "posting list" on like a "posting list" on like a "posting list" on zero level, we zero level, we zero level, we we see this we see this we see this exported version " exported version " posting list". But we don't posting list". But we don't posting list". But we don't we know where she is from we know where she is from we know where she is from actually comes from. actually comes from. actually comes from. We can't We can't We can't trace it back trace it back trace it back to the actual location, to the actual location, to the actual location, where it was defined. where it was defined. where it was defined. So that's exactly why So that's exactly why So that's exactly why need the first one need the first one need the first one level, right? He level, right? He level, right? He will be able to say: this is " will be able to say: this is " posting list", his posting list", his posting list", his canonical canonical canonical location here, in " location here, in " posting list", and this is actual posting list", and this is actual posting list", and this is actual the place is behind the place is behind the place is behind this way and on this this way and on this this way and on this line, right? It line, right? It line, right? It will allow us to find out, will allow us to find out, will allow us to find out, when elements when elements when elements re-exported for re-exported for re-exported for package boundaries, package boundaries, package boundaries, connect them with connect them with connect them with original package original package , and the agent will be able to pass , and the agent will be able to pass , and the agent will be able to pass on all connections between on all connections between on all connections between them, as soon as we them, as soon as we them, as soon as we Let's build the graph itself.

Let's build the graph itself. So now we are very So now we are very let's build it quickly let's build it quickly let's build it quickly second level, about second level, about second level, about which I mentioned which I mentioned which I mentioned before, where we have before, where we have before, where we have a certain index where we a certain index where we a certain index where we we take all the symbols, and we take all the symbols, and we take all the symbols, and then we launch then we launch then we launch a function that simply a function that simply a function that simply performs performs performs full-text search full-text search , right? So, we do , right? So, we do , right? So, we do search and we encounter search and we encounter search and we encounter the same problem that the same problem that the same problem that seen in grep, right? We seen in grep, right? We seen in grep, right? We we can search the list we can search the list we can search the list publications and publications and publications and to receive very to receive very to receive very useful information. useful information. useful information. We see that this We see that this We see that this re-export, this re-export, this re-export, this structure, this too structure, this too structure, this too structure, but that's all structure, but that's all structure, but that's all it won't show us anyway it won't show us anyway it won't show us anyway connections between these connections between these connections between these various things, various things, various things, truth? The agent can truth? The agent can truth? The agent can to conclude and, to conclude and, to conclude and, you know, guess: you know, guess: you know, guess: "Oh, I guess if "Oh, I guess if "Oh, I guess if list of publications— list of publications— this is a structure, then he, this is a structure, then he, this is a structure, then he, probably again probably again probably again exported axis exported axis exported axis here, therefore, this is probably here, therefore, this is probably here, therefore, this is probably its basic its basic its basic copy". But you copy". But you copy". But you don't you want him to don't you want him to don't you want him to always did this always did this always did this logical leap, because logical leap, because logical leap, because necessarily necessarily necessarily the one will be found the one will be found the one will be found once when he once when he once when he will be mistaken. And then will be mistaken. And then will be mistaken. And then he will be completely he will be completely he will be completely on the other side of this on the other side of this on the other side of this tracing. So our tracing. So our tracing. So our the next step is us the next step is us the next step is us we start building we start building we start building the count himself. We take the count himself. We take the count himself. We take all these different symbols all these different symbols all these different symbols and the information that and the information that and the information that just pulled out, and just pulled out, and just pulled out, and let's turn this into let's turn this into let's turn this into a set of nodes and edges, a set of nodes and edges, a set of nodes and edges, which we can fill which we can fill which we can fill in our Neo4j graph. AND in our Neo4j graph. AND in our Neo4j graph. AND then we take and then we take and then we take and we're loading all this we're loading all this we're loading all this within. So, when within. So, when within. So, when it's already in the graph, we it's already in the graph, we it's already in the graph, we we can even go into we can even go into we can even go into Neo4j. We can Neo4j. We can Neo4j. We can to see beautiful

to see beautiful to see beautiful little count from little count from little count from links to all links to all links to all different parts of our different parts of our different parts of our code base. Therefore, code base. Therefore, code base. Therefore, now that we have now that we have now that we have such a graph, we can such a graph, we can such a graph, we can create tools create tools , which are specially , which are specially , which are specially use it use it use it opportunities to opportunities to opportunities to provide to our agent provide to our agent provide to our agent access to what access to what access to what will make it easier for him will make it easier for him will make it easier for him work. So if we work. So if we work. So if we we create a tool we create a tool we create a tool and we want to say: " and we want to say: " Okay, I want to. Okay, I want to. Okay, I want to. look at look at look at function and see, function and see, function and see, what will change, what what will change, what what will change, what will break if I will break if I will break if I I'll change this function, okay? I'll change this function, okay? » So we can » So we can » So we can create function " create function " radius of damage", where radius of damage", where radius of damage", where we say, "Okay, you we say, "Okay, you we say, "Okay, you you pass on you pass on you pass on qualified name qualified name qualified name something, canonical something, canonical something, canonical name". He will come. name". He will come. name". He will come. inside and will try inside and will try inside and will try to figure out, to figure out, to figure out, to walk the count to walk the count to walk the count through all defined through all defined through all defined us ribs, to be sure us ribs, to be sure us ribs, to be sure understand that understand that understand that depends on this depends on this depends on this element and what element and what element and what will break if it will break if it will break if it change, right? So, in change, right? So, in change, right? So, in in this sense you can in this sense you can in this sense you can ask an agent ask an agent ask an agent respond to respond to respond to question for which question for which question for which would have been needed earlier would have been needed earlier would have been needed earlier several search engines several search engines several search engines requests from various requests from various requests from various files, and, theoretically, files, and, theoretically, files, and, theoretically, the agent can the agent can the agent can answer him/her answer him/her answer him/her at once. Therefore, at once. Therefore, at once. Therefore, of course, that's not of course, that's not of course, that's not it worked, because why not it worked, because why not it worked, because why not Was it supposed to work?

Was it supposed to work? Live demonstrations, Live demonstrations, Ladies and gentlemen. Ladies and gentlemen. OK. So, you OK. So, you you see here: when we you see here: when we you see here: when we we say we are going we say we are going we say we are going transmit radius transmit radius transmit radius damage to this damage to this damage to this structure, we can structure, we can structure, we can say: "Okay, cool" say: "Okay, cool" ". This is related here ". This is related here ". This is related here because of...It because of...It because of...It returns with this. returns with this. returns with this. There is a method that does this There is a method that does this There is a method that does this does. It does. It does. It returns here. returns here. returns here. So, we see all So, we see all So, we see all connections that we can connections that we can connections that we can install, simple install, simple install, simple having content having content having content inside the graph. Therefore inside the graph. Therefore finally, we can finally, we can finally, we can solve our solve our solve our the problem of collisions. the problem of collisions. the problem of collisions. Now, if we Now, if we Now, if we let's say we have let's say we have let's say we have two different lists two different lists two different lists newsletters and we want newsletters and we want newsletters and we want see the difference between see the difference between see the difference between them, we can them, we can them, we can call our new call our new call our new radius tools radius tools radius tools "defeat" for this "defeat" for this "defeat" for this list and for that list and for that list and for that list, and we get list, and we get list, and we get another set another set another set results. Change results. Change results. Change one list one list one list mailing list will not affect mailing list will not affect mailing list will not affect the same level of change the same level of change , as well as changing another , as well as changing another , as well as changing another list. So, although these list. So, although these list. So, although these things have the same things have the same things have the same name, their actual name, their actual name, their actual canonical definition canonical definition canonical definition allows us more allows us more allows us more to say with confidence that to say with confidence that to say with confidence that will change if you do it will change if you do it will change if you do it break, instead break, instead break, instead in order to simply in order to simply in order to simply search all files search all files search all files and try to guess.

and try to guess. So now we can So now we can take what take what take what built, and built, and built, and create a small create a small create a small agent with limited agent with limited agent with limited functions to functions to functions to use this use this use this new tools new tools . Now that we have . Now that we have . Now that we have this tool " this tool " radius of destruction, we radius of destruction, we radius of destruction, we we can quickly we can quickly we can quickly connect connect connect little agent. little agent. little agent. This agent is very This agent is very This agent is very simple. It's simple. simple. It's simple. simple. It's simple. pedantic agent, pedantic agent, pedantic agent, which is configured for which is configured for which is configured for using the system using the system using the system hints: "you have hints: "you have hints: "you have ability to launch ability to launch ability to launch these different tools these different tools ". We wrote for ". We wrote for ". We wrote for it is a few very it is a few very it is a few very basic tools basic tools so that he could so that he could so that he could use " use " radius of damage", radius of damage", radius of damage", implement methods, implement methods, implement methods, working with working with working with simple requests simple requests simple requests Cypher to receive Cypher to receive Cypher to receive information. Therefore, information. Therefore, information. Therefore, when you have this when you have this when you have this information in the graph, information in the graph, information in the graph, writing queries writing queries writing queries for its extraction becomes for its extraction becomes for its extraction becomes quite simple quite simple quite simple task. Um, if you task. Um, if you task. Um, if you can you express can you express can you express the idea you want the idea you want the idea you want to implement, then to implement, then to implement, then quite easy quite easy quite easy write Cypher or write Cypher or write Cypher or appropriate language for appropriate language for appropriate language for any graph, any graph, any graph, which one are you which one are you which one are you use to use to use to extract this data. AND extract this data. AND extract this data. AND this is a sign that you this is a sign that you this is a sign that you created the correct created the correct created the correct scheme for this scheme for this scheme for this task. So, if task. So, if task. So, if you will run this you will run this you will run this agent, he will spend agent, he will spend agent, he will spend a few steps to a few steps to a few steps to call tools, call tools, call tools, trying trying trying to understand to understand to understand situations. And then he situations. And then he situations. And then he will try to answer will try to answer will try to answer to our question.

to our question. to our question. So, if you are now So, if you are now So, if you are now you say: "What you say: "What you say: "What will break if I will break if I will break if I change the posting list? Names change the posting list? Names change the posting list? Names every dependent every dependent every dependent character with string character with string character with string file," he said exactly file," he said exactly file," he said exactly will tell you exactly what will tell you exactly what will tell you exactly what will break, will break, will break, based on analysis based on analysis based on analysis radius of damage. AND radius of damage. AND radius of damage. AND we see here what we see here what we see here what tools tools tools are called. He are called. He are called. He started with a search, started with a search, started with a search, performed several performed several performed several requests before requests before requests before cause radius cause radius cause radius damage, and then damage, and then damage, and then made another call made another call made another call radius of damage. radius of damage. radius of damage. So, we see that So, we see that So, we see that if you have an agent, if you have an agent, if you have an agent, quite skilled at quite skilled at quite skilled at use use use tools, he tools, he tools, he can apply can apply can apply these new tools for these new tools for these new tools for improving understanding improving understanding improving understanding code base. Now code base. Now code base. Now this part is more this part is more this part is more interactive, where you interactive, where you interactive, where you you can create any you can create any which instrument, which which instrument, which which instrument, which if you wish. Hmm, it works. if you wish. Hmm, it works. if you wish. Hmm, it works. it's something like this: we it's something like this: we it's something like this: we we can write we can write we can write tool that tool that tool that performs arbitrary performs arbitrary performs arbitrary Cypher code. So here we are Cypher code. So here we are Cypher code. So here we are write a Cypher query, write a Cypher query, write a Cypher query, who says: "Find who says: "Find who says: "Find I have all the symbols in I have all the symbols in I have all the symbols in this file path this file path this file path and just take them out." and just take them out." and just take them out." This tool This tool This tool will do just that. He will do just that. He will do just that. He will find all characters in will find all characters in will find all characters in this path to the file and this path to the file and this path to the file and will bring them out for me. Ahem will bring them out for me. Ahem . Yes. So, if I . Yes. So, if I . Yes. So, if I wanted to write wanted to write wanted to write new tool, new tool, new tool, I don't even I don't even I don't even would definitely be would definitely be would definitely be know Cypher, right?

know Cypher, right? know Cypher, right? Um, I could provide Um, I could provide Um, I could provide agent access to agent access to agent access to system prompt, system prompt, system prompt, which contains a diagram, which contains a diagram, which contains a diagram, right? Which says right? Which says right? Which says to the agent: "Here are the different to the agent: "Here are the different to the agent: "Here are the different nodes, here are different nodes, here are different nodes, here are different ribs", and I can ribs", and I can ribs", and I can write your requests write your requests write your requests natural language. natural language. For example, I want For example, I want to see each to see each to see each structure that structure that structure that implements the trait, together implements the trait, together implements the trait, together with the trait itself, or with the trait itself, or with the trait itself, or wrong? And if I wrong? And if I wrong? And if I I'll start this, I'll start this, I'll start this, It will take a second, It will take a second, It will take a second, because the agent because the agent because the agent will try will try will try to figure it out, and he does it to figure it out, and he does it to figure it out, and he does it does. He really is does. He really is does. He really is generates this Cypher request generates this Cypher request , which can perform, , which can perform, , which can perform, to pull this out to pull this out to pull this out information from the graph. information from the graph. We are not today. We are not today. let's get to the second one let's get to the second one let's get to the second one level, but we can level, but we can level, but we can a little bit with him a little bit with him a little bit with him to get acquainted. Therefore, to get acquainted. Therefore, to get acquainted. Therefore, So far everything has been great. So far everything has been great. So far everything has been great. deterministic. You deterministic. You deterministic. You get yours get yours get yours structure and its structure and its structure and its resolution. resolution. resolution. That third level, That third level, That third level, which I was talking about is which I was talking about is which I was talking about is semantic proper semantic proper semantic proper value. You can value. You can value. You can with LLM with LLM with LLM summarize these summarize these summarize these symbols, embed them symbols, embed them , and Type Duck has , and Type Duck has , and Type Duck has library with library with library with open source under open source under open source under called Fenwick. It called Fenwick. It called Fenwick. It library library library dataframes for dataframes for dataframes for combination combination combination traditional operations traditional operations traditional operations with data and LLM. Here we are with data and LLM. Here we are with data and LLM. Here we are quickly set up quickly set up quickly set up Fenwick, indicating that he Fenwick, indicating that he Fenwick, indicating that he will interact with Open will interact with Open will interact with Open Router. We collect Router. We collect Router. We collect certain information. We certain information. We certain information. We let's try let's try let's try classify this classify this classify this specific Rust code, specific Rust code, specific Rust code, taking his type, taking his type, taking his type, full name and string full name and string full name and string documentation. Then documentation. Then documentation. Then let's ask one let's ask one let's ask one in a concise sentence in a concise sentence in a concise sentence explain why explain why explain why this element in Rust. When this element in Rust. When this element in Rust. When we launch Fenwick we launch Fenwick we launch Fenwick for this case

for this case for this case data, then we get data, then we get data, then we get generalized generalized generalized answers from LLM. So answers from LLM. So answers from LLM. So in the long term in the long term in the long term perspective idea perspective idea perspective idea is to is to is to add this information add this information add this information to the count, so that she to the count, so that she to the count, so that she was available for was available for was available for agent. It's nice that agent. It's nice that agent. It's nice that this process can be this process can be this process can be check during check during check during generation. If you generation. If you generation. If you You see this and say: " You see this and say: " Hey, that's an explanation. Hey, that's an explanation. Hey, that's an explanation. function is incorrect", function is incorrect", function is incorrect", we can do it easily we can do it easily we can do it easily to fix. If the agent to fix. If the agent to fix. If the agent "on the fly" determines "on the fly" determines "on the fly" determines function incorrectly, function incorrectly, function incorrectly, you won't find out about you won't find out about you won't find out about mistake until he mistake until he mistake until he will not write code that does not will not write code that does not will not write code that does not will work. Perfectly. will work. Perfectly. Yes, of course. It was Yes, of course. It was first notebook. We have first notebook. We have first notebook. We have What else is there to talk about? What else is there to talk about? What else is there to talk about? but I am ready but I am ready but I am ready respond to respond to respond to questions about questions about questions about content, and I quickly content, and I quickly content, and I quickly I'll put it back on the screen. I'll put it back on the screen. . Yes, I see two behind.

. Yes, I see two behind. . Yes, I see two behind. You will now come with You will now come with You will now come with microphone. microphone. microphone. Testing. Testing. Testing. Testing. Testing. Tell me, is the count Tell me, is the count is being restored under is being restored under is being restored under execution time when execution time when execution time when you create these ASTs and you create these ASTs and you create these ASTs and Do you go through them? Do you go through them? Define "during" Define "during" implementation". implementation". implementation". During execution, as- During execution, as- that's when PR comes that's when PR comes that's when PR comes or CI/CD is started, or CI/CD is started, or CI/CD is started, or... or... or... Ideally, although we don't Ideally, although we don't Ideally, although we don't we earn a living we earn a living we earn a living by creating this by creating this by creating this product, it would have product, it would have product, it would have cache and cache and cache and to be kept in to be kept in to be kept in canonical form, like, canonical form, like, canonical form, like, for example, gold for example, gold for example, gold version based on version based on version based on main branches, right? Um, and main branches, right? Um, and main branches, right? Um, and when you do something when you do something when you do something are you correcting? are you correcting? are you correcting? change locally, change locally, change locally, you probably you probably you probably need need need support support support a local copy that would a local copy that would a local copy that would ideally ideally ideally was updated was updated was updated gradually. So you don't gradually. So you don't gradually. So you don't would have to do everything would have to do everything would have to do everything immediately. immediately. immediately. Got it. Got it. Yes, I think mine Yes, I think mine The question is similar. How are you? The question is similar. How are you? -e think about -e think about -e think about support cost support cost support cost of these structures in of these structures in of these structures in up-to-date up-to-date up-to-date compared to, well, any- compared to, well, any- what increase what increase what increase productivity or productivity or productivity or accuracy. Um, because accuracy. Um, because accuracy. Um, because at least for me, at least for me, at least for me, you know, I you know, I you know, I I use AI for I use AI for I use AI for making big eh making big eh making big eh changes. So, changes. So, changes. So, Yes.

Yes. hmm, so it's hard for me hmm, so it's hard for me to say, uh to say, uh to say, uh offhand, you know, or offhand, you know, or offhand, you know, or It would really be better, It would really be better, It would really be better, if I had to if I had to if I had to to support these to support these to support these structures. structures. Yes, definitely. I think Yes, definitely. I think I think the key to this is I think the key to this is I think the key to this is because uh a lot of because uh a lot of because uh a lot of these problems can be these problems can be these problems can be simplify and make simplify and make simplify and make incremental, incremental, incremental, because many because many because many the things you do, the things you do, the things you do, not necessarily not necessarily not necessarily reindex, reindex, reindex, if they don't if they don't if they don't are changing, right? AND are changing, right? AND are changing, right? AND when you enter when you enter when you enter large-scale changes in large-scale changes in large-scale changes in your codebase, you your codebase, you your codebase, you you change a lot you change a lot you change a lot files, but there are also files, but there are also files, but there are also many files that are not many files that are not many files that are not are changing, aren't they? are changing, aren't they? are changing, aren't they? So, if we are So, if we are So, if we are to approach wisely to approach wisely to approach wisely incremental incremental incremental reindexing, reindexing, reindexing, tracking only what tracking only what tracking only what what has changed, and what has changed, and what has changed, and reindexing only reindexing only reindexing only this part, then that, this part, then that, this part, then that, what we see so far, what we see so far, what we see so far, indicates that indicates that indicates that compromise actually compromise actually compromise actually worth it, especially worth it, especially worth it, especially when the volume increases when the volume increases when the volume increases context, which context, which context, which need to keep need to keep need to keep indexed. It indexed. It indexed. It actually becomes actually becomes actually becomes more useful because more useful because more useful because cost of execution cost of execution cost of execution this every time in this every time in this every time in real time becomes real time becomes real time becomes even higher. even higher. even higher. Hmm, thank you very much. It Hmm, thank you very much. It Hmm, thank you very much. It it was extraordinary it was extraordinary it was extraordinary useful. Um, I am useful. Um, I am useful. Um, I am I wonder how it is.

I wonder how it is. I wonder how it is. works with real works with real works with real code base on code base on code base on complex language, where, complex language, where, complex language, where, for example, there is for example, there is for example, there is dynamic dynamic dynamic dispatching or, dispatching or, dispatching or, let's say, not all types let's say, not all types let's say, not all types available during available during available during compilation or, compilation or, compilation or, sorry, before sorry, before sorry, before you are just analyzing you are just analyzing you are just analyzing code. So have you tried? code. So have you tried? code. So have you tried? you, what are the limitations? you, what are the limitations? you, what are the limitations? Maybe there are some Maybe there are some Maybe there are some advice? advice? advice? Yes. Um, I have Yes. Um, I have Yes. Um, I have the third notebook, where the third notebook, where the third notebook, where This is exactly what it's about, huh? This is exactly what it's about, huh? -e, which actually shows -e, which actually shows how to do it on how to do it on how to do it on a language like Python, a language like Python, a language like Python, which does not have such which does not have such which does not have such easy access to easy access to easy access to types like, say, Rust. types like, say, Rust. types like, say, Rust. We use, we We use, we We use, we we can we can we can use others use others use others tools. We tools. We tools. We we can we can we can use such use such use such tools like Griff and tools like Griff and tools like Griff and the same Type Sitter. Type Sitter the same Type Sitter. Type Sitter the same Type Sitter. Type Sitter will still give you AST, but will still give you AST, but will still give you AST, but Griff will provide static Griff will provide static Griff will provide static solving some solving some solving some other things, other things, other things, for example, types. You for example, types. You for example, types. You you can also you can also you can also to use something to use something to use something like Pyright for like Pyright for like Pyright for type checks to type checks to type checks to get more get more get more indices. But yes, it is indices. But yes, it is indices. But yes, it is becomes exponential becomes exponential becomes exponential more difficult with different more difficult with different more difficult with different languages. So, about this too languages. So, about this too languages. So, about this too I will tell you a little. I will tell you a little. I will tell you a little. subsequently. But yes, subsequently. But yes, subsequently. But yes, answer to your answer to your answer to your question—yes, I question—yes, I question—yes, I I thought about it. It I thought about it. It I thought about it. It much more complicated much more complicated much more complicated task, so today task, so today task, so today I decided to solve I decided to solve I decided to solve a little simpler. Yes.

a little simpler. Yes. Ugh. I don't think you should. Ugh. I don't think you should. it's worth building. it's worth building. it's worth building. To be clear, To be clear, To be clear, this is just an example, this is just an example, this is just an example, given from given from given from educational purpose. educational purpose. People constantly People constantly create things like this, create things like this, create things like this, like code graphs, and I wouldn't like code graphs, and I wouldn't like code graphs, and I wouldn't tried to build tried to build tried to build this one from scratch just for this one from scratch just for this one from scratch just for code analysis. I don't know. code analysis. I don't know. code analysis. I don't know. does this correspond to does this correspond to does this correspond to your question. And I your question. And I your question. And I I think they are worth it. I think they are worth it. I think they are worth it. build. I just don't build. I just don't build. I just don't sure, is it worth it? sure, is it worth it? sure, is it worth it? everyone to build everyone to build everyone to build your own. I think there is. your own. I think there is. your own. I think there is. many people in this many people in this many people in this areas that are already areas that are already areas that are already are engaged in similar are engaged in similar are engaged in similar analysis at the level analysis at the level analysis at the level code. So I would say, code. So I would say, code. So I would say, that this is probably not that this is probably not that this is probably not worth doing alone worth doing alone worth doing alone to a person with the purpose to a person with the purpose to a person with the purpose product creation.

product creation. product creation. This is probably worth it. This is probably worth it. This is probably worth it. to do if the goal is— to do if the goal is— learn to build learn to build learn to build agents, agents, agents, tools and tools and tools and context graphs. context graphs. context graphs. I think, I think, I think, scalability of this scalability of this scalability of this problems are where problems are where problems are where it is more useful. It it is more useful. It it is more useful. It answers your answers your answers your question? Perfectly. question? Perfectly. question? Perfectly. So, my question is So, my question is So, my question is relates to the level relates to the level relates to the level context of the data, about context of the data, about context of the data, about which you mentioned which you mentioned which you mentioned earlier. I suppose earlier. I suppose earlier. I suppose what is it useful for what is it useful for what is it useful for developers. You developers. You developers. You developing this developing this developing this context level, and context level, and context level, and he helps he helps he helps create better SQL- create better SQL- procedures, etc. Therefore, procedures, etc. Therefore, procedures, etc. Therefore, two questions. First, two questions. First, two questions. First, Did you see this? Did you see this? Did you see this? helped during helped during helped during implementation? For example, implementation? For example, implementation? For example, during the conclusion ( during the conclusion ( inference). This is the first, inference). This is the first, inference). This is the first, and secondly, do you have and secondly, do you have and secondly, do you have results results results benchmarks for benchmarks for benchmarks for demonstrations? demonstrations? Yes, definitely. That, Yes, definitely. That, What did Joni talk about in What did Joni talk about in What did Joni talk about in in our report, we in our report, we in our report, we we have results we have results we have results benchmarks for this. benchmarks for this. We are trying We are trying find find find representative representative representative data architectures that data architectures that data architectures that quite difficult, because quite difficult, because quite difficult, because can be created can be created can be created easy example and easy example and easy example and say, "Oh, my agent say, "Oh, my agent say, "Oh, my agent passes 100% of tests passes 100% of tests passes 100% of tests on my case with 30 on my case with 30 on my case with 30 models". And it's not very models". And it's not very models". And it's not very impressive. Therefore we impressive. Therefore we impressive. Therefore we trying to find trying to find trying to find examples of architectures examples of architectures examples of architectures data with open data with open data with open code that are larger.

code that are larger. code that are larger. After all, we After all, we After all, we stopped at stopped at stopped at use, I don't know use, I don't know , has anyone heard of , has anyone heard of , has anyone heard of Mattermost. This is a company that Mattermost. This is a company that Mattermost. This is a company that deals with instant deals with instant deals with instant response and response and response and chats. Um, they chats. Um, they chats. Um, they actually did actually did actually did your DBT project your DBT project your DBT project fully open, fully open, fully open, there are about 300-350 there are about 300-350 there are about 300-350 models. So we models. So we models. So we we can download we can download we can download him, model it there him, model it there him, model it there data, and then data, and then data, and then run assessments for run assessments for run assessments for given scenarios. given scenarios. given scenarios. Ask him/her Ask him/her Ask him/her to perform specific to perform specific to perform specific task, something there task, something there task, something there break, and then break, and then break, and then ask for correction, ask for correction, ask for correction, or create a set or create a set or create a set new tasks for new tasks for new tasks for development of something development of something development of something new. And we see, new. And we see, new. And we see, that thanks to the creation that thanks to the creation that thanks to the creation our level of assessment our level of assessment our level of assessment and distribution and distribution and distribution criteria—how much criteria—how much criteria—how much that's right, complete that's right, complete that's right, complete Are we effective? Are we effective? Are we effective? we can analyze we can analyze we can analyze results, right? We results, right? We results, right? We we see that from we see that from we see that from appropriate appropriate appropriate our context our context our context agents in general agents in general agents in general work better, and in work better, and in work better, and in certain categories certain categories certain categories even better even better even better cope with those cope with those cope with those the tasks themselves, the tasks themselves, the tasks themselves, than just cloudy than just cloudy than just cloudy writing code. Ahem. writing code. Ahem. What about What about previous previous previous question about time question about time question about time breeding?

breeding? That is, you have That is, you have means the time when the agent means the time when the agent means the time when the agent directly works directly works ? ? ? Yes, exactly when the agent Yes, exactly when the agent Yes, exactly when the agent works. works. works. That's when we That's when we That's when we we get from him we get from him we get from him the greatest benefit, because the greatest benefit, because the greatest benefit, because while working agent while working agent while working agent makes requests to everyone makes requests to everyone makes requests to everyone tools that we tools that we tools that we created, and it is here created, and it is here created, and it is here is revealed is revealed is revealed true value, true value, true value, Yes? If the agent can Yes? If the agent can Yes? If the agent can ask all this under ask all this under ask all this under execution time, it execution time, it execution time, it not necessary every time not necessary every time not necessary every time to list this to list this to list this again, right? again, right? So have you tried? So have you tried? this is with tests like NL to this is with tests like NL to this is with tests like NL to SQL, such as Bard or Spider, or SQL, such as Bard or Spider, or SQL, such as Bard or Spider, or similar? similar? Yes. That is, both yes and no. Yes. That is, both yes and no. We tried. We tried. We tried. use use use some of these tests. some of these tests. some of these tests. I think the difficulty is I think the difficulty is I think the difficulty is because the goal because the goal because the goal product—not product—not product—not be sure to write be sure to write be sure to write analytical SQL. Goal analytical SQL. Goal analytical SQL. Goal of what we are building of what we are building of what we are building for people, more for people, more for people, more focused on focused on focused on data engineering. So, data engineering. So, data engineering. So, I think that's enough. I think that's enough. I think that's enough. It is a common thing to say: " It is a common thing to say: " Oh, we're going. Oh, we're going. Oh, we're going. create create create text conversion text conversion text conversion in SQL. I think everyone in SQL. I think everyone in SQL. I think everyone are trying this are trying this are trying this to do, right? But I to do, right? But I to do, right? But I I think it's more important I think it's more important I think it's more important to say: we want to say: we want to say: we want create tools create tools which will allow you which will allow you which will allow you really trust really trust really trust text conversion text conversion text conversion in SQL. Not necessarily in SQL. Not necessarily in SQL. Not necessarily what we can do what we can do what we can do come and write come and write come and write best analytical best analytical best analytical requests for you. We requests for you. We requests for you. We we want to be sure, we want to be sure, we want to be sure, what when you perform what when you perform what when you perform your daily routine your daily routine your daily routine support work support work support work data architecture, we data architecture, we data architecture, we we can help you with this we can help you with this we can help you with this help. It help. It help. It answers your answers your answers your question? OK.

question? OK. So, returning to if I find a mouse, if I find a mouse, Returning to Returning to slides. So, what slides. So, what slides. So, what we saw we saw we saw at the end, at the end, at the end, That's right, that's what That's right, that's what That's right, that's what we, uh, we really we, uh, we really we, uh, we really we can distinguish these we can distinguish these we can distinguish these various lists various lists various lists publications, because publications, because publications, because we can, uh, we can, uh, we can, uh, ask our new ask our new ask our new the context of the graph, and we the context of the graph, and we the context of the graph, and we can we really can we really can we really find out everything find out everything find out everything depending on the given depending on the given depending on the given character for one character for one character for one transition, uh, without transition, uh, without transition, uh, without no repeat no repeat no repeat delight. Uh, and delight. Uh, and delight. Uh, and so how does the query work, so how does the query work, so how does the query work, you know, he you know, he you know, he corresponds to the full corresponds to the full corresponds to the full qualified name qualified name , and then goes and , and then goes and , and then goes and finds all finds all finds all addiction. And that's all, addiction. And that's all, addiction. And that's all, guys. Excellent guys. Excellent guys. Excellent seminar. We are finished. seminar. We are finished. seminar. We are finished. with No, no, I just with No, no, I just with No, no, I just I'm kidding. So, the point is I'm kidding. So, the point is I'm kidding. So, the point is because it's cool. It because it's cool. It because it's cool. It Actually, yes. This Actually, yes. This Actually, yes. This the tool really the tool really the tool really wonderful, isn't it? wonderful, isn't it? wonderful, isn't it? This tool can This tool can This tool can to do for one to do for one to do for one challenge what exactly challenge what exactly challenge what exactly it would take a few it would take a few it would take a few Counts. I don't think so.

Counts. I don't think so. Counts. I don't think so. you could achieve you could achieve you could achieve this in one graph, this in one graph, this in one graph, even if they did even if they did even if they did perfect graph, right? perfect graph, right? perfect graph, right? But, er, the problem is But, er, the problem is But, er, the problem is because the agent, you because the agent, you because the agent, you should actually should actually should actually go and give it to the agent. go and give it to the agent. go and give it to the agent. So when you give So when you give So when you give this agent this this agent this this agent this tool, and you tool, and you tool, and you give this agent give this agent give this agent this tool, and you this tool, and you this tool, and you give it along with give it along with give it along with all available all available all available tools that tools that tools that already has an agent, and already has an agent, and already has an agent, and you know, we have this one you know, we have this one you know, we have this one really cool really cool really cool tool. We saw tool. We saw how does he do these how does he do these how does he do these amazing things, and he amazing things, and he amazing things, and he I ran grep anyway. I ran grep anyway. I ran grep anyway. Uh, if you give Uh, if you give Uh, if you give he has access to he has access to he has access to graph tools graph tools graph tools and grep, it's the same and grep, it's the same and grep, it's the same run grep. And we run grep. And we run grep. And we gave him amazing gave him amazing gave him amazing tools, tools, tools, pre-calculated pre-calculated pre-calculated codebase model, codebase model, codebase model, which one can which one can which one can to look at, and he to look at, and he to look at, and he it just doesn't matter it just doesn't matter it just doesn't matter ran grep. He ran grep. He ran grep. He just grouped and just grouped and just grouped and read files, and read files, and read files, and ran grep, and sometimes, ran grep, and sometimes, ran grep, and sometimes, you know, the count touched and you know, the count touched and you know, the count touched and confused, and then confused, and then confused, and then was going back to, you know was going back to, you know , waiting for , waiting for , waiting for working with the graph. And this working with the graph. And this working with the graph. And this something that no one talks about something that no one talks about something that no one talks about not really not really not really speaks. Everyone says: speaks. Everyone says: speaks. Everyone says: "Oh, you created these "Oh, you created these "Oh, you created these cool, attractive cool, attractive cool, attractive tools, and they tools, and they tools, and they do these amazing do these amazing do these amazing things." But the agent things." But the agent things." But the agent should actually should actually should actually use them, use them, use them, Yes? You must Yes? You must Yes? You must make a prompt- make a prompt- engineering and other engineering and other engineering and other hard work and grades hard work and grades . This is the rest This is . This is the rest This is . This is the rest This is the hardest part the hardest part , right? You can , right? You can , right? You can create this notebook. create this notebook. create this notebook. You can create You can create You can create your amazing your amazing your amazing the context of the graph for the context of the graph for the context of the graph for one evening. Complicated one evening. Complicated one evening. Complicated part—this part—this part—this actually force actually force actually force agent agent agent use it use it . Uh, like uh, yeah. Uh, and . Uh, like uh, yeah. Uh, and . Uh, like uh, yeah. Uh, and Why is he doing this?

Why is he doing this? Why is he doing this? Well, it's because of the post- Well, it's because of the post- training. Models training. Models training. Models trained and trained and trained and designed for designed for designed for calling these calling these calling these tools, right? IN tools, right? IN tools, right? IN their education is their education is their education is so many situations, so many situations, so many situations, where the answer is grep and where the answer is grep and where the answer is grep and reading, right? And you reading, right? And you reading, right? And you not necessarily not necessarily not necessarily you can do this you can do this you can do this to resist, simply to resist, simply to resist, simply creating a better creating a better creating a better tool. You can tool. You can tool. You can create a better one create a better one create a better one tool, but tool, but tool, but muscle memory muscle memory muscle memory the agent will not allow the agent will not allow the agent will not allow him him him him him him use. It use. It use. It very similar to humans, very similar to humans, very similar to humans, truth? You can truth? You can truth? You can create for people create for people create for people the best tool, the best tool, the best tool, but if you try but if you try but if you try to force them to force them to force them to use it, to use it, to use it, they will be they will be they will be dissatisfied. dissatisfied. dissatisfied. Maybe it's a little bit off. Maybe it's a little bit off. Maybe it's a little bit off. on the topic, I don't know. But on the topic, I don't know. But on the topic, I don't know. But the important thing is that you don't the important thing is that you don't the important thing is that you don't necessarily necessarily necessarily compete in whether compete in whether compete in whether is your tool is your tool is your tool better than all the others in better than all the others in better than all the others in toolkit toolkit toolkit agent, right? You agent, right? You agent, right? You trying to fight trying to fight trying to fight with his own with his own with his own reflexes. So how is this? reflexes. So how is this? reflexes. So how is this? really do? Or really do? Or really do? Or wrong? Because I don't think so. wrong? Because I don't think so. wrong? Because I don't think so. that there is already a question that that there is already a question that that there is already a question that yes, if he could yes, if he could yes, if he could to ask the graph, it to ask the graph, it to ask the graph, it would be more effective and would be more effective and would be more effective and better. We will prove it. better. We will prove it. better. We will prove it. later, but how about us later, but how about us later, but how about us make him do it make him do it make him do it do? And the answer do? And the answer do? And the answer is to not is to not is to not to fight with to fight with to fight with previous experience previous experience previous experience models, but to rely on models, but to rely on models, but to rely on him, right? We him, right? We him, right? We we make the graph we make the graph we make the graph felt like a model felt like a model felt like a model how to use grep, how to use grep, how to use grep, truth? You take truth? You take truth? You take a tool that can a tool that can a tool that can be called query search, be called query search, be called query search, query graph or how query graph or how query graph or how whatever, and simply whatever, and simply whatever, and simply you call it search, you call it search, you call it search, truth? The agent will truth? The agent will truth? The agent will use it use it use it in a certain way. I in a certain way. I in a certain way. I found that if found that if found that if name the instrument name the instrument name the instrument search, agent practically search, agent practically search, agent practically will always do in will always do in will always do in search for it search for it search for it several key several key several key in words without any in words without any in words without any teaching. So, you teaching. So, you teaching. So, you models must go models must go models must go towards, right? You towards, right? You towards, right? You take the model, take the model, take the model, you call it search, you call it search, you call it search, take into account how take into account how take into account how the model will do the model will do the model will do request, and then just

request, and then just request, and then just you get results you get results you get results from the graph. And you from the graph. And you from the graph. And you return return return results from the graph in results from the graph in results from the graph in in the same form as in the same form as in the same form as rip grep, right? You rip grep, right? You rip grep, right? You take the same one take the same one take the same one structure structure structure tool and tool and tool and answers as in rip grep answers as in rip grep , and return it , and return it , and return it Back to the model, right? Back to the model, right? Back to the model, right? Because it is on this Because it is on this Because it is on this the model has been trained. the model has been trained. the model has been trained. She knows what it's like. She knows what it's like. She knows what it's like. do. And you don't do. And you don't do. And you don't force her completely force her completely force her completely change only for change only for change only for using hints, using hints, using hints, truth? And the last one: truth? And the last one: truth? And the last one: your tools your tools your tools must bring must bring must bring result from the first result from the first result from the first everyone's appeal everyone's appeal everyone's appeal once, because when once, because when once, because when the agent chooses a certain the agent chooses a certain the agent chooses a certain path and gets into path and gets into path and gets into a dead end or something, a dead end or something, a dead end or something, who considers deaf who considers deaf who considers deaf corner, it causes corner, it causes corner, it causes search and does not receive search and does not receive search and does not receive results. Well, results. Well, results. Well, sometimes he will think, sometimes he will think, sometimes he will think, that the search is broken. that the search is broken. that the search is broken. If I don't get If I don't get If I don't get results, although results, although results, although they should be, they should be, they should be, maybe the search is not maybe the search is not maybe the search is not works. I should works. I should works. I should try another one try another one try another one tool, right? He tool, right? He tool, right? He will not return and will not will not return and will not will not return and will not will use this will use this will use this tool again. AND tool again. AND tool again. AND if he returns, if he returns, if he returns, maybe he will try maybe he will try maybe he will try use it use it use it otherwise. You can't otherwise. You can't otherwise. You can't is it predictable or not is it predictable or not is it predictable or not Yes? Therefore you must Yes? Therefore you must Yes? Therefore you must to do everything possible, to do everything possible, to do everything possible, so that your tools so that your tools so that your tools immediately returned immediately returned immediately returned something very useful. something very useful. Otherwise they just Otherwise they just will return to grep and will return to grep and will return to grep and will never try will never try will never try again. So, the second again. So, the second again. So, the second the notebook is shorter. It the notebook is shorter. It the notebook is shorter. It a little more a little more purposeful purposeful example. And all this about example. And all this about example. And all this about evaluation, right? As evaluation, right? As evaluation, right? As actually show, actually show, actually show, that the agent does what that the agent does what , what do you think, he , what do you think, he , what do you think, he should do, right? And how should do, right? And how should do, right? And how actually prove that actually prove that actually prove that your tool your tool your tool better than the one already better than the one already better than the one already can exist, right?

can exist, right? So, that's all. So, that's all. pre-cached. pre-cached. pre-cached. I completed this run. I completed this run. I completed this run. If you want If you want If you want launch notepad launch notepad launch notepad locally, you can locally, you can locally, you can set size set size set size samples greater than zero and samples greater than zero and samples greater than zero and start assessment start assessment start assessment in real time. It in real time. It in real time. It It will take a lot of time. Not It will take a lot of time. Not It will take a lot of time. Not Tell me I don't love you. Tell me I don't love you. Tell me I don't love you. warned. But I warned. But I warned. But I ran it at night with ran it at night with ran it at night with five repetitions five repetitions five repetitions attempts and 30 attempts and 30 attempts and 30 questions in the set questions in the set questions in the set for evaluation. So for evaluation. So for evaluation. So that's where these come from that's where these come from that's where these come from results for this results for this results for this notebook. So, first, notebook. So, first, notebook. So, first, what do we do after what do we do after what do we do after settings, —this, settings, —this, settings, —this, actually, we create actually, we create actually, we create our test ones our test ones our test ones examples, right? Therefore, examples, right? Therefore, examples, right? Therefore, we will create an agent from we will create an agent from we will create an agent from our new ones our new ones our new ones graph graph graph tools. We tools. We tools. We we will give the agent access we will give the agent access we will give the agent access to the same model and to the same model and to the same model and our counts our counts our counts tools. We have tools. We have tools. We have will be our test will be our test will be our test an agent who will have an agent who will have an agent who will have access to the same access to the same access to the same model and set model and set model and set tools for tools for tools for file research file research file research systems that systems that systems that supported by Dantic. supported by Dantic. So these will be ours So these will be ours two test agents. AND two test agents. AND two test agents. AND then we will create then we will create then we will create corpus of questions.

corpus of questions. corpus of questions. Every question is Every question is Every question is real task. We real task. We real task. We Let's find out, for example, Let's find out, for example, Let's find out, for example, question: which public question: which public question: which public types will break if types will break if types will break if I will change the list. I will change the list. I will change the list. publications, or what publications, or what publications, or what me to use me to use ? You know, things ? You know, things ? You know, things of this kind. Where are we? of this kind. Where are we? of this kind. Where are we? we are trying to say: we are trying to say: we are trying to say: "Okay, that should be it." "Okay, that should be it." "Okay, that should be it." a question that is perfect a question that is perfect a question that is perfect "fit for a count." "fit for a count." "fit for a count." This is a question that, This is a question that, This is a question that, probably great probably great probably great suitable for grep. Uh, suitable for grep. Uh, suitable for grep. Uh, you know, these are controls you know, these are controls you know, these are controls questions that, questions that, questions that, probably good probably good probably good suitable for both, suitable for both, suitable for both, isn't that right? Therefore, isn't that right? Therefore, isn't that right? Therefore, reference data for reference data for reference data for this is what we really are this is what we really are this is what we really are we extract from the graph. We we extract from the graph. We we extract from the graph. We we compare them with we compare them with we compare them with source code. So source code. So source code. So we do not pollute we do not pollute we do not pollute reference context reference context reference context the graph output. We the graph output. We the graph output. We must verify them, must verify them, must verify them, to make sure that to make sure that to make sure that The standard does not "cheat". The standard does not "cheat". Uh, so now we Uh, so now we we can download we can download we can download our standard and, in fact, our standard and, in fact, our standard and, in fact, start setting up start setting up start setting up the structure itself the structure itself the structure itself evaluation. And that's exactly it. evaluation. And that's exactly it. evaluation. And that's exactly it. defines the difference between defines the difference between defines the difference between good question for good question for good question for evaluation and evaluation and evaluation and bad. Uh, you bad. Uh, you bad. Uh, you Do you want a question for Do you want a question for Do you want a question for assessment that can be assessment that can be ...Ideally yours ...Ideally yours ...Ideally yours answers for answers for answers for evaluations have evaluations have evaluations have to be determined to be determined to be determined evaluated. I believe evaluated. I believe that LLM as a judge is that LLM as a judge is that LLM as a judge is fantasy for fantasy for fantasy for many cases, many cases, many cases, but it is often but it is often but it is often use use use excessively, although you excessively, although you excessively, although you must try must try must try to do one's own thing to do one's own thing to do one's own thing evaluation evaluation evaluation maximum maximum maximum deterministic. It deterministic. It deterministic. It will reduce will reduce will reduce the variability of your the variability of your the variability of your starts, will reduce starts, will reduce starts, will reduce their costs and their costs and their costs and will make you will make you will make you happier in happier in happier in long-term long-term long-term perspective. Uh, so perspective. Uh, so perspective. Uh, so you want to create you want to create you want to create questions, where are the answers questions, where are the answers questions, where are the answers can be extracted from...you can be extracted from...you can be extracted from...you can you structure can you structure can you structure the answer to them, which the answer to them, which the answer to them, which is standard and works.

is standard and works. is standard and works. You can check You can check You can check answer for answer for answer for using a standard. THERE ARE- using a standard. THERE ARE- Well, you're not just Well, you're not just Well, you're not just ask for an agent ask for an agent ask for an agent to give your opinion, and to give your opinion, and to give your opinion, and then ask for another then ask for another then ask for another agent to deny this agent to deny this agent to deny this my opinion. Such my opinion. Such my opinion. Such question usually question usually question usually never work as never work as never work as evaluations. Uh-huh. Okay, that's it. evaluations. Uh-huh. Okay, that's it. evaluations. Uh-huh. Okay, that's it. the text of the graph, you know, the text of the graph, you know, the text of the graph, you know, don't worry about it don't worry about it don't worry about it too much. This, according to too much. This, according to too much. This, according to basically, simply basically, simply basically, simply creating test cases creating test cases creating test cases cases that I am talking about cases that I am talking about cases that I am talking about said. It said. It said. It deterministic deterministic deterministic the part where we the part where we the part where we we check whether we check whether we check whether he leads all he leads all he leads all the evidence we have the evidence we have the evidence we have we expect. We just we expect. We just we expect. We just creating a list creating a list creating a list examples, we create examples, we create examples, we create list of things to list of things to list of things to should be referred to, and should be referred to, and should be referred to, and then we check whether then we check whether then we check whether he refers to he refers to he refers to all of them. For ourselves all of them. For ourselves all of them. For ourselves we are estimates we are estimates we are estimates we use we use we use configuration in Pydantic, configuration in Pydantic, configuration in Pydantic, just because it is just because it is just because it is very easy very easy very easy to adjust. Uh, this is to adjust. Uh, this is to adjust. Uh, this is works like this: we works like this: we works like this: we we create two we create two we create two different agents, then different agents, then different agents, then we create ourselves we create ourselves we create ourselves agent launcher, and agent launcher, and agent launcher, and then we create then we create then we create appraiser, right? So appraiser, right? So appraiser, right? So it is he who actually it is he who actually it is he who actually will evaluate the agent will evaluate the agent will evaluate the agent based on based on based on put before put before put before them tasks. We them tasks. We them tasks. We let's just compare that let's just compare that let's just compare that what the agent said, with that what the agent said, with that what we expect from what we expect from what we expect from it for everyone it for everyone it for everyone task, right? And so task, right? And so task, right? And so how can we how can we how can we compare answers compare answers compare answers and results between and results between and results between our agent prep and our agent prep and our agent prep and by our agent graph, by our agent graph, by our agent graph, True? This is how we True? This is how we True? This is how we we can determine, by- we can determine, by- first, which one is real first, which one is real first, which one is real the usefulness of our the usefulness of our the usefulness of our skills, and secondly, skills, and secondly, skills, and secondly, how can we do something how can we do something how can we do something improve or what we improve or what we improve or what we we can do better with we can do better with we can do better with sometimes. So let's sometimes. So let's sometimes. So let's let's start. So, I don't

let's start. So, I don't let's start. So, I don't I will start email. I will start email. I will start email. again, because this again, because this again, because this will take about 30 will take about 30 will take about 30 minutes, but we see, minutes, but we see, minutes, but we see, that we launched it that we launched it that we launched it for the first time with k, which for the first time with k, which for the first time with k, which is equal to one. So is equal to one. So is equal to one. So we just say for we just say for we just say for each task in each task in each task in in our dataset: in our dataset: in our dataset: one step for the agent one step for the agent one step for the agent prep and the graph agent. AND prep and the graph agent. AND prep and the graph agent. AND when we do this, when we do this, when we do this, The graph agent is not working. The graph agent is not working. The graph agent is not working. very good You see, very good You see, very good You see, that the graph agent is a bit that the graph agent is a bit that the graph agent is a bit ahead of the other, ahead of the other, ahead of the other, but it's not very but it's not very but it's not very stable and it is not stable and it is not stable and it is not exactly what we exactly what we exactly what we You wanted to, right? But You wanted to, right? But You wanted to, right? But It is important to understand that It is important to understand that It is important to understand that the goal is never the goal is never the goal is never consisted in the fact that consisted in the fact that consisted in the fact that to create something that to create something that to create something that it would be more correct it would be more correct it would be more correct by grep. The goal, because by grep. The goal, because by grep. The goal, because all of us who all of us who all of us who uses grep uses grep uses grep every day, when every day, when every day, when we use our we use our we use our agents for agents for agents for coding, we know that coding, we know that coding, we know that he is very good at he is very good at he is very good at to your business. Let's say to your business. Let's say to your business. Let's say yes, grep yes, grep yes, grep are not used are not used are not used just like that. He is not just like that. He is not just like that. He is not will be wrong will be wrong will be wrong most of the time, most of the time, most of the time, True? But True? But True? But the most important thing is not that the most important thing is not that the most important thing is not that that he is right about this that he is right about this that he is right about this once when I once when I once when I tried, but what tried, but what tried, but what when I try 5, 10, 15, when I try 5, 10, 15, when I try 5, 10, 15, 20 times in a row, he 20 times in a row, he 20 times in a row, he does it right does it right does it right every time, right? Therefore, every time, right? Therefore, every time, right? Therefore, that's exactly what we need that's exactly what we need that's exactly what we need start sampling for start sampling for start sampling for these tests. Now, these tests. Now, these tests. Now, when we say that when we say that when we say that we launch the same we launch the same we launch the same set of tests on the volume set of tests on the volume set of tests on the volume the body itself has five the body itself has five the body itself has five times to get times to get times to get our metrics pass at K, we our metrics pass at K, we our metrics pass at K, we we get a slightly different we get a slightly different we get a slightly different answer, right? Therefore, answer, right? Therefore, answer, right? Therefore, we have two different we have two different we have two different statistics: our pass at K statistics: our pass at K statistics: our pass at K and our pass hat K. Pass at K and our pass hat K. Pass at K and our pass hat K. Pass at K says: was it says: was it says: was it at least once correctly at least once correctly at least once correctly these five launches? IN these five launches? IN these five launches? IN in this case they in this case they in this case they still very similar.

still very similar. still very similar. Actually, grep here has Actually, grep here has Actually, grep here has a small advantage. a small advantage. a small advantage. Graph does not select any Graph does not select any Graph does not select any from private variables, from private variables, from private variables, only private only private only private parts of packages, parts of packages, parts of packages, only public ones. Ago only public ones. Ago only public ones. Ago there are some controls there are some controls there are some controls questions that grep questions that grep questions that grep will always be better. will always be better. will always be better. But what's really interesting But what's really interesting , so this is when you , so this is when you , so this is when you change the metric and change the metric and change the metric and you ask: does it perform you ask: does it perform you ask: does it perform he does his job he does his job he does his job correctly during correctly during correctly during every launch? AND every launch? AND every launch? AND when you do this, when you do this, when you do this, numbers in full numbers in full numbers in full are changing, right? are changing, right? are changing, right? So, suddenly So, suddenly So, suddenly It turned out that the graph It turned out that the graph agent twice as good agent twice as good agent twice as good copes with copes with copes with correct correct correct by performing all five by performing all five by performing all five launches than launches than launches than A regular agent, right? A regular agent, right? A regular agent, right? We see that during We see that during We see that during implementation of these implementation of these implementation of these tasks he not only tasks he not only tasks he not only acts more consistently, but also acts more consistently, but also acts more consistently, but also more efficient, right? more efficient, right? more efficient, right? Yes? We see that Yes? We see that Yes? We see that throughout all our throughout all our throughout all our graph-approach launches graph-approach launches graph-approach launches used in used in used in on average 3.1 times on average 3.1 times on average 3.1 times fewer tokens. fewer tokens. fewer tokens. The last part: how The last part: how The last part: how we have to force the agent we have to force the agent we have to force the agent use graph use graph ? This is where they become ? This is where they become ? This is where they become useful assessment, useful assessment, useful assessment, because you because you because you need to write need to write need to write a test that will say, "Hey, a test that will say, "Hey, a test that will say, "Hey, view all view all view all tools in my tools in my tools in my Tracy, how often is my Tracy, how often is my Tracy, how often is my agent uses agent uses agent uses the tools that I the tools that I the tools that I "Do you want to?" And then you "Do you want to?" And then you "Do you want to?" And then you you will be able to get you will be able to get you will be able to get indicators indicators indicators use, right? It use, right? It use, right? It a way to get there a way to get there a way to get there to say, "Oh, I to say, "Oh, I to say, "Oh, I wrote a prompt without wrote a prompt without wrote a prompt without instructions regarding instructions regarding instructions regarding using using using tools, and tools, and tools, and received 34% received 34% use". I use". I use". I returned, changed returned, changed returned, changed your prompt, and now in your prompt, and now in your prompt, and now in me 72%use me 72%use me 72%use tools, right?

tools, right? tools, right? So, grades are So, grades are So, grades are the only way the only way the only way to understand whether it really to understand whether it really to understand whether it really agent uses agent uses agent uses those tools that those tools that those tools that you want. It's simple. you want. It's simple. you want. It's simple. checking this one checking this one checking this one concepts. We have two concepts. We have two concepts. We have two various prompts that various prompts that various prompts that are trying are trying are trying answer one and answer one and answer one and the same question for the same question for the same question for by means of that by means of that by means of that the set itself the set itself the set itself tools. If you tools. If you tools. If you just naively just naively just naively configure the prompt, configure the prompt, configure the prompt, he he he will use grep will use grep , and we will start doing , and we will start doing , and we will start doing something like that. If you something like that. If you something like that. If you correct it, in correct it, in correct it, in he has ideals he has ideals he has ideals perform a search and perform a search and perform a search and use get symbol, use get symbol, use get symbol, to find the right one to find the right one to find the right one respond. Recipe respond. Recipe respond. Recipe construction really construction really construction really reliable estimates reliable estimates reliable estimates is to is to is to to extract real to extract real to extract real task candidates task candidates task candidates from your body, from your body, from your body, create the maximum create the maximum create the maximum deterministic evaluation deterministic evaluation -anchor and first -anchor and first -anchor and first to conduct a unit- to conduct a unit- testing itself testing itself testing itself appraiser. Me appraiser. Me appraiser. Me like Atlantic evals. like Atlantic evals. like Atlantic evals. There are many others There are many others There are many others assessment systems, assessment systems, assessment systems, but their basic but their basic but their basic structure everywhere structure everywhere structure everywhere the same. Do you have the same. Do you have the same. Do you have dataset, you have dataset, you have dataset, you have cases and there is an evaluator. cases and there is an evaluator. cases and there is an evaluator. Make sure you Make sure you Make sure you launch yours launch yours launch yours evaluations many times and evaluations many times and evaluations many times and report on pass at K, pass report on pass at K, pass report on pass at K, pass at K and all your different at K and all your different at K and all your different metrics. AND metrics. AND metrics. AND make sure you make sure you erm, addition erm, addition examples of things, in examples of things, in examples of things, in which your agent does not which your agent does not which your agent does not necessarily strong, necessarily strong, necessarily strong, because you don't want to because you don't want to because you don't want to so that he passes so that he passes so that he passes 100% evaluation. It 100% evaluation. It 100% evaluation. It in vain. If your agent in vain. If your agent in vain. If your agent is being evaluated is being evaluated is being evaluated 100%, so 100%, so 100%, so evaluation too much evaluation too much evaluation too much easy. Your agent easy. Your agent easy. Your agent should be should be should be close to failure close to failure close to failure on your tests. Ahem, on your tests. Ahem, on your tests. Ahem, assessment—it's yours assessment—it's yours assessment—it's yours chance to see the moment chance to see the moment chance to see the moment truths and how your truths and how your truths and how your the agent will behave the agent will behave the agent will behave in the most difficult in the most difficult in the most difficult conditions. So this is for no one conditions. So this is for no one conditions. So this is for no one is of no use is of no use —give him only

—give him only —give him only very light and simple very light and simple very light and simple task for task for task for evaluation. OK. evaluation. OK. evaluation. OK. Does anyone have Does anyone have Does anyone have questions about this questions about this questions about this notebook with notebook with notebook with evaluation before evaluation before evaluation before I will return to I will return to I will return to slides? For slides? For slides? For the last part the last part the last part performance. performance. Well, I just wanted to Well, I just wanted to to ask, in the example, to ask, in the example, to ask, in the example, when you discussed, when you discussed, when you discussed, as an agent often as an agent often as an agent often uses grep uses grep uses grep instead of a graph, you instead of a graph, you instead of a graph, you had to had to had to reconfigure it reconfigure it reconfigure it on this instrument on this instrument on this instrument search. Were there search. Were there search. Were there attempts to take the graph and attempts to take the graph and attempts to take the graph and turn it into turn it into turn it into file structure, file structure, file structure, maybe with some maybe with some maybe with some losses, but such losses, but such losses, but such structure that grep could structure that grep could structure that grep could would be natural would be natural would be natural to work out so as not to to work out so as not to to work out so as not to to determine none to determine none to determine none separate tool separate tool ? ? ? Yes, not today. Yes, not today. Just because it was Just because it was a little short of time. a little short of time. a little short of time. But I think that this But I think that this But I think that this quite acceptable quite acceptable quite acceptable approach. I mean approach. I mean that this is what Copilot is, that this is what Copilot is, that this is what Copilot is, LLM Wiki, all these different LLM Wiki, all these different LLM Wiki, all these different things where you can things where you can things where you can get a lot get a lot get a lot advantages simply from advantages simply from advantages simply from Markdown corpus. But I Markdown corpus. But I Markdown corpus. But I I think it's true.

I think it's true. I think it's true. yes, and you still yes, and you still yes, and you still you will get a lot you will get a lot you will get a lot benefit from this, benefit from this, benefit from this, but I'm not sure but I'm not sure but I'm not sure regarding Markdown itself, like- regarding Markdown itself, like- here in LLM Wiki, you know? here in LLM Wiki, you know? I still think that I still think that your problem your problem your problem will consist in the fact that will consist in the fact that will consist in the fact that you need you need you need real connections and real connections and real connections and their significance, or their significance, or their significance, or wrong? And these connections— wrong? And these connections— this is what will be very this is what will be very this is what will be very difficult because difficult because difficult because even when you do even when you do even when you do something like The Brain, something like The Brain, something like The Brain, Do you understand? Yes, it is. Do you understand? Yes, it is. Do you understand? Yes, it is. builds all of this in Markdown, builds all of this in Markdown, builds all of this in Markdown, but it doesn't matter but it doesn't matter but it doesn't matter creates on top of it creates on top of it creates on top of it graph so you can graph so you can graph so you can see the timeline, see the timeline, see the timeline, connections etc., right? connections etc., right? connections etc., right? So, I think that on So, I think that on So, I think that on to a certain level, if to a certain level, if to a certain level, if your task your task your task based on need based on need based on need understand understand understand relationships between relationships between relationships between different things, then different things, then different things, then you don't care you don't care you don't care will need some will need some will need some Count, although I believe Count, although I believe Count, although I believe what can you get what can you get what can you get similar level similar level similar level success from the project on success from the project on success from the project on like this, yes. like this, yes. Hmm, what do you think? Hmm, what do you think? something is missing in something is missing in something is missing in in this setting, in this setting, in this setting, to allow to allow to allow non-technical professionals non-technical professionals , not data engineers , not data engineers , initiate PR for , initiate PR for , initiate PR for using this as using this as using this as code guide code guide code guide base? Have you tried this?

base? Have you tried this? base? Have you tried this? Is there anything you would like to Is there anything you would like to Is there anything you would like to added before added before added before to allow people to allow people to allow people to work in such a to work in such a to work in such a way? way? way? Oh, you mean Oh, you mean Oh, you mean is this like a tool? is this like a tool? is this like a tool? Yes. Yes. Yes. Oh, yes. I think, um, this is Oh, yes. I think, um, this is Oh, yes. I think, um, this is A valid question. Yes A valid question. Yes , I think , I think the simplest the simplest part, probably part, probably part, probably would be to start would be to start would be to start integration of this as integration of this as integration of this as MCP into tools, MCP into tools, MCP into tools, what are you already what are you already what are you already use. So, in use. So, in use. So, in you would be, you know, you would be, you know, you would be, you know, this code graph on this code graph on this code graph on I will add to yours. I will add to yours. I will add to yours. cloud code, and you cloud code, and you cloud code, and you could ask for yours could ask for yours could ask for yours cloud code cloud code cloud code to contact the count to contact the count to contact the count and perform other and perform other and perform other task. And then you would task. And then you would task. And then you would prescribed skills, prescribed skills, prescribed skills, so that he could really so that he could really so that he could really that's right that's right that's right use. I use. I use. I I think it would be I think it would be I think it would be the best approach. But the best approach. But the best approach. But I think so, if I think so, if I think so, if you are less technical you are less technical you are less technical savvy or not savvy or not savvy or not technical, I think technical, I think technical, I think the goal would be that the goal would be that the goal would be that if you want if you want if you want configure agent configure agent configure agent for the use of these for the use of these for the use of these things in context things in context things in context real coding, real coding, real coding, you would convert these you would convert these you would convert these prompt methods prompt methods engineering, about which engineering, about which engineering, about which I'm talking about skills. I'm talking about skills. I'm talking about skills. and gave them to their and gave them to their and gave them to their writing agent writing agent writing agent code. code. code. So that this PR So that this PR So that this PR got to got to got to engineer for the first time with engineer for the first time with engineer for the first time with pretty good pretty good pretty good first option first option first option thanks to more thanks to more thanks to more deterministic deterministic deterministic management by this management by this management by this systems.

systems. systems. Yes. Yes, and I think that Yes. Yes, and I think that Yes. Yes, and I think that deterministic deterministic deterministic part is useful, therefore part is useful, therefore part is useful, therefore what can you do, like us what can you do, like us what can you do, like us discussed earlier, discussed earlier, discussed earlier, responding to responding to responding to questions from the audience, questions from the audience, questions from the audience, just gradually just gradually just gradually review changes, review changes, review changes, right? right? right? Analyze this Analyze this Analyze this deterministic and deterministic and deterministic and to receive very to receive very to receive very quick conclusion about quick conclusion about quick conclusion about what is happening and what is happening and what is happening and what has changed since what has changed since what has changed since last time. On last time. On last time. On as opposed to as opposed to as opposed to wait for the agent wait for the agent wait for the agent will read everything, will read everything, will read everything, will review all changes and will review all changes and will review all changes and will give you the same will give you the same will give you the same result. This is it. result. This is it. result. This is it. questions behind, questions behind, questions behind, seems. seems. Um, did you spend Um, did you spend experiments or experiments or experiments or considered various considered various considered various levels of separation levels of separation levels of separation abilities? For example, abilities? For example, abilities? For example, I was thinking about what in I was thinking about what in I was thinking about what in some previous ones some previous ones some previous ones reports reports reports mentioned mentioned mentioned prognostication prognostication prognostication the next token, and the next token, and the next token, and you have a prefix like you have a prefix like you have a prefix like people said that people said that people said that simply boils down to simply boils down to simply boils down to that it is holistic that it is holistic that it is holistic a line of everything. Here, in a line of everything. Here, in a line of everything. Here, in codebase, you have codebase, you have codebase, you have symbols, but symbols symbols, but symbols symbols, but symbols have a certain have a certain have a certain granularity. granularity.

There are higher levels There are higher levels granularity, granularity, granularity, inherent in the code inherent in the code inherent in the code base, and make these base, and make these base, and make these things available for things available for things available for search agent, search agent, search agent, you know, maybe you know, maybe you know, maybe profitably. And I am here. profitably. And I am here. profitably. And I am here. I was wondering if you thought I was wondering if you thought I was wondering if you thought you about it, or do you have you about it, or do you have you about it, or do you have your thoughts on that, your thoughts on that, your thoughts on that, How would you like this? How would you like this? How would you like this? came up. came up. came up. This is fantastic. This is fantastic. This is fantastic. question. I really question. I really question. I really I thought about it. Yes, and I, I thought about it. Yes, and I, I thought about it. Yes, and I, Actually, I'll tell you about Actually, I'll tell you about Actually, I'll tell you about this is at the end this is at the end this is at the end presentations. presentations. presentations. Of course, there are certain Of course, there are certain Of course, there are certain the steps I would take here the steps I would take here the steps I would take here did extra. did extra. did extra. If you really If you really If you really tried this tried this tried this build, you would build, you would build, you would used something on used something on used something on like SCIP, so like SCIP, so like SCIP, so track graphs track graphs track graphs calls in code calls in code calls in code base. You can base. You can base. You can to see what and what to see what and what to see what and what causes, and how is it causes, and how is it causes, and how is it interacts because interacts because interacts because can be tracked can be tracked can be tracked symbols and their symbols and their symbols and their code movement code movement code movement base. But if you base. But if you base. But if you want to track what want to track what want to track what and what causes it and how does it all and what causes it and how does it all and what causes it and how does it all it's related, it's the one it's related, it's the one it's related, it's the one higher level, about higher level, about higher level, about which one you are talking about, and which one you are talking about, and which one you are talking about, and you need something you need something you need something an expanded approach, yet an expanded approach, yet an expanded approach, yet one degree one degree one degree deterministic deterministic deterministic parsing. And you can parsing. And you can parsing. And you can do it for do it for do it for with the help of, for example, with the help of, for example, with the help of, for example, Rust or SCIP. There are Rust or SCIP. There are Rust or SCIP. There are similar SCIP outputs for similar SCIP outputs for similar SCIP outputs for TypeScript or Python. Me too TypeScript or Python. Me too TypeScript or Python. Me too was going to do it, was going to do it, was going to do it, but I'm out of time but I'm out of time but I'm out of time time. But answering time. But answering time. But answering to your question: to your question: to your question: yes, of course we are yes, of course we are yes, of course we are we can get we can get we can get more structure with more structure with more structure with code base. And this code base. And this code base. And this exactly what we exactly what we exactly what we we are trying to do in we are trying to do in we are trying to do in Typedef: find different Typedef: find different Typedef: find different ways to extract ways to extract ways to extract as much as possible as much as possible as much as possible context from the available context from the available context from the available data.

data. Quick question: how, Quick question: how, in your opinion, this in your opinion, this in your opinion, this layer concept layer concept layer concept data context data context data context goes with something goes with something goes with something like a catalog like a catalog like a catalog data? Is the catalog data? Is the catalog data? Is the catalog data helps data helps data helps fill the graph? Or fill the graph? Or fill the graph? Or they are to some extent they are to some extent they are to some extent duplicate each other duplicate each other duplicate each other within the enterprise? within the enterprise? within the enterprise? I don't think they are. I don't think they are. I don't think they are. duplicate each other. duplicate each other. duplicate each other. I think they are I think they are I think they are answer to answer to answer to various questions. Your various questions. Your various questions. Your data catalog—this data catalog—this kind of fixed kind of fixed understanding that understanding that understanding that precisely present in precisely present in precisely present in your data warehouse, your data warehouse, your data warehouse, which, in my opinion, is which, in my opinion, is which, in my opinion, is with a slightly different request, with a slightly different request, with a slightly different request, than what than what than what trying to pull out trying to pull out trying to pull out context graph, or context graph, or context graph, or wrong? I think that wrong? I think that wrong? I think that they coexist. We they coexist. We they coexist. We we read the information we read the information we read the information from the catalog to from the catalog to from the catalog to to form to form to form context graph. context graph. context graph. So, you are trying So, you are trying So, you are trying say: "Okay, in say: "Okay, in say: "Okay, in I have my DBT project, in I have my DBT project, in I have my DBT project, in who has a bunch of models." who has a bunch of models." These models build These models build tables and tables and tables and Snowflake presentation Snowflake presentation , right? So we , right? So we , right? So we we can say that we can say that we can say that we are going we are going we are going import this DBT- import this DBT- project. We are going project. We are going project. We are going download all of this download all of this download all of this SQL. We will parse it.

SQL. We will parse it. , let's tie it all together , let's tie it all together , and then we'll say, "Oh, , and then we'll say, "Oh, , and then we'll say, "Oh, okay, this model okay, this model okay, this model corresponds to this corresponds to this corresponds to this tables". Allow tables". Allow tables". Allow I would like to make a request to I would like to make a request to I would like to make a request to catalog, get catalog, get catalog, get information about this information about this information about this table and also table and also table and also add it to the graph. « add it to the graph. « So, we can So, we can So, we can say: "Okay, this one say: "Okay, this one say: "Okay, this one the model creates this the model creates this the model creates this physical table, this physical table, this physical table, this the table has these the table has these the table has these columns, and these columns columns, and these columns columns, and these columns have profiles like this have profiles like this "". And this information "". And this information "". And this information will come from will come from will come from catalog, right? So this one catalog, right? So this one catalog, right? So this one information catalog, information catalog, information catalog, data schema, data schema, data schema, profile information profile information , all of this is an integral part , all of this is an integral part , all of this is an integral part part of our work, part of our work, part of our work, and it's all in the catalog. and it's all in the catalog. and it's all in the catalog. Therefore, I would say that Therefore, I would say that Therefore, I would say that they rather they rather they rather coexist than coexist than coexist than replace one replace one replace one one. one. one. Perfectly. Perfectly. Perfectly. Yes. Yes. Yes. Thank you. That Thank you. That Thank you. That return to topic return to topic return to topic data engineering and data engineering and data engineering and assessment concepts assessment concepts assessment concepts (Evals), do you think (Evals), do you think (Evals), do you think about this in the same vein: about this in the same vein: about this in the same vein: question question question regarding data and watch regarding data and watch how they react to how they react to how they react to use or use or use or non-use non-use non-use tool? tool? Yes. Yes. Yes. Yes, yes, exactly like that Yes. Yes, yes, exactly like that . Oh, sorry, do you have more?

. Oh, sorry, do you have more? . Oh, sorry, do you have more? What to add? Sorry. What to add? Sorry. What to add? Sorry. Please? Please? Please? Oh, I didn't know if you Oh, I didn't know if you Oh, I didn't know if you finished. Hmm, yes, finished. Hmm, yes, finished. Hmm, yes, certainly. Um, we certainly. Um, we certainly. Um, we really really really we use, like me we use, like me we use, like me I already said, we are in I already said, we are in I already said, we are in basically basically basically trying to find trying to find trying to find as real as possible as real as possible as real as possible data array, and then data array, and then data array, and then trying to find trying to find trying to find many different many different many different scenarios, right? Therefore, scenarios, right? Therefore, scenarios, right? Therefore, you have a basic " you have a basic " correct the error", where correct the error", where correct the error", where the scenario is the scenario is the scenario is because we contribute because we contribute because we contribute changes to the code base, changes to the code base, changes to the code base, who create a bug, who create a bug, who create a bug, Yes? And then you Yes? And then you Yes? And then you ask the agent to leave and ask the agent to leave and ask the agent to leave and try to fix try to fix try to fix this mistake. Hmm, and then this mistake. Hmm, and then this mistake. Hmm, and then you can check, you can check, you can check, did he look at did he look at did he look at the right places, or the right places, or the right places, or checked the necessary checked the necessary checked the necessary things, and what, things, and what, things, and what, fixing the bug, he fixing the bug, he fixing the bug, he didn't do this somewhere didn't do this somewhere didn't do this somewhere higher or lower than higher or lower than higher or lower than by the stream, simply by the stream, simply by the stream, simply by applying a "band-aid", by applying a "band-aid", by applying a "band-aid", Yes? He really is Yes? He really is Yes? He really is found the right one found the right one found the right one model and corrected model and corrected model and corrected the right problem. AND the right problem. AND the right problem. AND you can do these you can do these you can do these tests easier or tests easier or tests easier or more complicated, right? more complicated, right? more complicated, right? For example, you can For example, you can For example, you can change the wording change the wording change the wording hints: "Oh, I think hints: "Oh, I think hints: "Oh, I think what is the problem with what is the problem with what is the problem with "XYZ model". Or you can "XYZ model". Or you can "XYZ model". Or you can to say for to say for to say for a more complex example a more complex example : "My panel Y has : "My panel Y has : "My panel Y has problem, where is this number problem, where is this number problem, where is this number doubles. I am not doubles. I am not doubles. I am not I know why. Therefore, I know why. Therefore, I know why. Therefore, forcing the agent forcing the agent forcing the agent to work in to work in to work in in the opposite direction in the opposite direction in the opposite direction from addictions from addictions from addictions up the tree to up the tree to up the tree to to understand where from to understand where from to understand where from actually arose actually arose actually arose root cause root cause root cause problems. And then problems. And then problems. And then make sure that he make sure that he make sure that he really interferes really interferes really interferes and fixes this and fixes this and fixes this problem, not problem, not problem, not leaves it as is. Ahem, leaves it as is. Ahem, leaves it as is. Ahem, that's what it really is that's what it really is that's what it really is difficult. And the problem is difficult. And the problem is difficult. And the problem is because agents because agents because agents always always always will try will try will try to cheat. So in to cheat. So in to cheat. So in to his/her to his/her to his/her tools for tools for tools for you are evaluating you are evaluating you are evaluating must be very must be very must be very careful and careful and careful and strengthen it strengthen it strengthen it protection. Well, for example, protection. Well, for example, protection. Well, for example, your agent if you your agent if you your agent if you you say, "Oh, my you say, "Oh, my you say, "Oh, my test case test case test case is that is that is that the agent will check the agent will check the agent will check branch, and then branch, and then branch, and then will try to fix will try to fix will try to fix problem." Agent from problem." Agent from problem." Agent from He is happy to deceive and He is happy to deceive and He is happy to deceive and will use Git to will use Git to will use Git to return to

return to return to main branch and main branch and main branch and find out exactly how you find out exactly how you find out exactly how you created a problem in created a problem in created a problem in this branch. He is from this branch. He is from this branch. He is from will happily find a name will happily find a name will happily find a name branches where it says " branches where it says " this is a test", and thinks: " this is a test", and thinks: " Oh, this is a test. I can here Oh, this is a test. I can here Oh, this is a test. I can here to cheat." So you to cheat." So you to cheat." So you should be very should be very should be very careful with careful with careful with strengthening your strengthening your strengthening your environment so that it environment so that it environment so that it really reflected really reflected really reflected those tasks that you those tasks that you those tasks that you want to entrust want to entrust want to entrust to his agent. Ago to his agent. Ago to his agent. Ago that it will always be that it will always be that it will always be danger. danger. danger. Regarding these estimates, whether Regarding these estimates, whether Regarding these estimates, whether people mostly people mostly people mostly create them from scratch, create them from scratch, create them from scratch, or how? or how? or how? There are certain ready-made There are certain ready-made There are certain ready-made frameworks for this. frameworks for this. frameworks for this. ADE bench—one of the ADE bench—one of the ADE bench—one of the wonderful examples. It wonderful examples. It probably current probably current probably current industry standard industry standard industry standard for similar for similar for similar benchmarks in benchmarks in benchmarks in data development. They have data development. They have data development. They have very similar approach to very similar approach to very similar approach to creating problems in creating problems in creating problems in repositories for repositories for repositories for using diff files, using diff files, using diff files, and then the agent reads and then the agent reads and then the agent reads data from the storage and data from the storage and data from the storage and code, trying code, trying code, trying fix the problem. fix the problem. So they exist. So they exist. Their people Their people Their people use, and I use, and I use, and I I think it's becoming more and more I think it's becoming more and more I think it's becoming more and more more popular. I more popular. I more popular. I I think the hardest thing is— I think the hardest thing is— this is infrastructure this is infrastructure this is infrastructure around them because around them because around them because for the code you have for the code you have for the code you have codebase, but you codebase, but you codebase, but you need running need running need running database, it has database, it has database, it has to be filled to be filled to be filled correct correct correct erroneous data, and erroneous data, and erroneous data, and this data should be this data should be this data should be internally internally internally agreed. You are not agreed. You are not agreed. You are not do you want the agent do you want the agent do you want the agent wasted time, wasted time, wasted time, trying trying trying understand why numbers understand why numbers understand why numbers from 2024. Therefore, we have from 2024. Therefore, we have from 2024. Therefore, we have sample only data for sample only data for sample only data for January 2024. I January 2024. I January 2024. I I think you often I think you often I think you often encounter such encounter such encounter such problems. That's why problems. That's why problems. That's why people are trying people are trying people are trying standardize it in standardize it in standardize it in ADE. I believe that ADE. I believe that ADE. I believe that the hardest thing is to create the hardest thing is to create the hardest thing is to create such a benchmark that such a benchmark that such a benchmark that others could others could others could use and use and use and who really has who really has who really has big enough big enough big enough data set. This will be data set. This will be data set. This will be the most difficult the most difficult the most difficult part, and that's it part, and that's it part, and that's it no one succeeded no one succeeded no one succeeded make. At least,

make. At least, make. At least, in my opinion. On mine in my opinion. On mine in my opinion. On mine opinion. But yeah, you know opinion. But yeah, you know , we return to , we return to , we return to that it is only that it is only that it is only generalization of those generalization of those generalization of those the results themselves, the results themselves, the results themselves, which we just saw. which we just saw. which we just saw. We see that as We see that as We see that as how we how we Increasing Increasing opportunities and opportunities and opportunities and prompting the agent prompting the agent prompting the agent more often more often more often use use use tools, we tools, we tools, we we see that he is becoming we see that he is becoming we see that he is becoming more consistent in more consistent in more consistent in in their answers to in their answers to in their answers to the same questions and the same questions and the same questions and answers them answers them answers them more efficient. Um, and here it is more efficient. Um, and here it is more efficient. Um, and here it is an example of why an example of why an example of why very important very important very important to conduct these to conduct these to conduct these evaluation several evaluation several evaluation several times. It's not about times. It's not about times. It's not about accuracy from the first accuracy from the first accuracy from the first attempts, and in attempts, and in attempts, and in determinism. Um, determinism. Um, determinism. Um, and the indicator that and the indicator that and the indicator that it really matters it really matters —this is how to make sure, —this is how to make sure, —this is how to make sure, that your agent really that your agent really that your agent really uses new uses new uses new tools. Therefore you tools. Therefore you tools. Therefore you must make sure must make sure that you raise that you raise that you raise implementation level implementation level implementation level and track and track and track using using using tools, because tools, because tools, because This is what will allow you This is what will allow you This is what will allow you to reach the final to reach the final to reach the final result, right? result, right? result, right? You know, elegant. You know, elegant. You know, elegant. tool that tool that tool that only use only use only use in a third of cases— in a third of cases— this is only a third this is only a third this is only a third tool. Um, and tool. Um, and tool. Um, and every improvement every improvement every improvement opportunities thanks to opportunities thanks to opportunities thanks to the best tools the best tools the best tools limited to this limited to this limited to this indicator. So exactly indicator. So exactly indicator. So exactly that's why we created that's why we created that's why we created evaluation system: evaluation system: evaluation system: not in order to not in order to not in order to to prove that to prove that to prove that the tool is good, but the tool is good, but the tool is good, but to make sure that to make sure that to make sure that he does not stand idle without he does not stand idle without he does not stand idle without business. The most difficult in business. The most difficult in business. The most difficult in this report was this report was this report was acknowledge that grep is working acknowledge that grep is working acknowledge that grep is working really very good.

really very good. really very good. Um, I did all this Um, I did all this Um, I did all this work, built work, built work, built context graph, context graph, context graph, conducted an assessment, and conducted an assessment, and conducted an assessment, and the result was the result was the result was no one. It was no one. It was no one. It was quite disappointing, quite disappointing, quite disappointing, because I want to because I want to because I want to continue continue continue optimize, optimize, optimize, to develop and to develop and to develop and to squeeze everything out of it to squeeze everything out of it to squeeze everything out of it more results. more results. more results. But this is kind of But this is kind of But this is kind of trap, right? trap, right? trap, right? Instinct tells you Instinct tells you Instinct tells you build more. I build more. I build more. I I think it sometimes leads I think it sometimes leads I think it sometimes leads to expenses that are not to expenses that are not to expenses that are not are justified, right? are justified, right? are justified, right? The agent has his own The agent has his own The agent has his own logic. I have my own. logic. I have my own. logic. I have my own. vision of what he vision of what he vision of what he should do, and we should do, and we should do, and we trying to find trying to find trying to find common language, right? common language, right? common language, right? We are trying We are trying We are trying understand how understand how understand how move on, right? AND move on, right? AND move on, right? AND seems like, "Oh, maybe, seems like, "Oh, maybe, seems like, "Oh, maybe, just need just need just need more testing, more testing, more testing, evaluations or something else evaluations or something else evaluations or something else something, right?" And in something, right?" And in something, right?" And in results in results in results in deceptive feeling: " deceptive feeling: " I just need I just need I just need more strictness and more strictness and more strictness and tests, and then the agent tests, and then the agent tests, and then the agent better better better will use will use will use "tools". But, "tools". But, "tools". But, maybe it's worth it maybe it's worth it maybe it's worth it ask: "Hey, why ask: "Hey, why ask: "Hey, why grep works that well?" grep works that well?" grep works that well?" Instead of Instead of Instead of ask: "Why doesn't he ask: "Why doesn't he ask: "Why doesn't he uses my uses my uses my "tools?" Maybe "tools?" Maybe , the answer is that , the answer is that , the answer is that grep is actually grep is actually grep is actually works very well, works very well, works very well, truth? So, perhaps, truth? So, perhaps, truth? So, perhaps, you can't win this one you can't win this one you can't win this one fight. Perhaps, fight. Perhaps, fight. Perhaps, it's just a little bit it's just a little bit it's just a little bit to trust him. Therefore to trust him. Therefore , the agent will still be , the agent will still be , the agent will still be use grep.

use grep. use grep. So when you constantly So when you constantly So when you constantly trying trying trying outsmart grep, you outsmart grep, you outsmart grep, you you're just wasting your time you're just wasting your time you're just wasting your time time. If you can time. If you can time. If you can design a system design a system design a system so that she so that she so that she worked naturally, worked naturally, worked naturally, helping agents helping agents helping agents to do what one has to do to do what one has to do to do what one has to do they are already inclined to do so, it they are already inclined to do so, it they are already inclined to do so, it will help you a lot. will help you a lot. will help you a lot. I have already several times I have already several times I have already several times spoke from spoke from spoke from reports, where reports, where reports, where reached a certain point reached a certain point reached a certain point conclusion, concluded conclusion, concluded conclusion, concluded speech and said: " speech and said: " Let me do it again. Let me do it again. Let me do it again. run one set run one set run one set tests". And then I tests". And then I tests". And then I had to had to had to to recycle everything to recycle everything to recycle everything report, because the conclusions report, because the conclusions report, because the conclusions turned out to be completely turned out to be completely turned out to be completely others. The thing is, I others. The thing is, I others. The thing is, I wrote his tests, wrote his tests, wrote his tests, created a whole created a whole created a whole tools, but tools, but tools, but did it did it did it wrong. I wrong. I wrong. I got the results, got the results, got the results, which were correct which were correct which were correct by vector. They by vector. They by vector. They showed that grep showed that grep showed that grep is superior to grep, but I is superior to grep, but I is superior to grep, but I wanted to take a look wanted to take a look wanted to take a look deeper. And when? deeper. And when? deeper. And when? you look deeper, you look deeper, you look deeper, turns out: "Oh, that one turns out: "Oh, that one turns out: "Oh, that one the tool that we the tool that we the tool that we created, returns created, returns created, returns file path, but this file path, but this file path, but this "not quite the way." "not quite the way." "not quite the way." So sometimes when they So sometimes when they So sometimes when they cause a search, it cause a search, it cause a search, it finds the wrong one finds the wrong one finds the wrong one way and tries way and tries way and tries read the wrong ones at all read the wrong ones at all read the wrong ones at all files. Therefore, you files. Therefore, you files. Therefore, you I have to go in. I have to go in. I have to go in. and reread everything and reread everything and reread everything your tracings, your tracings, your tracings, Do you understand? Go to Do you understand? Go to Do you understand? Go to log files, log files, log files, make sure you make sure you make sure you check that place, check that place, check that place, where are they sent where are they sent where are they sent your traces. AND your traces. AND your traces. AND make sure you make sure you make sure you you really read them, you really read them, you really read them, because it's not worth it just...

because it's not worth it just... If you watch If you watch If you watch only for the table only for the table only for the table results, you results, you results, you you can get you can get you can get indicators that indicators that indicators that look right look right look right or great, but they or great, but they or great, but they can hide can hide can hide fundamental fundamental fundamental problems that problems that problems that actually much actually much actually much more serious and more serious and more serious and larger for larger for larger for your architecture. your architecture. your architecture. So, the solution is never So, the solution is never So, the solution is never does not consist only in does not consist only in does not consist only in in order to do in order to do in order to do Are the numbers higher or lower? Are the numbers higher or lower? Are the numbers higher or lower? better. Needed better. Needed better. Needed create more complex create more complex create more complex tests. Read tests. Read tests. Read tracing, tracing, tracing, find artifacts, find artifacts, find artifacts, improve improve improve testing system and testing system and testing system and start again. start again. start again. Never trust Never trust Never trust own table own table own table results. And yes, this is results. And yes, this is results. And yes, this is just some of the thoughts, just some of the thoughts, just some of the thoughts, which I have had. which I have had. which I have had. You know, models become You know, models become You know, models become only the best. So we only the best. So we only the best. So we we can continue we can continue we can continue believe that models believe that models believe that models will have less will have less will have less variability and variability and variability and will lean more will lean more will lean more to the execution of that to the execution of that to the execution of that work that we are work that we are work that we are We are expecting them. Ago We are expecting them. Ago We are expecting them. Ago the easiest part the easiest part the easiest part may turn out may turn out may turn out creating context creating context creating context and tools. and tools. and tools. The most difficult thing is The most difficult thing is The most difficult thing is build a cycle build a cycle build a cycle assessments, customize assessments, customize assessments, customize monitoring and monitoring and monitoring and observability, and observability, and observability, and also make sure also make sure also make sure that your agent really that your agent really that your agent really does the work that does the work that does the work that you want, for you want, for you want, for with the help of the necessary with the help of the necessary with the help of the necessary tools. So that's all tools. So that's all tools. So that's all it already exists. Evaluation, cycle it already exists. Evaluation, cycle —all this is in —all this is in —all this is in repositories if repositories if repositories if someone wants someone wants someone wants to experiment with to experiment with to experiment with this in my free time. Yes this in my free time. Yes , that's what I'm talking about , that's what I'm talking about , that's what I'm talking about spoke earlier.

spoke earlier. spoke earlier. You know, I'm a little You know, I'm a little You know, I'm a little cheated, cheated, cheated, using Rust. using Rust. using Rust. The Rust compiler is already The Rust compiler is already The Rust compiler is already did part did part did part work for us. I chose work for us. I chose work for us. I chose this language, because the lesson this language, because the lesson this language, because the lesson consisted in the fact that consisted in the fact that consisted in the fact that can be created can be created can be created high-quality engineering high-quality engineering high-quality engineering version of something on version of something on version of something on relatively simple relatively simple relatively simple example problem. example problem. In general, if you In general, if you you can fix it you can fix it you can fix it work for something work for something work for something a simpler example, a simpler example, a simpler example, then scale it to then scale it to then scale it to a bigger task will be a bigger task will be a bigger task will be easier. So, we are now easier. So, we are now easier. So, we are now we can say: "Oh, we can say: "Oh, we can say: "Oh, cool, we did it cool, we did it cool, we did it very easy for Rust." very easy for Rust." very easy for Rust." Now I will do it for Now I will do it for Now I will do it for Python. And Python is just Python. And Python is just Python. And Python is just more complicated, isn't it? more complicated, isn't it? more complicated, isn't it? Are you looking at these? Are you looking at these? Are you looking at these? lines of code and nothing lines of code and nothing lines of code and nothing You won't reach it, right? You won't reach it, right? You won't reach it, right? If you are looking for "client" If you are looking for "client" , you will drown in the sea with , you will drown in the sea with , you will drown in the sea with 50,000 customers. Can 50,000 customers. Can 50,000 customers. Can use use use static analysis for static analysis for static analysis for data loading, data loading, data loading, but it won't be easy. but it won't be easy. but it won't be easy. And this is where the code graph is And this is where the code graph is And this is where the code graph is really justifies really justifies really justifies yourself, right? IN yourself, right? IN yourself, right? IN in such cases when in such cases when in such cases when code bases are larger, and code bases are larger, and code bases are larger, and languages ​​less languages ​​less languages ​​less structured, graph structured, graph structured, graph code even code even code even more efficient than grep, more efficient than grep, more efficient than grep, because for Rust I can because for Rust I can because for Rust I can just look, just look, just look, who implements this who implements this who implements this trait. Okay, great.

trait. Okay, great. trait. Okay, great. I'm just looking for "imple", I'm just looking for "imple", I'm just looking for "imple", truth? It's very easy. truth? It's very easy. truth? It's very easy. But for Python it becomes But for Python it becomes But for Python it becomes much more difficult, much more difficult, much more difficult, isn't that right? Therefore here isn't that right? Therefore here isn't that right? Therefore here need to build need to build need to build something much something much something much more reliable. But more reliable. But more reliable. But graph structure and graph structure and graph structure and structure structure structure tools not tools not tools not must have must have must have change, right? change, right? change, right? Only changes Only changes Only changes the structure of how we the structure of how we the structure of how we we collect and we collect and we collect and we load the data. we load the data. we load the data. If you do this If you do this If you do this modular, then you can modular, then you can modular, then you can scale to any scale to any how many languages ​​or how many languages ​​or how many languages ​​or projects, right? projects, right? projects, right? So, this is a certain basic So, this is a certain basic So, this is a certain basic the structure of how the structure of how the structure of how TypeDaff works: we TypeDaff works: we TypeDaff works: we we are trying to create we are trying to create we are trying to create as much as possible as much as possible as much as possible various processes various processes various processes obtaining data to obtaining data to obtaining data to to collect them from many to collect them from many to collect them from many sources, and then sources, and then sources, and then synthesize everything into synthesize everything into synthesize everything into one whole. There is one more. one whole. There is one more. one whole. There is one more. a notebook for this. I a notebook for this. I a notebook for this. I I won't do it now. I won't do it now. I won't do it now. to consider, because we all to consider, because we all to consider, because we all been here for a while. been here for a while. been here for a while. But don't be shy. But don't be shy. But don't be shy. view it view it view it independently, if independently, if independently, if want to know want to know want to know more. The last thing about more. The last thing about more. The last thing about What I said is, What I said is, What I said is, actually, the level of content actually, the level of content . The structure indicates . The structure indicates . The structure indicates what with what what with what what with what related, and the content related, and the content related, and the content explains why this is so. explains why this is so. explains why this is so. There are comments in the code, There are comments in the code, There are comments in the code, documentation lines documentation lines documentation lines and other information.

and other information. and other information. Integration of the whole Integration of the whole Integration of the whole this information this information this information helps us helps us helps us understand intentions understand intentions understand intentions developers, except developers, except developers, except the code structure itself, the code structure itself, the code structure itself, which exists now. So which exists now. So which exists now. So we want to add we want to add we want to add small steps, small steps, small steps, for example, for for example, for for example, for generalization generalization generalization information from the code and information from the code and information from the code and her understanding line by line her understanding line by line her understanding line by line line by line. So, line by line. So, line by line. So, search can now search can now search can now respond to respond to respond to question what does question what does question what does code, not just where code, not just where code, not just where he is located. AND he is located. AND It is important that when It is important that when creating similar creating similar creating similar things—so-so things—so-so things—so-so audit audit audit semantic analysis semantic analysis , working on top , working on top , working on top deterministic— deterministic— need to bet need to bet need to bet many small ones many small ones many small ones questions, not one questions, not one questions, not one big. Respond big. Respond big. Respond is not about is not about is not about writing one, writing one, writing one, you know...So when we you know...So when we you know...So when we we do analysis in SQL, or we do analysis in SQL, or we do analysis in SQL, or wrong? We don't write. wrong? We don't write. wrong? We don't write. one extremely one extremely one extremely good question, no good question, no good question, no we throw SQL into it and we throw SQL into it and we throw SQL into it and we don't say: "Here it is we don't say: "Here it is we don't say: "Here it is pedantic model of 10 pedantic model of 10 pedantic model of 10 fields, tell me fields, tell me fields, tell me everything you can about this everything you can about this everything you can about this SQL, right? This is not SQL, right? This is not SQL, right? This is not will work. The model will start will work. The model will start will work. The model will start hallucinate, hallucinate, hallucinate, invent invent invent information and will not collect information and will not collect information and will not collect so much data, so much data, so much data, how much could I how much could I Instead of Instead of ask for a model ask for a model ask for a model to do 10 things for to do 10 things for to do 10 things for once, it is better to put once, it is better to put once, it is better to put she has 10 different questions she has 10 different questions she has 10 different questions about 10 smaller ones about 10 smaller ones about 10 smaller ones code snippets to code snippets to code snippets to get specific get specific get specific answers. You answers. You answers. You analyze the structure analyze the structure analyze the structure deterministic, and deterministic, and deterministic, and then bet a lot then bet a lot then bet a lot small, focused small, focused small, focused questions that questions that questions that correspond to the strong correspond to the strong correspond to the strong sides of the model.

sides of the model. sides of the model. Instead of Instead of Instead of ask for a model ask for a model ask for a model look at the whole look at the whole look at the whole SQL block and explain everything SQL block and explain everything SQL block and explain everything columns, you columns, you columns, you parse SQL and parse SQL and parse SQL and you say, "Here is one you say, "Here is one you say, "Here is one column. Here it is. column. Here it is. column. Here it is. type, here are the comments, or type, here are the comments, or type, here are the comments, or Is this an indicator? Is this Is this an indicator? Is this Is this an indicator? Is this measurement? It's all blah-blah- measurement? It's all blah-blah- blah? Then you can blah? Then you can blah? Then you can ask an agent ask an agent ask an agent perform perform perform classification and classification and classification and summing up, and this summing up, and this summing up, and this exactly what he does exactly what he does exactly what he does Okay, right? And if you Okay, right? And if you Okay, right? And if you want to provide more want to provide more want to provide more weights, you give him weights, you give him weights, you give him more tips for more tips for more tips for this one this one this one purposeful purposeful purposeful question. You are not question. You are not question. You are not trying to squeeze in trying to squeeze in trying to squeeze in more questions, you more questions, you more questions, you give him more give him more give him more context so that it context so that it context so that it could answer that could answer that could answer that only one question. Therefore only one question. Therefore , small, , small, , small, justified, justified, justified, verifiable verifiable verifiable semantic operations semantic operations semantic operations over our structure over our structure —that's what it's for —that's what it's for —that's what it's for we need ours we need ours we need ours Fennec library, and Fennec library, and Fennec library, and that's what we are for that's what we are for that's what we are for we use it in we use it in we use it in Type Def. So, the last one Type Def. So, the last one Type Def. So, the last one step on this one step on this one step on this one ladder, where we have everything ladder, where we have everything ladder, where we have everything structured, we structured, we structured, we we know how everything is we know how everything is we know how everything is connected to each other. connected to each other. connected to each other. Now, what is this? Now, what is this? Now, what is this? does it really mean? does it really mean? does it really mean? Now we can Now we can Now we can start receiving start receiving start receiving semantic understanding semantic understanding semantic understanding that all this that all this that all this means, means, means, using LLM on using LLM on using LLM on the last level. Yes the last level. Yes , as I said, here , as I said, here , as I said, here you can do a lot you can do a lot you can do a lot what if you really what if you really what if you really want to create want to create want to create agent for analysis agent for analysis agent for analysis code graph. You can code graph. You can code graph. You can use use use call graphs and SCIP, call graphs and SCIP, call graphs and SCIP, to understand who to understand who to understand who What causes this and who?

What causes this and who? What causes this and who? refers to this, or refers to this, or refers to this, or wrong? And he can wrong? And he can wrong? And he can read comments read comments read comments inside and outside inside and outside inside and outside code, and can also code, and can also code, and can also to involve description from to involve description from to involve description from external sources. He external sources. He external sources. He may involve may involve may involve problems with GitHub, your problems with GitHub, your problems with GitHub, your tasks in Linear, tasks in Linear, tasks in Linear, tickets, etc. This is how you tickets, etc. This is how you tickets, etc. This is how you turn this into turn this into turn this into something useful and something useful and something useful and justified: you justified: you justified: you start small, with start small, with start small, with the smallest possible the smallest possible the smallest possible step, namely—how do I step, namely—how do I step, namely—how do I extract the structure from extract the structure from extract the structure from of this codebase. AND of this codebase. AND of this codebase. AND then you move on then you move on then you move on before taking this before taking this before taking this structure and add to structure and add to structure and add to its semantic its semantic its semantic understanding that understanding that understanding that will really help will really help will really help build a substrate, build a substrate, build a substrate, useful for agents, useful for agents, useful for agents, so that they could so that they could so that they could to work in this to work in this to work in this spacious and contribute spacious and contribute spacious and contribute changes. And this, in fact, changes. And this, in fact, changes. And this, in fact, the same angle of view. the same angle of view. So, in any So, in any Thank you very much, Thank you very much, Thank you very much, guys. This was mine. guys. This was mine. guys. This was mine. report "Agent of everything" report "Agent of everything" report "Agent of everything" "I will understand." "I will understand." "I will understand." Brendan Calendar from Type Brendan Calendar from Type Brendan Calendar from Type Dev. Here is the link to Dev. Here is the link to Dev. Here is the link to workshop and my LinkedIn. workshop and my LinkedIn. workshop and my LinkedIn. So, thank you. I am ready. So, thank you. I am ready. So, thank you. I am ready. respond to respond to respond to question whether question whether question whether to discuss other things, to discuss other things, to discuss other things, but thank you very much everyone. but thank you very much everyone.

Have a question? You can Have a question? You can pass the microphone pass the microphone pass the microphone for one for one for one last lap last lap last lap questions. questions. Have you tried this? Have you tried this? workflow in workflow in workflow in various repositories, various repositories, various repositories, where can these be where can these be where can these be microservices or microservices or microservices or interaction via HTTP interaction via HTTP boundaries, and was it boundaries, and was it boundaries, and was it successfully? successfully? Well, not particularly, no. I think where it could be I think where it could be need, then it is in need, then it is in need, then it is in within one within one within one repository. We repository. We repository. We we do it in big ones we do it in big ones we do it in big ones on a scale for our on a scale for our on a scale for our product from product from product from data architecture. data architecture. data architecture. So, I think this is So, I think this is So, I think this is can work. You can work. You can work. You actually need— actually need— the most difficult thing here, the most difficult thing here, the most difficult thing here, of course, to determine of course, to determine of course, to determine boundaries between different boundaries between different boundaries between different components. You components. You components. You need to write need to write need to write boundary classifiers, boundary classifiers, boundary classifiers, who are trying who are trying who are trying understand...I wanted to understand...I wanted to understand...I wanted to give an example from Python give an example from Python give an example from Python and Rust, like with Maturin, and Rust, like with Maturin, and Rust, like with Maturin, could be done could be done could be done something like that, but I don't something like that, but I don't something like that, but I don't managed. But you managed. But you managed. But you need to create need to create need to create way of understanding way of understanding way of understanding when the code is in Rust when the code is in Rust when the code is in Rust calls Python or calls Python or calls Python or on the contrary, right? That on the contrary, right? That on the contrary, right? That build this build this build this level of communication between level of communication between level of communication between them. So, yes, you them. So, yes, you them. So, yes, you could do it.

could do it. could do it. I just think that this I just think that this I just think that this would require a lot would require a lot would require a lot more work. more work. Any more questions? That's fine. That's fine. Sorry. Sorry. Sorry. No, I understand. No, I understand. No, I understand. How does this compare to How does this compare to How does this compare to existing Harness? existing Harness? Cursor does a lot Cursor does a lot second-level things. second-level things. second-level things. When you see Cursor When you see Cursor When you see Cursor searching in your code searching in your code searching in your code base, he usually base, he usually base, he usually performs semantic performs semantic performs semantic search, for example, search, for example, search, for example, vector search by vector search by vector search by embeddings. And this embeddings. And this embeddings. And this really works very well really works very well really works very well good. I think it's not in vain. good. I think it's not in vain. good. I think it's not in vain. so many appeared so many appeared so many appeared products with code products with code products with code RAGs that do RAGs that do RAGs that do semantic search semantic search semantic search or data retrieval, or data retrieval, or data retrieval, Yes? And I think that this Yes? And I think that this Yes? And I think that this very cool Simply very cool Simply very cool Simply this approach gives this approach gives this approach gives opportunity to find opportunity to find opportunity to find all the places in the code that all the places in the code that all the places in the code that you want later you want later you want later to process by an agent, to process by an agent, to process by an agent, but you don't care but you don't care but you don't care has to has to has to check these places check these places check these places agent, right? agent, right? The purpose of this approach The purpose of this approach is to is to is to to be able to say yes to be able to say yes to be able to say yes once: I know that once: I know that once: I know that I'm looking at this one. I'm looking at this one. I'm looking at this one. function, you understand? I function, you understand? I function, you understand? I I want to have the opportunity.

I want to have the opportunity. I want to have the opportunity. click in the IDE to click in the IDE to click in the IDE to to see all its places to see all its places to see all its places challenge, but challenge, but challenge, but inside my inside my inside my agent, without agent, without agent, without agent's need agent's need agent's need to search everywhere myself. to search everywhere myself. to search everywhere myself. Does this make sense? Does this make sense? Any more questions? Any more questions? Okay, one thing here. Okay, one thing here. Okay, one thing here. One second. One second. Mm. Mm. I imagine that part I imagine that part I imagine that part pleasure from pleasure from pleasure from creating this creating this creating this is that you is that you is that you you get a kind of you get a kind of you get a kind of vector of the graph, which vector of the graph, which vector of the graph, which can be called. Me can be called. Me can be called. Me very interesting about very interesting about very interesting about caching and caching and caching and storage when you storage when you storage when you you start thinking about you start thinking about you start thinking about scaling, scaling, scaling, because confidence because confidence because confidence in that the agent in that the agent in that the agent will do everything will do everything will do everything right from the start right from the start right from the start challenge, is key challenge, is key challenge, is key for control. for control. I think that when you I think that when you you start you start you start scale your scale your scale your graph database, graph database, graph database, you will quickly notice, you will quickly notice, you will quickly notice, anything can happen anything can happen anything can happen very confusing, and you very confusing, and you very confusing, and you Absolutely right. So Absolutely right. So Absolutely right. So I believe that I believe that I believe that data storage—this data storage—this data storage—this less of a problem. For less of a problem. For less of a problem. For yes, these graphs are inclined yes, these graphs are inclined yes, these graphs are inclined grow, grow, grow, especially over time, and their especially over time, and their especially over time, and their harder to clean harder to clean harder to clean garbage, but I am less garbage, but I am less garbage, but I am less I worry about storage, I worry about storage, I worry about storage, and more— and more— computational computational computational capacities, capacities, capacities, necessary for necessary for necessary for graph analysis through graph analysis through graph analysis through all these different elements all these different elements . Therefore, trying . Therefore, trying . Therefore, trying scale the system scale the system , the main thing, on what , the main thing, on what , the main thing, on what it is worth concentrating, it is worth concentrating, it is worth concentrating, as I said, it is as I said, it is as I said, it is incrementality, incrementality, incrementality, that is, finding ways that is, finding ways that is, finding ways update without update without update without full recalculation.

full recalculation. I think this is I think this is the most important part the most important part with which we at Type Dev with which we at Type Dev with which we at Type Dev we fight at we fight at we fight at building systems building systems building systems large scale, large scale, large scale, because you don't want to because you don't want to because you don't want to always contact always contact always contact your data warehouse your data warehouse your data warehouse 500 times in a row, 500 times in a row, 500 times in a row, truth? You don't want to truth? You don't want to truth? You don't want to every time anew every time anew every time anew check and check and check and compile all SQL- compile all SQL- code, right? You code, right? You code, right? You do you want to have do you want to have do you want to have possibility possibility possibility update data in update data in update data in as they arrive. as they arrive. as they arrive. Therefore the best Therefore the best Therefore the best way to avoid way to avoid way to avoid this—to design this—to design this—to design system with system with system with counting on counting on counting on incremental incremental incremental indexation as per indexation as per indexation as per priority priority priority task; is it or not task; is it or not task; is it or not the only way to the only way to the only way to solving others solving others solving others problems problems problems scaling. You scaling. You scaling. You must figure it out must figure it out must figure it out with a problem with a problem with a problem repeated repeated repeated synchronization synchronization synchronization information for a long time information for a long time information for a long time before before before you will face your own you will face your own you will face your own problem problem problem scaling scaling scaling storage or computing. storage or computing. Yes. I found that for Yes. I found that for us creation and us creation and us creation and support for the reference support for the reference support for the reference dataset—this is dataset—this is dataset—this is always the biggest always the biggest always the biggest problem.

problem. Yes. I I agree with you. Yes. I I agree with you. 100%, and that's what 100%, and that's what 100%, and that's what what do I think I have what do I think I have what do I think I have there is no great there is no great there is no great answers. Because I answers. Because I answers. Because I I think I think I have I think I think I have I think I think I have two answers to this, two answers to this, two answers to this, actually. One of them actually. One of them actually. One of them is that is that reference set reference set data not required data not required if I were building this myself if I were building this myself and tested on something, and tested on something, and tested on something, I would choose a specific one. I would choose a specific one. I would choose a specific one. codebase snapshot, codebase snapshot, codebase snapshot, True? So I could True? So I could True? So I could to say that it is mine to say that it is mine to say that it is mine snapshot, and when I snapshot, and when I snapshot, and when I I run assessments, in I run assessments, in I run assessments, in I have it. I have it. I have it. deterministic set deterministic set deterministic set control questions control questions , do you understand? This is a bit , do you understand? This is a bit , do you understand? This is a bit evasive response to evasive response to evasive response to your question, your question, your question, truth? But that's just how I am. truth? But that's just how I am. truth? But that's just how I am. would do if would do if would do if designed this designed this designed this really, right? I would really, right? I would really, right? I would designed it on designed it on designed it on based on a snapshot, based on a snapshot, based on a snapshot, so that my questions are for so that my questions are for so that my questions are for the grades were the grades were the grades were deterministic and not deterministic and not deterministic and not changed over time, changed over time, changed over time, Do you understand? Because in Do you understand? Because in Do you understand? Because in in a perfect world, yes in a perfect world, yes ? Your agent will become ? Your agent will become ? Your agent will become all the best in all the best in all the best in answers to the same answers to the same answers to the same question, but change question, but change question, but change questions through questions through questions through hull update hull update hull update data will not help you data will not help you data will not help you to assess whether they are becoming to assess whether they are becoming to assess whether they are becoming your agents better, your agents better, your agents better, truth? Best truth? Best truth? Best a way to assess whether a way to assess whether a way to assess whether become your agents become your agents become your agents better—is to force better—is to force better—is to force them to do the same them to do the same them to do the same task again, task again, task again, truth? Respond truth? Respond truth? Respond like this: you don't do this like this: you don't do this like this: you don't do this do it because for the purposes do it because for the purposes do it because for the purposes your assessment your assessment your assessment the standard should be the standard should be the standard should be unchanged regarding unchanged regarding unchanged regarding unchanging body unchanging body unchanging body data. This sounds data. This sounds data. This sounds logically? Of course, if logically? Of course, if logically? Of course, if you are testing live, you are testing live, you are testing live, you need you need you need find new ones find new ones find new ones ways to check for ways to check for ways to check for your codebase, your codebase, your codebase, but I think that this is but I think that this is but I think that this is your fixed dial your fixed dial your fixed dial for evaluation—various for evaluation—various for evaluation—various things. Is that clear?

things. Is that clear? things. Is that clear? Yes, but...Yes, but I Yes, but...Yes, but I Yes, but...Yes, but I I mean, if we I mean, if we I mean, if we let's take an example from let's take an example from let's take an example from code: suppose, in code: suppose, in code: suppose, in you have a complex error you have a complex error , there is a function that , there is a function that , there is a function that causes another, and in causes another, and in causes another, and in one of them has a bug or one of them has a bug or one of them has a bug or condition, as you said, and condition, as you said, and condition, as you said, and do you have a test for do you have a test for do you have a test for ratings: if this bug ratings: if this bug ratings: if this bug implemented, agent implemented, agent implemented, agent should find this function should find this function should find this function and fix it. But and fix it. But and fix it. But then you do then you do then you do refactoring, and this refactoring, and this refactoring, and this function more function more function more not at all not at all not at all is used. is used. is used. So, the test for evaluation So, the test for evaluation So, the test for evaluation breaks. breaks. breaks. Oh, now I understand, Oh, now I understand, Oh, now I understand, What are you talking about? In such a What are you talking about? In such a What are you talking about? In such a In this case, I believe that my In this case, I believe that my In this case, I believe that my respond respond respond remains remains remains unchanged, because, as I unchanged, because, as I unchanged, because, as I I already said when I I already said when I I already said when I I create an estimate, I I create an estimate, I I create an estimate, I I tie her to I tie her to I tie her to a specific commit, a specific commit, a specific commit, on which it is located on which it is located on which it is located my code, right? my code, right? my code, right? The specific diff that The specific diff that The specific diff that I want to implement, I want to implement, I want to implement, to check whether to check whether to check whether can his agent can his agent can his agent to fix. And then— to fix. And then— resulting set resulting set resulting set changes after that changes after that changes after that prompt, right? This has prompt, right? This has prompt, right? This has always stay always stay always stay unchangeable. I understand.

unchangeable. I understand. unchangeable. I understand. your opinion, because yes, your opinion, because yes, your opinion, because yes, the code will change and it the code will change and it the code will change and it no longer needed no longer needed no longer needed will fix that will fix that will fix that problem, but you problem, but you problem, but you This is not what you are testing. This is not what you are testing. This is not what you are testing. truth? You are testing truth? You are testing truth? You are testing inability of the agent inability of the agent inability of the agent react to a new react to a new react to a new code, you are testing code, you are testing code, you are testing agent ability agent ability agent ability make that change in those make that change in those make that change in those the circumstances that existed, the circumstances that existed, the circumstances that existed, when did you write the test when did you write the test when did you write the test for evaluation, right? It for evaluation, right? It for evaluation, right? It task, I think, not task, I think, not task, I think, not changed, even changed, even changed, even if it is no longer if it is no longer if it is no longer relevant. It's all relevant. It's all relevant. It's all just checking something just checking something just checking something useful. Does this make sense? useful. Does this make sense? Good question. Wonderful report. Me Wonderful report. Me I really liked it. Uh, I really liked it. Uh, I really liked it. Uh, this reminds me this reminds me this reminds me ontology. Classes, and this ontology. Classes, and this ontology. Classes, and this what Palantir is doing now what Palantir is doing now what Palantir is doing now earns crazy money earns crazy money earns crazy money money. money. money. Yes. Yes. Yes. So, uh, after So, uh, after So, uh, after ontology construction ontology construction ontology construction and dependencies, or and dependencies, or and dependencies, or can we now can we now can we now communicate with LLM at communicate with LLM at communicate with LLM at with the help of these higher with the help of these higher with the help of these higher structures? structures? structures? Yes. Yes. Yes. With the help of these With the help of these With the help of these classes. classes. classes. Yes. Yes. Yes. And this is, in fact, what And this is, in fact, what And this is, in fact, what you hinted at this you hinted at this you hinted at this moment?

moment? moment? Yes. This, in essence, is Yes. This, in essence, is Yes. This, in essence, is our product, right? our product, right? ? This is a concept, according to ? This is a concept, according to ? This is a concept, according to with which we build with which we build with which we build context graph, we context graph, we context graph, we we build a layer we build a layer we build a layer ontologies floor ontologies floor ontologies floor him, where are we him, where are we him, where are we we are trying we are trying we are trying to organize everything to organize everything to organize everything information about your information about your information about your data architecture: data architecture: data architecture: where everything points, where where everything points, where where everything points, where everything is saved, that everything is saved, that everything is saved, that this means, and then we this means, and then we this means, and then we we provide to the agent we provide to the agent we provide to the agent access to this access to this access to this tool kit, tool kit, tool kit, so that he could so that he could so that he could to answer to answer to answer questions and make questions and make questions and make changes in your changes in your changes in your data architecture. data architecture. OK. Maybe this OK. Maybe this will sound a little will sound a little will sound a little naively, but naively, but naively, but returning to returning to returning to the product you the product you the product you create, and what create, and what create, and what you are here you are here you are here demonstrated. demonstrated. demonstrated. Much of what you Much of what you Much of what you showed, very similar showed, very similar showed, very similar on the fact that Neo4j on the fact that Neo4j on the fact that Neo4j demonstrated on demonstrated on demonstrated on previous previous previous seminar, that is: here seminar, that is: here seminar, that is: here knowledge graph. You are like knowledge graph. You are like knowledge graph. You are like developer from developer from developer from subject subject subject you build with expertise you build with expertise you build with expertise knowledge graph in this knowledge graph in this knowledge graph in this industry, provide it industry, provide it industry, provide it agent, and then agent, and then agent, and then forcing this forcing this forcing this agent agent agent use use use Cypher requests for Cypher requests for Cypher requests for obtaining a certain obtaining a certain obtaining a certain semantic semantic semantic information. I information. I information. I I am responsible for, I am responsible for, I am responsible for, let's say it like this, big let's say it like this, big let's say it like this, big branched array branched array branched array data, where there is a lot data, where there is a lot data, where there is a lot third-party SaaS solutions.

third-party SaaS solutions. third-party SaaS solutions. We have others too We have others too We have others too internal systems. internal systems. internal systems. My team My team My team is responsible for collecting is responsible for collecting is responsible for collecting a large number a large number a large number information from these information from these information from these sources. Some of them, sources. Some of them, sources. Some of them, like Salesforce, very like Salesforce, very like Salesforce, very flexible in flexible in flexible in settings. You settings. You settings. You you collect information, you collect information, you collect information, place it in place it in place it in data warehouses, etc., and data warehouses, etc., and data warehouses, etc., and eventually eventually eventually you get business you get business analytics, as well as analytics, as well as analytics, as well as interactions. So the goal interactions. So the goal interactions. So the goal like this: how can I make like this: how can I make like this: how can I make my small team my small team my small team qualified qualified qualified specialists to determine specialists to determine specialists to determine semantic connections semantic connections semantic connections between all this in one between all this in one between all this in one place, at one time, with place, at one time, with place, at one time, with under proper control under proper control under proper control versions so that later I versions so that later I versions so that later I could provide could provide could provide context and context and context and tools for AI tools for AI agents mainly agents mainly agents mainly for analytical for analytical for analytical work, but also work, but also work, but also sometimes for recording sometimes for recording sometimes for recording data back, as well as data back, as well as data back, as well as create create create documentation for documentation for documentation for people who have people who have people who have use these use these use these systems, whether for systems, whether for systems, whether for adaptation, or for adaptation, or for adaptation, or for work in one place work in one place . I haven't found it yet. . I haven't found it yet. . I haven't found it yet. the right way the right way the right way do it. do it. do it. Let's Let's Let's Let's chat. Let's chat. Let's chat. So you are going So you are going So you are going give me this because you give me this because you give me this because you also said that no also said that no also said that no will you do will you do will you do data catalog. data catalog. data catalog. Right? And you Right? And you Right? And you said said said Well, the thing is, I Well, the thing is, I Well, the thing is, I I think I want to say, I think I want to say, I think I want to say, that we are not going that we are not going that we are not going to limit oneself to limit oneself to limit oneself data catalog, data catalog, data catalog, because we will not because we will not because we will not read only yours read only yours read only yours Snowflake catalog, only, Snowflake catalog, only, Snowflake catalog, only, you know, just you know, just you know, just your information your information your information storage. The goal is storage. The goal is storage. The goal is in order to read in order to read in order to read data from your Salesforce, from data from your Salesforce, from data from your Salesforce, from your Fivetran, from your your Fivetran, from your your Fivetran, from your Airbyte, from anything, Airbyte, from anything, Airbyte, from anything, from all mechanisms from all mechanisms from all mechanisms the transmissions that you have, the transmissions that you have, the transmissions that you have, your Airflow or any your Airflow or any What else?

What else? Combine all this into Combine all this into the only clear thing the only clear thing the only clear thing context tree, context tree, context tree, context graph, context graph, context graph, who we really are who we really are who we really are we can move. we can move. we can move. And then, from there, the goal And then, from there, the goal And then, from there, the goal is to is to is to to highlight from this to highlight from this to highlight from this those things that you those things that you those things that you talked. Home talked. Home talked. Home semantic problem semantic problem semantic problem layers in that no one layers in that no one layers in that no one does not want to support does not want to support does not want to support semantic layer, or semantic layer, or semantic layer, or wrong? This is a kind of wrong? This is a kind of wrong? This is a kind of a swear word, therefore a swear word, therefore a swear word, therefore what if you do it what if you do it what if you do it you build, it's all very you build, it's all very you build, it's all very cool and classy, ​​but cool and classy, ​​but cool and classy, ​​but then you have to then you have to then you have to constantly update constantly update constantly update him every time you him every time you him every time you you add models, and that's it, you add models, and that's it, you add models, and that's it, Well, this is real. Well, this is real. Well, this is real. nightmare. So, the goal nightmare. So, the goal nightmare. So, the goal our Typedef is our Typedef is our Typedef is so that you really so that you really so that you really just...It was just...It was just...It was perfectly. Thank you for that. perfectly. Thank you for that. perfectly. Thank you for that. that gave a wonderful that gave a wonderful that gave a wonderful understanding that understanding that understanding that that's what we want that's what we want that's what we want the product could do, and the product could do, and the product could do, and this is exactly what you are talking about this is exactly what you are talking about this is exactly what you are talking about said. It said. It said. It opportunity to reason opportunity to reason opportunity to reason about all those different about all those different about all those different the levels you have in the levels you have in the levels you have in to its architecture to its architecture to its architecture data, build data, build data, build the only graph by which the only graph by which the only graph by which can be tracked as can be tracked as can be tracked as data moves from data moves from data moves from Salesforce all the way to yours Salesforce all the way to yours Salesforce all the way to yours BI level, right? AND BI level, right? AND BI level, right? AND then when your agent then when your agent then when your agent asks when you asks when you asks when you put your own put your own put your own to the agent a question like: " to the agent a question like: " Hey, where is it from? Hey, where is it from? Hey, where is it from? my receivable my receivable my receivable debt?""He debt?""He debt?""He can say: "Oh yes, can say: "Oh yes, can say: "Oh yes, perfectly. This comes from perfectly. This comes from perfectly. This comes from from here, from the data from here, from the data from here, from the data Stripe, actually, a Stripe, actually, a Stripe, actually, a then it then it then it joins joins joins your Salesforce data here your Salesforce data here your Salesforce data here down below, so that, you know, down below, so that, you know, down below, so that, you know, figure this thing out figure this thing out figure this thing out down here, where it actually is down here, where it actually is down here, where it actually is data is being calculated " data is being calculated " .

. . Yes, this pedigree is very Yes, this pedigree is very Yes, this pedigree is very useful, but for us useful, but for us useful, but for us this is, you know, partly this is, you know, partly this is, you know, partly for the data command, for the data command, for the data command, which deals with this which deals with this which deals with this development, great. development, great. development, great. But I want this But I want this But I want this eventually became eventually became eventually became available for available for available for business, right? business, right? business, right? So, they can So, they can So, they can look at yours look at yours look at yours report and such: "This is not true report and such: "This is not true . What the hell? "" And . What the hell? "" And . What the hell? "" And instead of instead of instead of come to mine come to mine come to mine commands and shout at commands and shout at commands and shout at me, they can me, they can me, they can say: "Oh, well, that's say: "Oh, well, that's say: "Oh, well, that's actually belongs to actually belongs to actually belongs to to these people in this to these people in this to these people in this system and in this system and in this system and in this record, and, like, this is that record, and, like, this is that record, and, like, this is that the person I am with the person I am with the person I am with "We need to talk." " "We need to talk." " So keep all of this. So keep all of this. So keep all of this. together. together. together. I agree with you 100%. I agree with you 100%. I agree with you 100%. I think one of I think one of I think one of things that LLM allows things that LLM allows things that LLM allows we have to do on we have to do on we have to do on high level, what high level, what high level, what previously was previously was previously was impossible to achieve— impossible to achieve— it is to be the same it is to be the same it is to be the same level of translation between level of translation between level of translation between business users business users , accountants, , accountants, , accountants, analysts, right? AND analysts, right? AND analysts, right? AND then date- then date- analysts, analysts, analysts, engineers or any engineers or any Who is like that, right? For Who is like that, right? For Who is like that, right? For we want to say:" we want to say:" we want to say:" Okay, business. Okay, business. the analyst says, 'Hey, the analyst says, 'Hey, the analyst says, 'Hey, this is the number on the panel this is the number on the panel this is the number on the panel income is incorrect, income is incorrect, income is incorrect, Yes?' "" So the process Yes?' "" So the process Yes?' "" So the process correction for them correction for them correction for them would be like this ":" Okay, would be like this ":" Okay, would be like this ":" Okay, I need somewhere. I need somewhere. I need somewhere. create a ticket". He create a ticket". He create a ticket". He hangs somewhere in hangs somewhere in hangs somewhere in system. Someone him system. Someone him system. Someone him picks up, sorts, picks up, sorts, picks up, sorts, tries tries tries find out who find out who find out who responsible for this, responsible for this, responsible for this, Yes? Sends it Yes? Sends it Yes? Sends it to this person. Them to this person. Them to this person. Them has to has to has to communicate with communicate with communicate with original, so this original, so this original, so this madness, right? I madness, right? I madness, right? I worked at Salesforce, worked at Salesforce, worked at Salesforce, so I understand this a little bit so I understand this a little bit so I understand this a little bit at a high level.

at a high level. at a high level. So, the purpose of Typedef is to give So, the purpose of Typedef is to give So, the purpose of Typedef is to give business opportunity business opportunity analytics simply analytics simply analytics simply to come in and say, "Hey, to come in and say, "Hey, to come in and say, "Hey, my dashboard my dashboard my dashboard broken, right? "They broken, right? "They broken, right? "They can enter the name can enter the name can enter the name panels. They can panels. They can panels. They can open the panel in that open the panel in that open the panel in that the session itself, where they the session itself, where they the session itself, where they are located. We are located. We are located. We we can trace we can trace we can trace back and say, "Oh, back and say, "Oh, back and say, "Oh, yes, I think I see, yes, I think I see, yes, I think I see, What's the problem here?" What's the problem here?" What's the problem here?" This is probably This is probably This is probably takes place on takes place on takes place on a few levels earlier. a few levels earlier. a few levels earlier. I can determine that I can determine that I can determine that this team owns this team owns this team owns this model right here. this model right here. this model right here. I can create it myself. I can create it myself. I can create it myself. ticket for you from now on ticket for you from now on ticket for you from now on added information added information added information about the solution, because I about the solution, because I about the solution, because I I see the root cause, I see the root cause, I see the root cause, because I looked because I looked