Citation Needed: Provenance for LLM-Built Knowledge Graphs — Daniel Chalef, Zep AI

So, LLMs are really great at pulling So, LLMs are really great at pulling together data from many sources. Uh, but together data from many sources. Uh, but together data from many sources. Uh, but they do so non sorry, they do so they do so non sorry, they do so they do so non sorry, they do so non-deterministically. non-deterministically. non-deterministically. They interpret and synthesize data They interpret and synthesize data They interpret and synthesize data generating a summary, an extracted fact, generating a summary, an extracted fact, generating a summary, an extracted fact, uh a structured record. And this output uh a structured record. And this output uh a structured record. And this output artifact may not appear verbatim in the artifact may not appear verbatim in the artifact may not appear verbatim in the source inputs. source inputs. source inputs. Synthesis often destroys the paper trail Synthesis often destroys the paper trail Synthesis often destroys the paper trail of how these outputs were originated. of how these outputs were originated. of how these outputs were originated. And I'm going to be talking today about And I'm going to be talking today about And I'm going to be talking today about provenence, which is tracing how an provenence, which is tracing how an provenence, which is tracing how an artifact was built and why. Legal artifact was built and why. Legal artifact was built and why. Legal compliance often demands provenence, but compliance often demands provenence, but compliance often demands provenence, but it's also useful for debugging. Deciding it's also useful for debugging. Deciding it's also useful for debugging. Deciding which sources you trust and which which sources you trust and which which sources you trust and which artifacts to delete. And solving this at artifacts to delete. And solving this at artifacts to delete. And solving this at scale presents a real engineering scale presents a real engineering scale presents a real engineering challenge. If I can get my next slide going here. If I can get my next slide going here. So my team and I built Graffiti uh the So my team and I built Graffiti uh the So my team and I built Graffiti uh the open-source temporal graph framework and open-source temporal graph framework and open-source temporal graph framework and Zep, our enterprise agent memory Zep, our enterprise agent memory Zep, our enterprise agent memory infrastructure is built on graffiti. Our infrastructure is built on graffiti. Our infrastructure is built on graffiti. Our customers derive context or agent memory customers derive context or agent memory customers derive context or agent memory for many user touch points. Those could for many user touch points. Those could for many user touch points. Those could be chat, but not only chat. Often it's be chat, but not only chat. Often it's be chat, but not only chat. Often it's voice transcripts, email, business data.

voice transcripts, email, business data. voice transcripts, email, business data. And our customers have struggled with And our customers have struggled with And our customers have struggled with provenence. Where did this fact come provenence. Where did this fact come provenence. Where did this fact come from? What is the veracity of this fact? from? What is the veracity of this fact? from? What is the veracity of this fact? And over the next few slides, I'll share And over the next few slides, I'll share And over the next few slides, I'll share how we engineered solutions to this how we engineered solutions to this how we engineered solutions to this problem. problem. problem. So here's an stylized failure mode. An So here's an stylized failure mode. An So here's an stylized failure mode. An agent retrieves context about a patient. agent retrieves context about a patient. agent retrieves context about a patient. So this is a healthcare scenario. So this is a healthcare scenario. So this is a healthcare scenario. And what comes back is a clean confident And what comes back is a clean confident And what comes back is a clean confident fact. Patient has a penicellin allergy. fact. Patient has a penicellin allergy. fact. Patient has a penicellin allergy. And the context was synthesized from And the context was synthesized from And the context was synthesized from three sources. A lengthy EHR record, three sources. A lengthy EHR record, three sources. A lengthy EHR record, electronic health record, a PDF lab electronic health record, a PDF lab electronic health record, a PDF lab report, report, report, and something a patient typed into an AI and something a patient typed into an AI and something a patient typed into an AI intake chat. intake chat. intake chat. If the agent presents the fact to a If the agent presents the fact to a If the agent presents the fact to a doctor in treat in a treatment scenario doctor in treat in a treatment scenario doctor in treat in a treatment scenario without clearly indicating the source without clearly indicating the source without clearly indicating the source was from the patient themselves, it may was from the patient themselves, it may was from the patient themselves, it may mislead the doctor. When an agent mislead the doctor. When an agent mislead the doctor. When an agent retrieves context, can we point to the retrieves context, can we point to the retrieves context, can we point to the exact source and its veracity?

exact source and its veracity? exact source and its veracity? For complex agent applications, the For complex agent applications, the For complex agent applications, the answer is often no. So, I can imagine you're probably So, I can imagine you're probably thinking, thinking, thinking, um, but can't we just store like a um, but can't we just store like a um, but can't we just store like a source ID on the fact? source ID on the fact? source ID on the fact? This can work well in structured data This can work well in structured data This can work well in structured data warehouses or data lakes. A pipeline warehouses or data lakes. A pipeline warehouses or data lakes. A pipeline outputs one value copied or mutated outputs one value copied or mutated outputs one value copied or mutated deterministically, deterministically, deterministically, and the sources are known and easily and the sources are known and easily and the sources are known and easily marked. But with context pipelines run marked. But with context pipelines run marked. But with context pipelines run by LLMs, this breaks in several ways. by LLMs, this breaks in several ways. by LLMs, this breaks in several ways. You prompt an LLM with several sources. You prompt an LLM with several sources. You prompt an LLM with several sources. Many facts are each synthesized from one Many facts are each synthesized from one Many facts are each synthesized from one or more of the sources. or more of the sources. or more of the sources. Somebody like J. Smith and John Smith Somebody like J. Smith and John Smith Somebody like J. Smith and John Smith are merged into a single entity, one are merged into a single entity, one are merged into a single entity, one identity. And John's facts are derived identity. And John's facts are derived identity. And John's facts are derived from many different places. from many different places. from many different places. So new data might invalidate old facts. So new data might invalidate old facts. So new data might invalidate old facts. The store keeps changing underneath your The store keeps changing underneath your The store keeps changing underneath your pointer and an appendon log which often pointer and an appendon log which often pointer and an appendon log which often might you know might come to mind here might you know might come to mind here might you know might come to mind here gets very hard to manage at scale as gets very hard to manage at scale as gets very hard to manage at scale as there's so many changes occurring. So there's so many changes occurring. So there's so many changes occurring. So lineage needs to be an evolving set and lineage needs to be an evolving set and lineage needs to be an evolving set and survive mutation.

So sets of links between facts and their So sets of links between facts and their sources can be modeled on a graph as sources can be modeled on a graph as sources can be modeled on a graph as relationships. relationships. relationships. So provenence in a context store So provenence in a context store So provenence in a context store containing facts is a knowledge graph. containing facts is a knowledge graph. containing facts is a knowledge graph. We have three in this example we have We have three in this example we have We have three in this example we have three source data in graffiti that three source data in graffiti that three source data in graffiti that turned episodes. We have two entities turned episodes. We have two entities turned episodes. We have two entities extracted from the episodes patient and extracted from the episodes patient and extracted from the episodes patient and penicellin penicellin penicellin and an edge between them. and an edge between them. and an edge between them. This graph triple the two entities and This graph triple the two entities and This graph triple the two entities and the edge can be hydrated as a fact. the edge can be hydrated as a fact. the edge can be hydrated as a fact. Patient has a penicellin allergy. Patient has a penicellin allergy. Patient has a penicellin allergy. Tracing a fact to its source is just a Tracing a fact to its source is just a Tracing a fact to its source is just a graph walk. So it's pretty simple and easy to map So it's pretty simple and easy to map source to fact on the first right source to fact on the first right source to fact on the first right but keeping it correct while the graph but keeping it correct while the graph but keeping it correct while the graph changes can be really hard when new data changes can be really hard when new data changes can be really hard when new data uh so for example when two entities uh so for example when two entities uh so for example when two entities merge the merged entity needs to keep merge the merged entity needs to keep merge the merged entity needs to keep all source links from both otherwise we all source links from both otherwise we all source links from both otherwise we silently drop a source and we lose silently drop a source and we lose silently drop a source and we lose lineage.

lineage. lineage. And when new data contradicts existing And when new data contradicts existing And when new data contradicts existing data, mutating it, we need to capture data, mutating it, we need to capture data, mutating it, we need to capture this lineage too. this lineage too. this lineage too. In the rightmost card, a fact is In the rightmost card, a fact is In the rightmost card, a fact is rendered invalid by new data. And in rendered invalid by new data. And in rendered invalid by new data. And in graffiti, an invalid date is added graffiti, an invalid date is added graffiti, an invalid date is added to the mutated edge. And the source to the mutated edge. And the source to the mutated edge. And the source episodes that resulted in the edge episodes that resulted in the edge episodes that resulted in the edge mutating mutating mutating are noted against the fact. So gaining graffiti, the relationship So gaining graffiti, the relationship between source data and derived between source data and derived between source data and derived artifacts such as facts is easily artifacts such as facts is easily artifacts such as facts is easily modeled on the graph. modeled on the graph. modeled on the graph. With metadata projection, we can also With metadata projection, we can also With metadata projection, we can also model classifications that span many model classifications that span many model classifications that span many different episodes. and facts derived different episodes. and facts derived different episodes. and facts derived from them. And so I'll give you an from them. And so I'll give you an from them. And so I'll give you an example here. example here. example here. In in the prior healthc care scenario, In in the prior healthc care scenario, In in the prior healthc care scenario, episodes may originate from an EHR episodes may originate from an EHR episodes may originate from an EHR record and have an EHR tag associate record and have an EHR tag associate record and have an EHR tag associate associated with them, but not all associated with them, but not all associated with them, but not all records are. And so on ingestion, we tag records are. And so on ingestion, we tag records are. And so on ingestion, we tag the episodes with the EHR tag. All the episodes with the EHR tag. All the episodes with the EHR tag. All subsequent entities and facts derived subsequent entities and facts derived subsequent entities and facts derived from the episode inherit the tag. And so from the episode inherit the tag. And so from the episode inherit the tag. And so if the agent wants to retrieve if the agent wants to retrieve if the agent wants to retrieve only facts from verified clinical

only facts from verified clinical only facts from verified clinical sources, it's very simple to filter for sources, it's very simple to filter for sources, it's very simple to filter for the appropriate tag as we walk the the appropriate tag as we walk the the appropriate tag as we walk the graph. graph. graph. So one tagging action at ingestion So one tagging action at ingestion So one tagging action at ingestion supports evaluating the veracity of a supports evaluating the veracity of a supports evaluating the veracity of a fact. But what if the fact is three parents or But what if the fact is three parents or more? Here we have a verified flag as more? Here we have a verified flag as more? Here we have a verified flag as our metadata and in this case one parent our metadata and in this case one parent our metadata and in this case one parent is not verified. is not verified. is not verified. So is the fact verified So is the fact verified So is the fact verified for the allergy flag which could be a for the allergy flag which could be a for the allergy flag which could be a life and death situation. the agent life and death situation. the agent life and death situation. the agent missing it, missing that particular flag missing it, missing that particular flag missing it, missing that particular flag could be a deadly mistake. So not could be a deadly mistake. So not could be a deadly mistake. So not retrieving the fact and any source of retrieving the fact and any source of retrieving the fact and any source of the three should block the three should block the three should block that prescription being issued. that prescription being issued. that prescription being issued. But for something like a consent But for something like a consent But for something like a consent on file for procedure fact, the mistake on file for procedure fact, the mistake on file for procedure fact, the mistake is operating on unverified is operating on unverified is operating on unverified consent. So the patient hasn't actually consent. So the patient hasn't actually consent. So the patient hasn't actually given their consent and every parent given their consent and every parent given their consent and every parent needs to be verified. So every single needs to be verified. So every single needs to be verified. So every single episode should have that tag.

episode should have that tag. episode should have that tag. So the facts are very similar shapes. So the facts are very similar shapes. So the facts are very similar shapes. They have three three parent episodes They have three three parent episodes They have three three parent episodes but opposite policies. And here but opposite policies. And here but opposite policies. And here graffiti or the underlying store exposes graffiti or the underlying store exposes graffiti or the underlying store exposes that choice. It exposes which of the that choice. It exposes which of the that choice. It exposes which of the episodes have the gra the particular episodes have the gra the particular episodes have the gra the particular tag, tag, tag, but your agent needs to execute or apply but your agent needs to execute or apply but your agent needs to execute or apply your business rules. So that's not your business rules. So that's not your business rules. So that's not necessarily something that we bake into necessarily something that we bake into necessarily something that we bake into the graph. [snorts] the graph. [snorts] the graph. [snorts] It's situational. It's situational. It's situational. Another Another Another situation where lineage is really situation where lineage is really situation where lineage is really important, important, important, we may have to delete source data we may have to delete source data we may have to delete source data due to retention policies or right to be due to retention policies or right to be due to retention policies or right to be forgotten requests. So privacy forgotten requests. So privacy forgotten requests. So privacy compliance compliance compliance and this is really tricky because if we and this is really tricky because if we and this is really tricky because if we have context derived from multiple have context derived from multiple have context derived from multiple sources, sources, sources, how do we do so? how do we do so? how do we do so? Mapping lineage here is really useful.

Mapping lineage here is really useful. Mapping lineage here is really useful. We know which facts are derived from the We know which facts are derived from the We know which facts are derived from the source data we intend on deleting. But source data we intend on deleting. But source data we intend on deleting. But what if only some of the source data what if only some of the source data what if only some of the source data needs to be deleted but not all? needs to be deleted but not all? needs to be deleted but not all? So in this example, we need to delete So in this example, we need to delete So in this example, we need to delete the intake chart data. So what the the intake chart data. So what the the intake chart data. So what the patient filled in, which is only one of patient filled in, which is only one of patient filled in, which is only one of three source data. In graffiti's model, three source data. In graffiti's model, three source data. In graffiti's model, the allergy fact survives the deletion the allergy fact survives the deletion the allergy fact survives the deletion and that's because there are two parents and that's because there are two parents and that's because there are two parents parent episode still supporting the parent episode still supporting the parent episode still supporting the fact. However, the contact preference fact. However, the contact preference fact. However, the contact preference fact fact fact is deleted as it was derived solely from is deleted as it was derived solely from is deleted as it was derived solely from the deleted source data. the deleted source data. the deleted source data. So, the rule is pretty simple here and So, the rule is pretty simple here and So, the rule is pretty simple here and it's easier to apply because the link it's easier to apply because the link it's easier to apply because the link exists. A fact is only deleted if no exists. A fact is only deleted if no exists. A fact is only deleted if no remaining episodes support it.

So to sum it all up, So to sum it all up, deriving context is lossy and deriving context is lossy and deriving context is lossy and generative. generative. generative. Lineage needs to be built in to the data Lineage needs to be built in to the data Lineage needs to be built in to the data structure, engineered into the data structure, engineered into the data structure, engineered into the data structure, which is a graph, not logged structure, which is a graph, not logged structure, which is a graph, not logged afterwards. afterwards. afterwards. And in graffiti we keep the sources And in graffiti we keep the sources And in graffiti we keep the sources verbatim and we link everything back verbatim and we link everything back verbatim and we link everything back everything derived from those sources everything derived from those sources everything derived from those sources back to the source. back to the source. back to the source. And provenence offers many benefits to And provenence offers many benefits to And provenence offers many benefits to users of graffiti. users of graffiti. users of graffiti. You have compliance built in which makes You have compliance built in which makes You have compliance built in which makes your chief compliance officer very your chief compliance officer very your chief compliance officer very happy. happy. happy. You can verify a fact based on its You can verify a fact based on its You can verify a fact based on its sources sources sources so you understand veracity. Should I so you understand veracity. Should I so you understand veracity. Should I trust this fact? trust this fact? trust this fact? It's easy to debug where something came It's easy to debug where something came It's easy to debug where something came from. So, why do I have this fact? How from. So, why do I have this fact? How from. So, why do I have this fact? How was it generated? And also determining was it generated? And also determining was it generated? And also determining what to delete. what to delete. what to delete. And most of what I've covered today is And most of what I've covered today is And most of what I've covered today is in the graffiti framework. So, you can in the graffiti framework. So, you can in the graffiti framework. So, you can go to uh the graffiti repo on GitHub and go to uh the graffiti repo on GitHub and go to uh the graffiti repo on GitHub and I have a little uh QR code QR code later I have a little uh QR code QR code later I have a little uh QR code QR code later that you can zap um and try it out.

that you can zap um and try it out. that you can zap um and try it out. So, by the way, as an aside, So, by the way, as an aside, So, by the way, as an aside, lineage and provenence is expensive. lineage and provenence is expensive. lineage and provenence is expensive. Graph construction is really expensive Graph construction is really expensive Graph construction is really expensive uh in the way that graffiti does it. And uh in the way that graffiti does it. And uh in the way that graffiti does it. And so we've put significant effort into so we've put significant effort into so we've put significant effort into reducing cost and latency of generating reducing cost and latency of generating reducing cost and latency of generating graph artifacts. And I'd be happy to graph artifacts. And I'd be happy to graph artifacts. And I'd be happy to speak to how we've done that uh in in speak to how we've done that uh in in speak to how we've done that uh in in the Q&A. the Q&A. the Q&A. So thanks for attending. Um if you'd So thanks for attending. Um if you'd So thanks for attending. Um if you'd like to learn a little bit more about like to learn a little bit more about like to learn a little bit more about Zep or Graffiti, you can zap the QR Zep or Graffiti, you can zap the QR Zep or Graffiti, you can zap the QR codes. Uh Zep is on the left and codes. Uh Zep is on the left and codes. Uh Zep is on the left and graffiti on the right. And I don't know graffiti on the right. And I don't know graffiti on the right. And I don't know if we're doing Q&A here or outside. if we're doing Q&A here or outside. if we're doing Q&A here or outside. Okay, happy to do Q&A. Okay, happy to do Q&A. Okay, happy to do Q&A. >> Just repeat the question. >> Just repeat the question. >> Just repeat the question. >> Yeah, >> Yeah, >> Yeah, >> we have time. >> we have time. >> we have time. >> Anybody have a question? >> Anybody have a question? >> Anybody have a question? >> Yeah, one right from the front. >> How do we mutate the graph >> How do we mutate the graph at the edge?

>> Oh, to account for weight changes in >> Oh, to account for weight changes in relevancy. relevancy. relevancy. Um Um Um that is some structure that we've that is some structure that we've that is some structure that we've actually built into Zap not into actually built into Zap not into actually built into Zap not into graffiti and graffiti and graffiti and um what we do is we do have for that a um what we do is we do have for that a um what we do is we do have for that a some tracing that we do which is kind of some tracing that we do which is kind of some tracing that we do which is kind of x of the graph. So not all of the x of the graph. So not all of the x of the graph. So not all of the um provenence is in the graph. Sorry, um provenence is in the graph. Sorry, um provenence is in the graph. Sorry, you you you >> if you have 50 edges, >> uh, sorry, is was that a question? If >> uh, sorry, is was that a question? If you have 50. Oh, there you go. you have 50. Oh, there you go. you have 50. Oh, there you go. >> I said I I use 50. >> I said I I use 50. >> I said I I use 50. >> You use 50. >> You use 50. >> You use 50. >> Yeah, >> Yeah, >> Yeah, >> 50 edges. Um, so so you know, Zep is >> 50 edges. Um, so so you know, Zep is >> 50 edges. Um, so so you know, Zep is able to uh look at provenence across able to uh look at provenence across able to uh look at provenence across those but using a separate data those but using a separate data those but using a separate data structure from the graph. structure from the graph. structure from the graph. >> Okay. Thank you. Thank you >> Okay. Thank you. Thank you >> Okay. Thank you. Thank you >> for that particular problem.

>> So uh does the agent also create the >> So uh does the agent also create the edges edge types and the entities itself edges edge types and the entities itself edges edge types and the entities itself or how does it resolve those edges or how does it resolve those edges or how does it resolve those edges types? types? types? >> So in graffiti you can search across the >> So in graffiti you can search across the >> So in graffiti you can search across the entire graph. Uh it has um vector entire graph. Uh it has um vector entire graph. Uh it has um vector similarity search against various similarity search against various similarity search against various textual artifacts textual artifacts textual artifacts um full text search as well as graph um full text search as well as graph um full text search as well as graph relational operations things like BFS relational operations things like BFS relational operations things like BFS um it depends on the underlying uh graph um it depends on the underlying uh graph um it depends on the underlying uh graph database that's used and so your agent database that's used and so your agent database that's used and so your agent can walk the graph it can search can walk the graph it can search can walk the graph it can search semantically etc. semantically etc. semantically etc. And obviously from anywhere you hit in And obviously from anywhere you hit in And obviously from anywhere you hit in the graph you're then able to understand the graph you're then able to understand the graph you're then able to understand the provenence of a particular artifact the provenence of a particular artifact the provenence of a particular artifact that you've hit. Uh I find it very uh fascinating this Uh I find it very uh fascinating this temporal support out right out of the temporal support out right out of the temporal support out right out of the gravity database. Can you uh tell just a gravity database. Can you uh tell just a gravity database. Can you uh tell just a little more little more little more um under the hood what are those really um under the hood what are those really um under the hood what are those really the the episodes are nodes in the graph the the episodes are nodes in the graph the the episodes are nodes in the graph just like the other nodes and how do you just like the other nodes and how do you just like the other nodes and how do you extract so I saw the API that let's say extract so I saw the API that let's say extract so I saw the API that let's say I give some uh ad ed episode how do we I give some uh ad ed episode how do we I give some uh ad ed episode how do we extract information from the ad episode extract information from the ad episode extract information from the ad episode call under the hood like large language call under the hood like large language call under the hood like large language models and and edges models and and edges models and and edges >> yeah yeah so uh yes episodes are an >> yeah yeah so uh yes episodes are an >> yeah yeah so uh yes episodes are an entity on the graph uh or a node on the entity on the graph uh or a node on the entity on the graph uh or a node on the graph. Um it makes sense to model them graph. Um it makes sense to model them graph. Um it makes sense to model them that way. Um in Zep and graffiti we have

that way. Um in Zep and graffiti we have that way. Um in Zep and graffiti we have various derived artifacts that are um various derived artifacts that are um various derived artifacts that are um nodes on the graph as well because they nodes on the graph as well because they nodes on the graph as well because they too need to have lineage and we need to too need to have lineage and we need to too need to have lineage and we need to understand how they were derived. Um, understand how they were derived. Um, understand how they were derived. Um, and in terms of how the ad episode and in terms of how the ad episode and in terms of how the ad episode method works, method works, method works, there's a pretty complicated there's a pretty complicated there's a pretty complicated uh, pipeline that gets run on um, uh, uh, pipeline that gets run on um, uh, uh, pipeline that gets run on um, uh, episode ingestion and I'll just give it episode ingestion and I'll just give it episode ingestion and I'll just give it the very high level uh, uh, outline for the very high level uh, uh, outline for the very high level uh, uh, outline for you. So there's a structured extraction you. So there's a structured extraction you. So there's a structured extraction extracting entities and the extracting entities and the extracting entities and the relationships between them and candidate relationships between them and candidate relationships between them and candidate facts and those are the materialized or facts and those are the materialized or facts and those are the materialized or hydrated fact triples. So two entities hydrated fact triples. So two entities hydrated fact triples. So two entities and a fact. Um and it's structured as uh and a fact. Um and it's structured as uh and a fact. Um and it's structured as uh a fact is structured as um um uh subject a fact is structured as um um uh subject a fact is structured as um um uh subject well well well subject uh verb object. Um and after subject uh verb object. Um and after subject uh verb object. Um and after that there is a uh deconliction process that there is a uh deconliction process that there is a uh deconliction process that runs dduplication and deconliction that runs dduplication and deconliction that runs dduplication and deconliction process. We dduplicate entities and we process. We dduplicate entities and we process. We dduplicate entities and we deconlict facts because there might be deconlict facts because there might be deconlict facts because there might be existing facts in the graph existing facts in the graph existing facts in the graph that are going to be mutated by a new that are going to be mutated by a new that are going to be mutated by a new learned fact. So Daniel loves Adidas learned fact. So Daniel loves Adidas learned fact. So Daniel loves Adidas shoes.

shoes. shoes. Three months later, Daniel's shoes fell Three months later, Daniel's shoes fell Three months later, Daniel's shoes fell apart. He sends it back to those the apart. He sends it back to those the apart. He sends it back to those the shoes back to the return application and shoes back to the return application and shoes back to the return application and he sends a nasty gram along with it. We he sends a nasty gram along with it. We he sends a nasty gram along with it. We now Daniel returned shoes as a fact and now Daniel returned shoes as a fact and now Daniel returned shoes as a fact and Daniel was unhappy about Adidas. We need Daniel was unhappy about Adidas. We need Daniel was unhappy about Adidas. We need to invalidate the Daniel loves Adidas to invalidate the Daniel loves Adidas to invalidate the Daniel loves Adidas shoes fact and so that is part of that shoes fact and so that is part of that shoes fact and so that is part of that pipeline as well. A lot of what we do pipeline as well. A lot of what we do pipeline as well. A lot of what we do uses LLMs but we try very hard not to uses LLMs but we try very hard not to uses LLMs but we try very hard not to use LLMs in this process as well. So use LLMs in this process as well. So use LLMs in this process as well. So where we're able to deploy more where we're able to deploy more where we're able to deploy more traditional information retrieval traditional information retrieval traditional information retrieval techniques, more traditional NLP techniques, more traditional NLP techniques, more traditional NLP techniques, uh looking at things like techniques, uh looking at things like techniques, uh looking at things like entropy and a bunch of other, you know, entropy and a bunch of other, you know, entropy and a bunch of other, you know, using um sim hash and a bunch of other using um sim hash and a bunch of other using um sim hash and a bunch of other approaches to ddupe, uh we do so uh far approaches to ddupe, uh we do so uh far approaches to ddupe, uh we do so uh far cheaper, far faster, deter more more far cheaper, far faster, deter more more far cheaper, far faster, deter more more far more deterministic. more deterministic. more deterministic. So hopefully that answer your questions. So hopefully that answer your questions. So hopefully that answer your questions. >> Yep. >> Yep. >> Yep. Uh thank you for the great talk. Um just Uh thank you for the great talk. Um just Uh thank you for the great talk. Um just wanted to ask a question. So it seems wanted to ask a question. So it seems wanted to ask a question. So it seems like a common theme these days in uh like a common theme these days in uh like a common theme these days in uh memory systems is more filebased memory memory systems is more filebased memory memory systems is more filebased memory and wikis and knowledge bases and uh I'm and wikis and knowledge bases and uh I'm and wikis and knowledge bases and uh I'm just wondering have is Zep working on just wondering have is Zep working on just wondering have is Zep working on something like that and also uh could something like that and also uh could something like that and also uh could some of the ideas here be represented in some of the ideas here be represented in some of the ideas here be represented in that paradigm?

that paradigm? that paradigm? >> Yeah. >> Yeah. >> Yeah. uh uh uh markdown suffers from provenence. markdown suffers from provenence. markdown suffers from provenence. File-based um file-based uh memory File-based um file-based uh memory File-based um file-based uh memory starts to break down with provenence. starts to break down with provenence. starts to break down with provenence. It's very difficult when you m mutate It's very difficult when you m mutate It's very difficult when you m mutate lines in a file to understand the lines in a file to understand the lines in a file to understand the lineage or the provenence of why those lineage or the provenence of why those lineage or the provenence of why those changes occurred. changes occurred. changes occurred. Um, not only that, but in multi- aent, Um, not only that, but in multi- aent, Um, not only that, but in multi- aent, multi-user, multi-user, multi-user, and multi-source scenarios, it can be and multi-source scenarios, it can be and multi-source scenarios, it can be very challenging to manage markdown very challenging to manage markdown very challenging to manage markdown files at scale. I think they work really files at scale. I think they work really files at scale. I think they work really well well well um for desktop usage. Uh, they sometimes um for desktop usage. Uh, they sometimes um for desktop usage. Uh, they sometimes work well in uh agentic use cases that work well in uh agentic use cases that work well in uh agentic use cases that are server based, not necessarily are server based, not necessarily are server based, not necessarily desktop or single user, single agent desktop or single user, single agent desktop or single user, single agent scenarios. Um scenarios. Um scenarios. Um but what we found is that it's um they but what we found is that it's um they but what we found is that it's um they just break down with the types of just break down with the types of just break down with the types of enterprise problems that we solving in enterprise problems that we solving in enterprise problems that we solving in particularly in particular provenence as particularly in particular provenence as particularly in particular provenence as an example. Does that answer your an example. Does that answer your an example. Does that answer your question?

Oh more I don't know how much time we Oh more I don't know how much time we have left but maybe one more have left but maybe one more have left but maybe one more >> uh Daniel thank you. Um question on the >> uh Daniel thank you. Um question on the >> uh Daniel thank you. Um question on the right bag. How do you do that explicitly right bag. How do you do that explicitly right bag. How do you do that explicitly or implicitly? Um, how do you create the or implicitly? Um, how do you create the or implicitly? Um, how do you create the facts? Do you ask the LLM to summarize facts? Do you ask the LLM to summarize facts? Do you ask the LLM to summarize the conversation or at every turn you do the conversation or at every turn you do the conversation or at every turn you do that? And and which component does it, that? And and which component does it, that? And and which component does it, Zep or? Zep or? Zep or? >> Uh, >> Uh, >> Uh, >> yeah. So, so we do um as part we >> yeah. So, so we do um as part we >> yeah. So, so we do um as part we actually as part of the extraction, actually as part of the extraction, actually as part of the extraction, we've managed to get um a singleshot we've managed to get um a singleshot we've managed to get um a singleshot extraction working that extracts extraction working that extracts extraction working that extracts entities and the relationships between entities and the relationships between entities and the relationships between them and facts. them and facts. them and facts. and we're able to do so really cheaply and we're able to do so really cheaply and we're able to do so really cheaply as a consequence. Um, as a consequence. Um, as a consequence. Um, and so yes, we're using an LLM for that. and so yes, we're using an LLM for that. and so yes, we're using an LLM for that. Uh, we do have a reflection step or some Uh, we do have a reflection step or some Uh, we do have a reflection step or some reflection built in to ensure that the reflection built in to ensure that the reflection built in to ensure that the things that we've retrieved um are things that we've retrieved um are things that we've retrieved um are actually accurate actually accurate actually accurate um as well as to do some other stuff um as well as to do some other stuff um as well as to do some other stuff around uh uh more more richness to around uh uh more more richness to around uh uh more more richness to lineage. So why did something change?

lineage. So why did something change? lineage. So why did something change? Not just this was related but also why Not just this was related but also why Not just this was related but also why did it change. Uh that's uh partly in graffiti partly Uh that's uh partly in graffiti partly in zap. in zap. in zap. Yeah. All right. Well, thank you Yeah. All right. Well, thank you Yeah. All right. Well, thank you everybody.