Memory Harnesses for Long-Running Research Agents — Stefania Druga, Sakana.ai

>> Hello. Welcome. Uh this is a big room, >> Hello. Welcome. Uh this is a big room, so you're if you're in the back, don't so you're if you're in the back, don't so you're if you're in the back, don't hesitate to come closer. Um My name is hesitate to come closer. Um My name is hesitate to come closer. Um My name is Stefania Druga. I'm a research scientist Stefania Druga. I'm a research scientist Stefania Druga. I'm a research scientist at Sakana AI in Tokyo. Uh I used to be at Sakana AI in Tokyo. Uh I used to be at Sakana AI in Tokyo. Uh I used to be based here and AI engineering uh is home based here and AI engineering uh is home based here and AI engineering uh is home community for me before being the community for me before being the community for me before being the hyperloop. So, it's very good to be hyperloop. So, it's very good to be hyperloop. So, it's very good to be back. back. back. And today I'm going to talk to you about And today I'm going to talk to you about And today I'm going to talk to you about memory harnesses for long-running memory harnesses for long-running memory harnesses for long-running research agents on device. research agents on device. research agents on device. So, if you work with long horizon tasks, So, if you work with long horizon tasks, So, if you work with long horizon tasks, you probably run into this issue of you probably run into this issue of you probably run into this issue of context blow. Right? Like when the model context blow. Right? Like when the model context blow. Right? Like when the model starts contradicting itself, or it has starts contradicting itself, or it has starts contradicting itself, or it has to redo the work because it forgot it to redo the work because it forgot it to redo the work because it forgot it did that task in the first place, or it did that task in the first place, or it did that task in the first place, or it starts to drift from your questions starts to drift from your questions starts to drift from your questions because it forgot them. And this this because it forgot them. And this this because it forgot them. And this this matters now more than ever because from matters now more than ever because from matters now more than ever because from this recent projections from Meter, this recent projections from Meter, this recent projections from Meter, we see that the trend is to solve longer we see that the trend is to solve longer we see that the trend is to solve longer and longer uh horizon tasks, and also and longer uh horizon tasks, and also and longer uh horizon tasks, and also that we're getting fewer and fewer model that we're getting fewer and fewer model that we're getting fewer and fewer model releases. So, at some point later this releases. So, at some point later this releases. So, at some point later this year, we're going to have this year, we're going to have this year, we're going to have this convergence, right? Where we'll get many convergence, right? Where we'll get many convergence, right? Where we'll get many more long-term horizon tasks and fewer more long-term horizon tasks and fewer more long-term horizon tasks and fewer model releases.

model releases. model releases. So, that makes this issue of dealing So, that makes this issue of dealing So, that makes this issue of dealing with context rot a priority. with context rot a priority. with context rot a priority. And why did I wanted to to tackle this And why did I wanted to to tackle this And why did I wanted to to tackle this problem on local models and with a local problem on local models and with a local problem on local models and with a local harness? harness? harness? Uh maybe some of you have seen this Uh maybe some of you have seen this Uh maybe some of you have seen this tweet. It's only 2 days old. Uh the CEO tweet. It's only 2 days old. Uh the CEO tweet. It's only 2 days old. Uh the CEO of Coinbase actually shared how their of Coinbase actually shared how their of Coinbase actually shared how their company managed to reduce their AI spent company managed to reduce their AI spent company managed to reduce their AI spent while actually increasing uh the AI while actually increasing uh the AI while actually increasing uh the AI usage. And the way they did that was by usage. And the way they did that was by usage. And the way they did that was by transitioning to use many more local transitioning to use many more local transitioning to use many more local models, but also having better models, but also having better models, but also having better practices, like using better routing, practices, like using better routing, practices, like using better routing, better caching, keeping the context better caching, keeping the context better caching, keeping the context clean, clean, clean, and then having better visibility for and then having better visibility for and then having better visibility for what people are using it for what. what people are using it for what. what people are using it for what. Uh what kind of task. Uh what kind of task. Uh what kind of task. So, we are seeing the local models like So, we are seeing the local models like So, we are seeing the local models like crossing the line, right? Like GLM is on crossing the line, right? Like GLM is on crossing the line, right? Like GLM is on everyone's minds, like especially with everyone's minds, like especially with everyone's minds, like especially with Fable going away. Uh Fable going away. Uh Fable going away. Uh Deep Seek V4 Flash can now be run on uh Deep Seek V4 Flash can now be run on uh Deep Seek V4 Flash can now be run on uh M3 Ultra.

M3 Ultra. M3 Ultra. And there's still a bottleneck for RAM. And there's still a bottleneck for RAM. And there's still a bottleneck for RAM. It's tricky. It's tricky. It's tricky. But these local models are starting to But these local models are starting to But these local models are starting to be useful for agentic tasks and for tool be useful for agentic tasks and for tool be useful for agentic tasks and for tool use. use. use. So, I wanted to show you what has been So, I wanted to show you what has been So, I wanted to show you what has been my setup for the experiments I'm going my setup for the experiments I'm going my setup for the experiments I'm going to share with you today. to share with you today. to share with you today. Uh Uh Uh This This is my Mac. It's still running This This is my Mac. It's still running This This is my Mac. It's still running evaluations right now uh back in my desk evaluations right now uh back in my desk evaluations right now uh back in my desk in Tokyo, and I'm controlling it from my in Tokyo, and I'm controlling it from my in Tokyo, and I'm controlling it from my phone. Um and after running evals phone. Um and after running evals phone. Um and after running evals non-stop for a couple of days, it non-stop for a couple of days, it non-stop for a couple of days, it started to get hot. So, I had my husband started to get hot. So, I had my husband started to get hot. So, I had my husband put fans around it. Um we're running out put fans around it. Um we're running out put fans around it. Um we're running out of fans, but the the machine is still of fans, but the the machine is still of fans, but the the machine is still running and the evals are still giving running and the evals are still giving running and the evals are still giving results. Um results. Um results. Um on this M3 Ultra with 96 GB on this M3 Ultra with 96 GB on this M3 Ultra with 96 GB and 28 core CPUs, I'm using two models. and 28 core CPUs, I'm using two models. and 28 core CPUs, I'm using two models. I'm using the Qwen 27B quantized at I'm using the Qwen 27B quantized at I'm using the Qwen 27B quantized at 4-bit 4-bit 4-bit and the Deep Seek V4 Flash. and the Deep Seek V4 Flash. and the Deep Seek V4 Flash. And And And before I show you how I build the memory before I show you how I build the memory before I show you how I build the memory harness on this machine, I wanted to harness on this machine, I wanted to harness on this machine, I wanted to tell you what this little What is this tell you what this little What is this tell you what this little What is this an example of, right? Like memory, when an example of, right? Like memory, when an example of, right? Like memory, when we design a harness for memory, this is we design a harness for memory, this is we design a harness for memory, this is the mental model I want you to have in the mental model I want you to have in the mental model I want you to have in mind. Um you can think of memory as a mind. Um you can think of memory as a mind. Um you can think of memory as a write-manage-read loop.

write-manage-read loop. write-manage-read loop. So, it's not just a database store. It's So, it's not just a database store. It's So, it's not just a database store. It's actually this control loop around the actually this control loop around the actually this control loop around the model. model. model. More concretely, how did I take that More concretely, how did I take that More concretely, how did I take that loop and customize it? So, this is my loop and customize it? So, this is my loop and customize it? So, this is my harness design. Like, I started with harness design. Like, I started with harness design. Like, I started with research agents that are the small research agents that are the small research agents that are the small agents because they have zero durable agents because they have zero durable agents because they have zero durable memory, and I wanted all the memory to memory, and I wanted all the memory to memory, and I wanted all the memory to come from the harness. come from the harness. come from the harness. And then, um in the middle, I have a And then, um in the middle, I have a And then, um in the middle, I have a core, which is always shown to to the core, which is always shown to to the core, which is always shown to to the agent um of traces. And then, I have a agent um of traces. And then, I have a agent um of traces. And then, I have a recall block where I'm testing different recall block where I'm testing different recall block where I'm testing different modes. modes. modes. And then, an archival block where I'm And then, an archival block where I'm And then, an archival block where I'm kee- keeping track of information across kee- keeping track of information across kee- keeping track of information across different um sessions. different um sessions. different um sessions. And in that recall block, I'm actually And in that recall block, I'm actually And in that recall block, I'm actually going through a ladder of modes that I'm going through a ladder of modes that I'm going through a ladder of modes that I'm testing. testing. testing. The baseline is like not to use memory The baseline is like not to use memory The baseline is like not to use memory at all, no recall at all. So, I'm I'm at all, no recall at all. So, I'm I'm at all, no recall at all. So, I'm I'm testing for that. testing for that. testing for that. Uh next is to use a rag vector, Uh next is to use a rag vector, Uh next is to use a rag vector, vector rag, um just to see whatever like vector rag, um just to see whatever like vector rag, um just to see whatever like the harness would pull in terms of the harness would pull in terms of the harness would pull in terms of similarity.

similarity. similarity. Then is to use a decisions uh ledger Then is to use a decisions uh ledger Then is to use a decisions uh ledger where I actually keep track of what where I actually keep track of what where I actually keep track of what decisions are being made for every turn, decisions are being made for every turn, decisions are being made for every turn, and then I can prioritize them. and then I can prioritize them. and then I can prioritize them. And last but not least, and this piece And last but not least, and this piece And last but not least, and this piece is very important, I have uh what I call is very important, I have uh what I call is very important, I have uh what I call an oracle, but basically this is the an oracle, but basically this is the an oracle, but basically this is the ground truth. So, this is like telling ground truth. So, this is like telling ground truth. So, this is like telling the harness for every loop what the the harness for every loop what the the harness for every loop what the correct memory that needs to be correct memory that needs to be correct memory that needs to be retrieved is. retrieved is. retrieved is. And the model is fixed across all the And the model is fixed across all the And the model is fixed across all the different tasks. So, the only things different tasks. So, the only things different tasks. So, the only things that I'm changing is like these that I'm changing is like these that I'm changing is like these different variables in the recall block. different variables in the recall block. different variables in the recall block. And I wanted to to give you an example And I wanted to to give you an example And I wanted to to give you an example of a first task that I tested. So, of a first task that I tested. So, of a first task that I tested. So, I wanted to see if I give the agent a I wanted to see if I give the agent a I wanted to see if I give the agent a task of doing literature review, task of doing literature review, task of doing literature review, and I'm including a lot of papers in the and I'm including a lot of papers in the and I'm including a lot of papers in the corpus where there was a big scientific corpus where there was a big scientific corpus where there was a big scientific claim. Like, this is actually a nature claim. Like, this is actually a nature claim. Like, this is actually a nature paper where they said they discovered paper where they said they discovered paper where they said they discovered 742,000 742,000 742,000 promising materials. Like, it was a very promising materials. Like, it was a very promising materials. Like, it was a very big claim, which got retracted later.

big claim, which got retracted later. big claim, which got retracted later. But, the retraction to it's a much But, the retraction to it's a much But, the retraction to it's a much smaller like haystack needle in that smaller like haystack needle in that smaller like haystack needle in that corpus than the headlines and the corpus than the headlines and the corpus than the headlines and the citations. citations. citations. So, I wanted to see if if the system can So, I wanted to see if if the system can So, I wanted to see if if the system can retrieve the right answer retrieve the right answer retrieve the right answer for these type of questions. for these type of questions. for these type of questions. And what I found was because like for And what I found was because like for And what I found was because like for these tasks, these tasks, these tasks, all the papers and all the information all the papers and all the information all the papers and all the information fit into the context, the memory fit into the context, the memory fit into the context, the memory actually didn't add actually didn't add actually didn't add more capability. It was the same more capability. It was the same more capability. It was the same performance with memory and without performance with memory and without performance with memory and without memory, and it only added more cost. So, memory, and it only added more cost. So, memory, and it only added more cost. So, when your task fits in context, the when your task fits in context, the when your task fits in context, the harness doesn't add much. harness doesn't add much. harness doesn't add much. However, However, However, if I start to run tasks that are longer if I start to run tasks that are longer if I start to run tasks that are longer term horizon, and the entire task and term horizon, and the entire task and term horizon, and the entire task and the relevant context doesn't the relevant context doesn't the relevant context doesn't uh fit, then having a good memory uh fit, then having a good memory uh fit, then having a good memory harness really starts to pay off. harness really starts to pay off. harness really starts to pay off. So, this is another example of a task So, this is another example of a task So, this is another example of a task that I ran. This is actually from an that I ran. This is actually from an that I ran. This is actually from an established benchmark for long horizon established benchmark for long horizon established benchmark for long horizon uh tasks memory. It's called X-Bench.

uh tasks memory. It's called X-Bench. uh tasks memory. It's called X-Bench. And this is an example of a question, And this is an example of a question, And this is an example of a question, right? So, I'm asking a question, and right? So, I'm asking a question, and right? So, I'm asking a question, and then like then like then like the right answer is in a the right answer is in a the right answer is in a like step 124, like step 124, like step 124, but the moment when I ask the question, but the moment when I ask the question, but the moment when I ask the question, I'm asking it like at step 500. So, I'm asking it like at step 500. So, I'm asking it like at step 500. So, it's completely outside of the context it's completely outside of the context it's completely outside of the context window, and the model needs to use the window, and the model needs to use the window, and the model needs to use the memory harness to retrieve the specific memory harness to retrieve the specific memory harness to retrieve the specific answer from the right step. answer from the right step. answer from the right step. So, I'm testing this by uh changing the So, I'm testing this by uh changing the So, I'm testing this by uh changing the different policy ladder that I explained different policy ladder that I explained different policy ladder that I explained before with memory off uh by deploying before with memory off uh by deploying before with memory off uh by deploying recall different types of recall and by recall different types of recall and by recall different types of recall and by using the Oracle as a reference. using the Oracle as a reference. using the Oracle as a reference. And what I found was that And what I found was that And what I found was that with the ranked recall the model gets with the ranked recall the model gets with the ranked recall the model gets the right answer the right answer the right answer more frequently than without. more frequently than without. more frequently than without. And here is a breakdown of the And here is a breakdown of the And here is a breakdown of the decomposition of performance on this X decomposition of performance on this X decomposition of performance on this X bench tasks.

bench tasks. bench tasks. So I ran over 68 questions and for each So I ran over 68 questions and for each So I ran over 68 questions and for each of these questions there were like of these questions there were like of these questions there were like multiple multiple multiple cells and lots of different seeds. cells and lots of different seeds. cells and lots of different seeds. And what I found was that the rank only And what I found was that the rank only And what I found was that the rank only ledger performed the best. ledger performed the best. ledger performed the best. And it performed better than like just And it performed better than like just And it performed better than like just gating gating gating the harness by saying do you need to use the harness by saying do you need to use the harness by saying do you need to use memory or do you not need to use memory. memory or do you not need to use memory. memory or do you not need to use memory. And you're probably going to ask like And you're probably going to ask like And you're probably going to ask like why is the Oracle not hitting like the why is the Oracle not hitting like the why is the Oracle not hitting like the max? And I'm going to explain that too. max? And I'm going to explain that too. max? And I'm going to explain that too. So the Oracle what it does, it provides So the Oracle what it does, it provides So the Oracle what it does, it provides the right information, the right memory the right information, the right memory the right information, the right memory to the model but it doesn't force it to to the model but it doesn't force it to to the model but it doesn't force it to use it. So the model can use it. So the model can use it. So the model can get the right memory but still retrieve get the right memory but still retrieve get the right memory but still retrieve the wrong information or choose to the wrong information or choose to the wrong information or choose to ignore it or be confused. So that's why ignore it or be confused. So that's why ignore it or be confused. So that's why the Oracle in this case doesn't hit the the Oracle in this case doesn't hit the the Oracle in this case doesn't hit the max performance. max performance. max performance. And I've done lots of ablations on these And I've done lots of ablations on these And I've done lots of ablations on these tasks to see like what happens if I give tasks to see like what happens if I give tasks to see like what happens if I give arbitrary arbitrary arbitrary um um um examples. What happens if I give it the examples. What happens if I give it the examples. What happens if I give it the wrong step? What happens if I give it wrong step? What happens if I give it wrong step? What happens if I give it the most recent step? And I still found the most recent step? And I still found the most recent step? And I still found that the best performing that the best performing that the best performing condition was the one with the ranked condition was the one with the ranked condition was the one with the ranked policy for recall.

policy for recall. policy for recall. And And And this actually works on several models, this actually works on several models, this actually works on several models, not only on the Qwen 27B but also on the not only on the Qwen 27B but also on the not only on the Qwen 27B but also on the DS4 flash and it also works across DS4 flash and it also works across DS4 flash and it also works across different benchmarks. I also tried it on different benchmarks. I also tried it on different benchmarks. I also tried it on the Spider V2 benchmark. the Spider V2 benchmark. the Spider V2 benchmark. And it's not just that it gives you And it's not just that it gives you And it's not just that it gives you better recall, it actually costs less. better recall, it actually costs less. better recall, it actually costs less. So, maybe a good heuristic to have here So, maybe a good heuristic to have here So, maybe a good heuristic to have here is that bad memory is expensive because is that bad memory is expensive because is that bad memory is expensive because it spends more token and it can send the it spends more token and it can send the it spends more token and it can send the agent the wrong way. But having like a agent the wrong way. But having like a agent the wrong way. But having like a good structural policy for recall can good structural policy for recall can good structural policy for recall can save you a lot of tokens and uh save you a lot of tokens and uh save you a lot of tokens and uh budget. budget. budget. So, one thing that I want to encourage So, one thing that I want to encourage So, one thing that I want to encourage you from this experiment is to consider you from this experiment is to consider you from this experiment is to consider the recall policy as a first-class the recall policy as a first-class the recall policy as a first-class metric metric metric and to start to think about how you and to start to think about how you and to start to think about how you might use it in your systems. Like, what might use it in your systems. Like, what might use it in your systems. Like, what are the type of memories that you want are the type of memories that you want are the type of memories that you want to store? to store? to store? What How do you rank them? Like, how do What How do you rank them? Like, how do What How do you rank them? Like, how do you design your recall function?

you design your recall function? you design your recall function? And then um what are the type What And then um what are the type What And then um what are the type What survives when you run this over and over survives when you run this over and over survives when you run this over and over and over and um multiple sessions, and over and um multiple sessions, and over and um multiple sessions, multiple runs? multiple runs? multiple runs? And this is just a simple first kind of And this is just a simple first kind of And this is just a simple first kind of experiment experiment experiment but the memory technique landscape is but the memory technique landscape is but the memory technique landscape is very rich. Um so, there's over 30 very rich. Um so, there's over 30 very rich. Um so, there's over 30 runnable cookbooks that are shared in runnable cookbooks that are shared in runnable cookbooks that are shared in this open-source repository from um this open-source repository from um this open-source repository from um Diamond Diamond Diamond and and and memory is complex. We have short-term, memory is complex. We have short-term, memory is complex. We have short-term, long-term, different cognitive tech- long-term, different cognitive tech- long-term, different cognitive tech- techniques. Uh we can use start to use techniques. Uh we can use start to use techniques. Uh we can use start to use evaluation results as well. evaluation results as well. evaluation results as well. Um and right now, there's actually a a Um and right now, there's actually a a Um and right now, there's actually a a pretty broad landscape of solutions, pretty broad landscape of solutions, pretty broad landscape of solutions, right? So, going from simple file system right? So, going from simple file system right? So, going from simple file system retrieval to training memory models retrieval to training memory models retrieval to training memory models um um um there's there's a wide spectrum of there's there's a wide spectrum of there's there's a wide spectrum of solutions from less structural to solutions from less structural to solutions from less structural to completely structured. Um so, I think completely structured. Um so, I think completely structured. Um so, I think there's a lot of research we're going to there's a lot of research we're going to there's a lot of research we're going to see in this space. Uh it's important um see in this space. Uh it's important um see in this space. Uh it's important um it becomes more and more relevant. And it becomes more and more relevant. And it becomes more and more relevant. And for me, it's been super fun to to test for me, it's been super fun to to test for me, it's been super fun to to test this on local models this on local models this on local models um because I got to control everything.

um because I got to control everything. um because I got to control everything. I got to control the data I was using, I got to control the data I was using, I got to control the data I was using, the entire traces of compute and the entire traces of compute and the entire traces of compute and evaluations, evaluations, evaluations, and um and um and um yeah, I I see that as an example of yeah, I I see that as an example of yeah, I I see that as an example of sovereignty. sovereignty. sovereignty. And it comes at a cost. Uh I didn't tell And it comes at a cost. Uh I didn't tell And it comes at a cost. Uh I didn't tell you that these local models I can only you that these local models I can only you that these local models I can only what run them in serial, like they don't what run them in serial, like they don't what run them in serial, like they don't support batch querying for the deep seed support batch querying for the deep seed support batch querying for the deep seed 4 flash. So, that's why I am still 4 flash. So, that's why I am still 4 flash. So, that's why I am still running evaluations back on my computer running evaluations back on my computer running evaluations back on my computer in Tokyo or I I was doing it on the in Tokyo or I I was doing it on the in Tokyo or I I was doing it on the flight on my way here because it takes a flight on my way here because it takes a flight on my way here because it takes a long time. Um but I still think it's long time. Um but I still think it's long time. Um but I still think it's very powerful, and it's a very good test very powerful, and it's a very good test very powerful, and it's a very good test for what memory can do when you can for what memory can do when you can for what memory can do when you can control every single step of the control every single step of the control every single step of the pipeline. pipeline. pipeline. And this sovereign capability is part of And this sovereign capability is part of And this sovereign capability is part of a bigger ecosystem that is very a bigger ecosystem that is very a bigger ecosystem that is very important for us at Sakana AI in Japan. important for us at Sakana AI in Japan. important for us at Sakana AI in Japan. Um we believe in the importance of Um we believe in the importance of Um we believe in the importance of sovereign AI today more than ever. And sovereign AI today more than ever. And sovereign AI today more than ever. And we are also hiring. So, if you're we are also hiring. So, if you're we are also hiring. So, if you're interested and want to hear more about interested and want to hear more about interested and want to hear more about this, and if you want to come join us in this, and if you want to come join us in this, and if you want to come join us in Japan, come talk to me.

Japan, come talk to me. Japan, come talk to me. Uh thank you very much. Uh thank you very much. Uh thank you very much. >> [applause]