Beyond Static Intelligence: Evaluating Continual Learning — Parth Asawa, UC Berkeley

Hi everyone. Uh my name is Partasawa. Hi everyone. Uh my name is Partasawa. I'm a PhD student at UC Berkeley. We're I'm a PhD student at UC Berkeley. We're I'm a PhD student at UC Berkeley. We're all at these AI and machine learning all at these AI and machine learning all at these AI and machine learning conferences, but everyone is just conferences, but everyone is just conferences, but everyone is just talking about how smart these agents talking about how smart these agents talking about how smart these agents are. We're not actually talking about are. We're not actually talking about are. We're not actually talking about learning ability or how much these learning ability or how much these learning ability or how much these agents learn. And so today I want to agents learn. And so today I want to agents learn. And so today I want to talk about that. talk about that. talk about that. You've seen the way we evaluate language You've seen the way we evaluate language You've seen the way we evaluate language models today. Every time there's a new models today. Every time there's a new models today. Every time there's a new model release, we see charts and graphs model release, we see charts and graphs model release, we see charts and graphs that look something like this, right? that look something like this, right? that look something like this, right? The way we evaluate these language The way we evaluate these language The way we evaluate these language models is we ask them to do one task and models is we ask them to do one task and models is we ask them to do one task and then completely independently we ask then completely independently we ask then completely independently we ask them to do another task and then them to do another task and then them to do another task and then another. And then we repeat this across another. And then we repeat this across another. And then we repeat this across a variety of different benchmarks, a variety of different benchmarks, a variety of different benchmarks, right? And so you've probably seen a right? And so you've probably seen a right? And so you've probably seen a variety of all of these. In the end, variety of all of these. In the end, variety of all of these. In the end, those benchmarks in aggregate when we those benchmarks in aggregate when we those benchmarks in aggregate when we take all of those independent evaluation take all of those independent evaluation take all of those independent evaluation scores give us leaderboards that look scores give us leaderboards that look scores give us leaderboards that look like this. like this. like this. Now, the challenge here is that if you Now, the challenge here is that if you Now, the challenge here is that if you look at the way we actually come to look at the way we actually come to look at the way we actually come to these single independent scores, what these single independent scores, what these single independent scores, what we've done is we've kind of told the we've done is we've kind of told the we've done is we've kind of told the models, imagine that every time you do models, imagine that every time you do models, imagine that every time you do something, you completely forget your something, you completely forget your something, you completely forget your memory. Okay? Like imagine if your life memory. Okay? Like imagine if your life memory. Okay? Like imagine if your life was like that. You're restarting from was like that. You're restarting from was like that. You're restarting from scratch every single time you do scratch every single time you do scratch every single time you do anything. That's the premise under which anything. That's the premise under which anything. That's the premise under which we're evaluating language models today.

we're evaluating language models today. we're evaluating language models today. I'm going to argue that it's not I'm going to argue that it's not I'm going to argue that it's not necessarily what it should look like. If necessarily what it should look like. If necessarily what it should look like. If you were to look at learning ability, you were to look at learning ability, you were to look at learning ability, you might see charts where across you might see charts where across you might see charts where across different instances, performance doesn't different instances, performance doesn't different instances, performance doesn't look like a scattered line, but it looks look like a scattered line, but it looks look like a scattered line, but it looks like as a function of prior experience, like as a function of prior experience, like as a function of prior experience, models are actually able to improve models are actually able to improve models are actually able to improve their performance in contrast to what their performance in contrast to what their performance in contrast to what they've done in the past. they've done in the past. they've done in the past. Let's take a step back. What is Let's take a step back. What is Let's take a step back. What is continual learning? continual learning? continual learning? Continual learning to me is sample Continual learning to me is sample Continual learning to me is sample efficient online learning that is stable efficient online learning that is stable efficient online learning that is stable over long horizons. It's a challenging over long horizons. It's a challenging over long horizons. It's a challenging problem because you have to deal with problem because you have to deal with problem because you have to deal with both your ability to retain prior both your ability to retain prior both your ability to retain prior information without forgetting while information without forgetting while information without forgetting while simultaneously being able to update from simultaneously being able to update from simultaneously being able to update from new information and adjust your priors new information and adjust your priors new information and adjust your priors to new new data you're receiving over to new new data you're receiving over to new new data you're receiving over time. In language models specifically, time. In language models specifically, time. In language models specifically, you know, the way we train language you know, the way we train language you know, the way we train language models is we take a bunch of data we models is we take a bunch of data we models is we take a bunch of data we want to train them on and we kind of put want to train them on and we kind of put want to train them on and we kind of put them in a box in some offline training them in a box in some offline training them in a box in some offline training process and then maybe for a few weeks process and then maybe for a few weeks process and then maybe for a few weeks or something we train the model and then or something we train the model and then or something we train the model and then we extract a frozen checkpoint, right? we extract a frozen checkpoint, right? we extract a frozen checkpoint, right? And this frozen checkpoint is what's And this frozen checkpoint is what's And this frozen checkpoint is what's deployed to the world. The weights don't deployed to the world. The weights don't deployed to the world. The weights don't really change after that in today's really change after that in today's really change after that in today's paradigm. And continual learning is all paradigm. And continual learning is all paradigm. And continual learning is all about trying to change that, trying to about trying to change that, trying to about trying to change that, trying to enable mechanisms for models to learn enable mechanisms for models to learn enable mechanisms for models to learn over time. So in language models, people over time. So in language models, people over time. So in language models, people have looked to approaches like in have looked to approaches like in have looked to approaches like in context learning. Put more information context learning. Put more information context learning. Put more information in the model's context and the model can in the model's context and the model can in the model's context and the model can simulate some amount of learning in simulate some amount of learning in simulate some amount of learning in context. People have looked towards context. People have looked towards context. People have looked towards external memory stores, right? You could external memory stores, right? You could external memory stores, right? You could externalize some of your memory to a externalize some of your memory to a externalize some of your memory to a notepad that you then get to refer to notepad that you then get to refer to notepad that you then get to refer to every single time or a key value store every single time or a key value store every single time or a key value store or your choice of external memory. Or or your choice of external memory. Or or your choice of external memory. Or people look towards the parametric

people look towards the parametric people look towards the parametric approaches. What would it actually look approaches. What would it actually look approaches. What would it actually look like if we had ways to update the like if we had ways to update the like if we had ways to update the model's weights online such that it model's weights online such that it model's weights online such that it could improve and learn from the new could improve and learn from the new could improve and learn from the new information? information? information? Now, today I'm not going to make a case Now, today I'm not going to make a case Now, today I'm not going to make a case for which one of these can solve for which one of these can solve for which one of these can solve continual learning, though I do have an continual learning, though I do have an continual learning, though I do have an opinion and I'll share it at the end. opinion and I'll share it at the end. opinion and I'll share it at the end. But what I am going to make a case for But what I am going to make a case for But what I am going to make a case for is that we're actually not evaluating it is that we're actually not evaluating it is that we're actually not evaluating it at all today. And the question that we at all today. And the question that we at all today. And the question that we need to ask ourselves is that if need to ask ourselves is that if need to ask ourselves is that if continual learning doesn't look like continual learning doesn't look like continual learning doesn't look like point capabilities, we need to be point capabilities, we need to be point capabilities, we need to be measuring it the right way to optimize measuring it the right way to optimize measuring it the right way to optimize for the right objective as a field. for the right objective as a field. for the right objective as a field. So let's get into it, right? What does So let's get into it, right? What does So let's get into it, right? What does continual learning kind of look like continual learning kind of look like continual learning kind of look like today when you read the literature? To today when you read the literature? To today when you read the literature? To be clear, continual learning by itself be clear, continual learning by itself be clear, continual learning by itself is not a new idea. Continual learning is not a new idea. Continual learning is not a new idea. Continual learning for neural networks has been around for for neural networks has been around for for neural networks has been around for decades, right? There have been famous decades, right? There have been famous decades, right? There have been famous problems in catastrophic forgetting that problems in catastrophic forgetting that problems in catastrophic forgetting that people have spent years trying to solve. people have spent years trying to solve. people have spent years trying to solve. But with language and you know an But with language and you know an But with language and you know an example of that is some of these sorts example of that is some of these sorts example of that is some of these sorts of papers where you'll see graphs like of papers where you'll see graphs like of papers where you'll see graphs like this where you train the models on a this where you train the models on a this where you train the models on a particular task distribution task A then particular task distribution task A then particular task distribution task A then you train it on the task B distribution you train it on the task B distribution you train it on the task B distribution and then you train it on task C. And the and then you train it on task C. And the and then you train it on task C. And the objective you're looking for is the does objective you're looking for is the does objective you're looking for is the does the model's performance not degrade on the model's performance not degrade on the model's performance not degrade on prior tasks while it's still able to prior tasks while it's still able to prior tasks while it's still able to learn the new tasks. And people have learn the new tasks. And people have learn the new tasks. And people have taken this similar methodology and taken this similar methodology and taken this similar methodology and applied it to language models too.

applied it to language models too. applied it to language models too. Right? In some of the more recent papers Right? In some of the more recent papers Right? In some of the more recent papers we've seen, you're you're starting to we've seen, you're you're starting to we've seen, you're you're starting to see similar graphs of you train on one see similar graphs of you train on one see similar graphs of you train on one task and you train on the next. There's task and you train on the next. There's task and you train on the next. There's also a different class of evaluations also a different class of evaluations also a different class of evaluations for language models that look more like for language models that look more like for language models that look more like a factual recall sort of test where over a factual recall sort of test where over a factual recall sort of test where over these extremely long horizons, can these extremely long horizons, can these extremely long horizons, can models remember particular parts of the models remember particular parts of the models remember particular parts of the conversation and improve from that. The conversation and improve from that. The conversation and improve from that. The case I'm going to make though is that case I'm going to make though is that case I'm going to make though is that these sorts of evaluations aren't these sorts of evaluations aren't these sorts of evaluations aren't actually sufficient for continual actually sufficient for continual actually sufficient for continual learning. Okay, they don't consider learning. Okay, they don't consider learning. Okay, they don't consider sample efficiency a first order sample efficiency a first order sample efficiency a first order requirement. They don't necessarily requirement. They don't necessarily requirement. They don't necessarily always measure learning. And perhaps always measure learning. And perhaps always measure learning. And perhaps most importantly, they won't work for most importantly, they won't work for most importantly, they won't work for frontier language models. And the reason frontier language models. And the reason frontier language models. And the reason being that frontier language models are being that frontier language models are being that frontier language models are pre-trained on vast distributions of the pre-trained on vast distributions of the pre-trained on vast distributions of the entire internet or economically valuable entire internet or economically valuable entire internet or economically valuable tasks we care about. And so if you're tasks we care about. And so if you're tasks we care about. And so if you're expecting to see improvement on new expecting to see improvement on new expecting to see improvement on new tasks, it's it's actually really hard to tasks, it's it's actually really hard to tasks, it's it's actually really hard to come up with what those sorts of tasks come up with what those sorts of tasks come up with what those sorts of tasks should look like. should look like. should look like. So given all of that, what should a So given all of that, what should a So given all of that, what should a continual learning benchmark look like? continual learning benchmark look like? continual learning benchmark look like? The biggest question that I get when I The biggest question that I get when I The biggest question that I get when I when I talk about this with people is when I talk about this with people is when I talk about this with people is why can't I just chain existing why can't I just chain existing why can't I just chain existing benchmarks together? Like we've we've benchmarks together? Like we've we've benchmarks together? Like we've we've seen all of these Amy data sets. Why seen all of these Amy data sets. Why seen all of these Amy data sets. Why can't I just take my AM problems and can't I just take my AM problems and can't I just take my AM problems and solve them in a sequence? And the solve them in a sequence? And the solve them in a sequence? And the fundamental problem there is that fundamental problem there is that fundamental problem there is that benchmark instances are in traditional benchmark instances are in traditional benchmark instances are in traditional language model evaluation are designed language model evaluation are designed language model evaluation are designed to be independent. That means they don't to be independent. That means they don't to be independent. That means they don't have shared structure across tasks and have shared structure across tasks and have shared structure across tasks and as a result you can't meaningfully as a result you can't meaningfully as a result you can't meaningfully expect them to improve from earlier expect them to improve from earlier expect them to improve from earlier experience in future instances. And this experience in future instances. And this experience in future instances. And this is the fundamental problem that exists is the fundamental problem that exists is the fundamental problem that exists with any approach that tries to chain

with any approach that tries to chain with any approach that tries to chain prior benchmarks together. prior benchmarks together. prior benchmarks together. In contrast, when we're designing In contrast, when we're designing In contrast, when we're designing continual learning benchmarks, we looked continual learning benchmarks, we looked continual learning benchmarks, we looked at three main design criteria as things at three main design criteria as things at three main design criteria as things that are kind of a fundamental that are kind of a fundamental that are kind of a fundamental requirement for evaluating continual requirement for evaluating continual requirement for evaluating continual learning. The first one here is learning. The first one here is learning. The first one here is headroom. And this kind of goes to my headroom. And this kind of goes to my headroom. And this kind of goes to my last point. Frontier language models are last point. Frontier language models are last point. Frontier language models are trained on almost everything in the trained on almost everything in the trained on almost everything in the internet. If you want to evaluate internet. If you want to evaluate internet. If you want to evaluate continual learning, you need some sort continual learning, you need some sort continual learning, you need some sort of task that actually requires online of task that actually requires online of task that actually requires online adaptation or learning from the language adaptation or learning from the language adaptation or learning from the language models. And that doesn't exist for a models. And that doesn't exist for a models. And that doesn't exist for a wide variety of the benchmarks where the wide variety of the benchmarks where the wide variety of the benchmarks where the models are just trained offline on that models are just trained offline on that models are just trained offline on that data. If the model can improve on your data. If the model can improve on your data. If the model can improve on your tasks by just training offline and not tasks by just training offline and not tasks by just training offline and not actually require any online learning, actually require any online learning, actually require any online learning, then it's not a good task for measuring then it's not a good task for measuring then it's not a good task for measuring continual learning. continual learning. continual learning. The second point that I'm going to talk The second point that I'm going to talk The second point that I'm going to talk about is shared structure. What this about is shared structure. What this about is shared structure. What this means is when going back to my point of means is when going back to my point of means is when going back to my point of pre prior language modeling benchmarks pre prior language modeling benchmarks pre prior language modeling benchmarks all being independent instances. If you all being independent instances. If you all being independent instances. If you want there to be an expectation of want there to be an expectation of want there to be an expectation of improvement in how you're evaluating improvement in how you're evaluating improvement in how you're evaluating continual learning, then there needs to continual learning, then there needs to continual learning, then there needs to exist some shared latent structure exist some shared latent structure exist some shared latent structure between the tasks. You can kind of think between the tasks. You can kind of think between the tasks. You can kind of think of this as some sort of shared latent in of this as some sort of shared latent in of this as some sort of shared latent in the environment that your models or the environment that your models or the environment that your models or agents are learning over time and agents are learning over time and agents are learning over time and they're seeking to exploit these these they're seeking to exploit these these they're seeking to exploit these these latent structures that exist in the latent structures that exist in the latent structures that exist in the environments to improve their environments to improve their environments to improve their performance on future tasks as a result performance on future tasks as a result performance on future tasks as a result of their prior information. And the of their prior information. And the of their prior information. And the third criteria that I'll talk about is a third criteria that I'll talk about is a third criteria that I'll talk about is a learning mechanism. And what this means learning mechanism. And what this means learning mechanism. And what this means is just there has to be some realistic is just there has to be some realistic is just there has to be some realistic expectation that the models are able to expectation that the models are able to expectation that the models are able to learn as a result of their prior

learn as a result of their prior learn as a result of their prior experience. This could look like scalar experience. This could look like scalar experience. This could look like scalar reward. It could look like error reward. It could look like error reward. It could look like error messages. It could look like textual messages. It could look like textual messages. It could look like textual feedback. The point being, there needs feedback. The point being, there needs feedback. The point being, there needs to exist something in the environment to exist something in the environment to exist something in the environment that's giving agents signal to learn and that's giving agents signal to learn and that's giving agents signal to learn and improve on future tasks. Otherwise, it's improve on future tasks. Otherwise, it's improve on future tasks. Otherwise, it's not really a fair measure of continual not really a fair measure of continual not really a fair measure of continual learning. learning. learning. Let's talk a little bit more about the Let's talk a little bit more about the Let's talk a little bit more about the metrics before we get into examples. metrics before we get into examples. metrics before we get into examples. Obviously, there's a simple metric for a Obviously, there's a simple metric for a Obviously, there's a simple metric for a lot of tasks that we've seen recently, lot of tasks that we've seen recently, lot of tasks that we've seen recently, which is reward. And in our benchmark, which is reward. And in our benchmark, which is reward. And in our benchmark, it's no different. Every task defines a it's no different. Every task defines a it's no different. Every task defines a per instance reward metric where reward per instance reward metric where reward per instance reward metric where reward serves as a proxy for learning. Higher serves as a proxy for learning. Higher serves as a proxy for learning. Higher reward is better. You could think of reward is better. You could think of reward is better. You could think of this as efficiency. You could think of this as efficiency. You could think of this as efficiency. You could think of this as profit. You could think of this this as profit. You could think of this this as profit. You could think of this as how close you were to the actual as how close you were to the actual as how close you were to the actual predictions. There's some single unit of predictions. There's some single unit of predictions. There's some single unit of reward that operates on a per instance reward that operates on a per instance reward that operates on a per instance basis. And you might ask like why can't basis. And you might ask like why can't basis. And you might ask like why can't I just evaluate continual learning by I just evaluate continual learning by I just evaluate continual learning by looking at cumulative reward across looking at cumulative reward across looking at cumulative reward across these sequences of tasks we're these sequences of tasks we're these sequences of tasks we're constructing? And the answer is that constructing? And the answer is that constructing? And the answer is that total reward alone might confound total reward alone might confound total reward alone might confound [clears throat] continual learning [clears throat] continual learning [clears throat] continual learning ability with base model strength.

ability with base model strength. ability with base model strength. Let's look at a diagram to kind of Let's look at a diagram to kind of Let's look at a diagram to kind of explain why this might work. Here we explain why this might work. Here we explain why this might work. Here we have two different continual learning have two different continual learning have two different continual learning systems. Okay, the the red one and the systems. Okay, the the red one and the systems. Okay, the the red one and the black one, the ones connected by the black one, the ones connected by the black one, the ones connected by the dotted lines. The gray system at the dotted lines. The gray system at the dotted lines. The gray system at the bottom is the black system except it bottom is the black system except it bottom is the black system except it wasn't allowed to maintain state across wasn't allowed to maintain state across wasn't allowed to maintain state across time. Okay, so that's what the black time. Okay, so that's what the black time. Okay, so that's what the black system looks like if its memory was system looks like if its memory was system looks like if its memory was wiped every single time. But if its wiped every single time. But if its wiped every single time. But if its memory wasn't wiped, if it had some memory wasn't wiped, if it had some memory wasn't wiped, if it had some mechanism of learning, then you can kind mechanism of learning, then you can kind mechanism of learning, then you can kind of see that it tends to improve over of see that it tends to improve over of see that it tends to improve over time. The challenge here is that if we time. The challenge here is that if we time. The challenge here is that if we looked at the cumulative reward metric looked at the cumulative reward metric looked at the cumulative reward metric alone, the red system would look like it alone, the red system would look like it alone, the red system would look like it was the best the best performing system was the best the best performing system was the best the best performing system on this task, right? It let's assume for on this task, right? It let's assume for on this task, right? It let's assume for a second that there is per room to do a second that there is per room to do a second that there is per room to do better and it hasn't saturated it. It better and it hasn't saturated it. It better and it hasn't saturated it. It has a higher cumulative reward, but you has a higher cumulative reward, but you has a higher cumulative reward, but you can kind of tell that in comparison to can kind of tell that in comparison to can kind of tell that in comparison to uh to the black system, it doesn't uh to the black system, it doesn't uh to the black system, it doesn't necessarily improve over the stateless necessarily improve over the stateless necessarily improve over the stateless baseline. It's just a better system to baseline. It's just a better system to baseline. It's just a better system to begin with. And this is one of the begin with. And this is one of the begin with. And this is one of the confounds we run into when we're trying confounds we run into when we're trying confounds we run into when we're trying to measure cumulative reward. To deal to measure cumulative reward. To deal to measure cumulative reward. To deal with this, we look at the gain metric in with this, we look at the gain metric in with this, we look at the gain metric in addition to reward.

addition to reward. addition to reward. Gain refers to the difference between Gain refers to the difference between Gain refers to the difference between stateful reward and stateless reward. stateful reward and stateless reward. stateful reward and stateless reward. What that means practically in our What that means practically in our What that means practically in our benchmark is that for any system, we run benchmark is that for any system, we run benchmark is that for any system, we run it through the benchmark twice. Once in it through the benchmark twice. Once in it through the benchmark twice. Once in the normal way where it's allowed to the normal way where it's allowed to the normal way where it's allowed to maintain state across all of the maintain state across all of the maintain state across all of the instances in the benchmark. That might instances in the benchmark. That might instances in the benchmark. That might mean it's learning and updating its mean it's learning and updating its mean it's learning and updating its policy. It might mean that it's updating policy. It might mean that it's updating policy. It might mean that it's updating its notes. It might mean that it's just its notes. It might mean that it's just its notes. It might mean that it's just growing its context length, but it's growing its context length, but it's growing its context length, but it's allowed to maintain state. The second allowed to maintain state. The second allowed to maintain state. The second thing we run is a stateless baseline. thing we run is a stateless baseline. thing we run is a stateless baseline. What that means is we reset the system What that means is we reset the system What that means is we reset the system between every instance of a task so that between every instance of a task so that between every instance of a task so that the model isn't actually allowed to the model isn't actually allowed to the model isn't actually allowed to continually learn in any meaningful way. continually learn in any meaningful way. continually learn in any meaningful way. The difference between these two numbers The difference between these two numbers The difference between these two numbers then for any instance intuitively is then for any instance intuitively is then for any instance intuitively is kind of a question. It answers the kind of a question. It answers the kind of a question. It answers the question of how much did if we're question of how much did if we're question of how much did if we're looking at gain on task five, how much looking at gain on task five, how much looking at gain on task five, how much did my prior experience on the first did my prior experience on the first did my prior experience on the first four tasks actually lead to an four tasks actually lead to an four tasks actually lead to an improvement in my performance on the improvement in my performance on the improvement in my performance on the fifth task. It isolates out what your fifth task. It isolates out what your fifth task. It isolates out what your benefit of from actually learning was benefit of from actually learning was benefit of from actually learning was versus your base model's initial versus your base model's initial versus your base model's initial capability. To to be clear, I think capability. To to be clear, I think capability. To to be clear, I think reward, gain, and cost are all measured reward, gain, and cost are all measured reward, gain, and cost are all measured on Parto Frontiers. There isn't one on Parto Frontiers. There isn't one on Parto Frontiers. There isn't one single metric that I think defines single metric that I think defines single metric that I think defines continual learning because we still care continual learning because we still care continual learning because we still care about the base model strength. We still about the base model strength. We still about the base model strength. We still care about the ability to learn and we care about the ability to learn and we care about the ability to learn and we still care about the cost we're still care about the cost we're still care about the cost we're expending for these systems and so we expending for these systems and so we expending for these systems and so we try to measure everything on PTO try to measure everything on PTO try to measure everything on PTO Frontiers in this benchmark.

Frontiers in this benchmark. Frontiers in this benchmark. Let's jump into an example of a concrete Let's jump into an example of a concrete Let's jump into an example of a concrete task to ground some of what we've been task to ground some of what we've been task to ground some of what we've been talking about here. I'm going to walk talking about here. I'm going to walk talking about here. I'm going to walk through the database exploration task through the database exploration task through the database exploration task which is hopefully a little approachable which is hopefully a little approachable which is hopefully a little approachable for uh which is hopefully approachable for uh which is hopefully approachable for uh which is hopefully approachable to understand. If you guys have ever to understand. If you guys have ever to understand. If you guys have ever worked in data engineering or know data worked in data engineering or know data worked in data engineering or know data engineers, you know that one of the engineers, you know that one of the engineers, you know that one of the things they do is operate over these things they do is operate over these things they do is operate over these vast databases with large set many vast databases with large set many vast databases with large set many different sets of tables, schemas different sets of tables, schemas different sets of tables, schemas they're unfamiliar with and they're unfamiliar with and they're unfamiliar with and idiosyncrasies in the data. And if idiosyncrasies in the data. And if idiosyncrasies in the data. And if you're a human data engineer, you learn you're a human data engineer, you learn you're a human data engineer, you learn those schemas over time. You learn how those schemas over time. You learn how those schemas over time. You learn how tables link together. You learn the tables link together. You learn the tables link together. You learn the idiosyncrasies in the data. But if I was idiosyncrasies in the data. But if I was idiosyncrasies in the data. But if I was an agent where my memory was being reset an agent where my memory was being reset an agent where my memory was being reset every single time, I wouldn't learn any every single time, I wouldn't learn any every single time, I wouldn't learn any of that, right? And so here what we can of that, right? And so here what we can of that, right? And so here what we can use as a measure for our ability to use as a measure for our ability to use as a measure for our ability to continually learn is the efficiency in continually learn is the efficiency in continually learn is the efficiency in the the task here is you're given a the the task here is you're given a the the task here is you're given a natural language question over the data natural language question over the data natural language question over the data and you have to answer it by performing and you have to answer it by performing and you have to answer it by performing some set of SQL queries right that gives some set of SQL queries right that gives some set of SQL queries right that gives you the answer from the from the you the answer from the from the you the answer from the from the databases but if you don't know anything databases but if you don't know anything databases but if you don't know anything about the databases it's going to take about the databases it's going to take about the databases it's going to take you a lot of SQL queries at the start you a lot of SQL queries at the start you a lot of SQL queries at the start and at the first questions you're going and at the first questions you're going and at the first questions you're going to be doing a lot of different to be doing a lot of different to be doing a lot of different exploration into figuring out what do exploration into figuring out what do exploration into figuring out what do the schemas look like what is the data the schemas look like what is the data the schemas look like what is the data format how do these link together but as format how do these link together but as format how do these link together but as you progress through the benchmark. If you progress through the benchmark. If you progress through the benchmark. If you're a continually learning system, you're a continually learning system, you're a continually learning system, maybe let's say by the 10th instance of maybe let's say by the 10th instance of maybe let's say by the 10th instance of a task, it'll look a little different, a task, it'll look a little different, a task, it'll look a little different, right? You'll you'll be able to know a right? You'll you'll be able to know a right? You'll you'll be able to know a lot more about the schemas, the lot more about the schemas, the lot more about the schemas, the idiosyncrasies of the data. You'll know idiosyncrasies of the data. You'll know idiosyncrasies of the data. You'll know how things link together, and it'll take how things link together, and it'll take how things link together, and it'll take you much less to actually answer the you much less to actually answer the you much less to actually answer the question.

question. question. But there's a bit of a there's a bit of But there's a bit of a there's a bit of But there's a bit of a there's a bit of a conundrum here. We're going to add a conundrum here. We're going to add a conundrum here. We're going to add something a little more challenging to something a little more challenging to something a little more challenging to this task, too. In the real world, this task, too. In the real world, this task, too. In the real world, there's often concept drift in the there's often concept drift in the there's often concept drift in the things you're doing. In the database things you're doing. In the database things you're doing. In the database example, let's say there's a migration example, let's say there's a migration example, let's say there's a migration of your database, right? Columns get of your database, right? Columns get of your database, right? Columns get dropped. There's new columns with dropped. There's new columns with dropped. There's new columns with different names added. The data format different names added. The data format different names added. The data format changes. As a human, you reason through changes. As a human, you reason through changes. As a human, you reason through all this uncertainty and you're able to all this uncertainty and you're able to all this uncertainty and you're able to update your priors. You know that update your priors. You know that update your priors. You know that there's still some information from the there's still some information from the there's still some information from the past that may be relevant, but from your past that may be relevant, but from your past that may be relevant, but from your exploration, you learn what to forget exploration, you learn what to forget exploration, you learn what to forget and you learn what's actually relevant and you learn what's actually relevant and you learn what's actually relevant to maintain over time. You kind of have to maintain over time. You kind of have to maintain over time. You kind of have an innate ability to maintain the an innate ability to maintain the an innate ability to maintain the stability and plasticity trade-off in stability and plasticity trade-off in stability and plasticity trade-off in your mind. But this isn't native to a your mind. But this isn't native to a your mind. But this isn't native to a lot of language models. And so if we lot of language models. And so if we lot of language models. And so if we give it a task after the database give it a task after the database give it a task after the database migration, it might struggle to require migration, it might struggle to require migration, it might struggle to require it might struggle in its ability to it might struggle in its ability to it might struggle in its ability to detect and discard stale experience and detect and discard stale experience and detect and discard stale experience and simultaneously update from new simultaneously update from new simultaneously update from new experience. And so this n this notion of experience. And so this n this notion of experience. And so this n this notion of concept drifts is something we try to concept drifts is something we try to concept drifts is something we try to add a lot to our tasks to further test add a lot to our tasks to further test add a lot to our tasks to further test the limits of what memory and continual the limits of what memory and continual the limits of what memory and continual learning might look like in language learning might look like in language learning might look like in language model systems.

model systems. model systems. In continual learning bench 1.0, know In continual learning bench 1.0, know In continual learning bench 1.0, know which is the benchmark I'm talking about which is the benchmark I'm talking about which is the benchmark I'm talking about today. We have tasks from six different today. We have tasks from six different today. We have tasks from six different domains of tasks. Okay, we have blind domains of tasks. Okay, we have blind domains of tasks. Okay, we have blind spectrum monitoring which is a signal spectrum monitoring which is a signal spectrum monitoring which is a signal processing task, codebase adaptation processing task, codebase adaptation processing task, codebase adaptation which is a software engineering which is a software engineering which is a software engineering efficiency, cohort studies and efficiency, cohort studies and efficiency, cohort studies and epidemiology task, exploitable poker, a epidemiology task, exploitable poker, a epidemiology task, exploitable poker, a strategic game playing task and because strategic game playing task and because strategic game playing task and because my friends like to play poker, and a my friends like to play poker, and a my friends like to play poker, and a database exploration task, the one I database exploration task, the one I database exploration task, the one I just showed you, and sales prediction, just showed you, and sales prediction, just showed you, and sales prediction, which is more of a data sciency which is more of a data sciency which is more of a data sciency prediction error style task. Across each prediction error style task. Across each prediction error style task. Across each of these task domains, we construct of these task domains, we construct of these task domains, we construct sequences of individual task instances sequences of individual task instances sequences of individual task instances with defined reward metrics that agents with defined reward metrics that agents with defined reward metrics that agents or models are tasked with solving or models are tasked with solving or models are tasked with solving gradually over time. And all of these gradually over time. And all of these gradually over time. And all of these task instances across domains we task instances across domains we task instances across domains we validate with domain experts to see is validate with domain experts to see is validate with domain experts to see is this learnable? Is this realistic? Is this learnable? Is this realistic? Is this learnable? Is this realistic? Is these the sorts of drifts you would these the sorts of drifts you would these the sorts of drifts you would expect and things you would expect to expect and things you would expect to expect and things you would expect to remember in an environment. remember in an environment. remember in an environment. Let's jump quickly into the results. Let's jump quickly into the results. Let's jump quickly into the results. In the first in initial release of the In the first in initial release of the In the first in initial release of the continual learning bench, we evaluated a continual learning bench, we evaluated a continual learning bench, we evaluated a lot of these context management systems lot of these context management systems lot of these context management systems to begin with and there's some to begin with and there's some to begin with and there's some parametric systems down the line too. If parametric systems down the line too. If parametric systems down the line too. If we look at just aggregate reward for a we look at just aggregate reward for a we look at just aggregate reward for a second and ignore gain and cost, you'll second and ignore gain and cost, you'll second and ignore gain and cost, you'll kind of see that the incontext learning kind of see that the incontext learning kind of see that the incontext learning systems, this is vanilla in context systems, this is vanilla in context systems, this is vanilla in context learning where you just put the learning where you just put the learning where you just put the experience in the context and you don't experience in the context and you don't experience in the context and you don't do any of the fancy context management do any of the fancy context management do any of the fancy context management that some of these other systems do. It that some of these other systems do. It that some of these other systems do. It tops the leaderboard and and it's not tops the leaderboard and and it's not tops the leaderboard and and it's not just on reward. It's actually also when just on reward. It's actually also when just on reward. It's actually also when we look at the Pareto frontiers, this we look at the Pareto frontiers, this we look at the Pareto frontiers, this kind of holds across reward uh versus kind of holds across reward uh versus kind of holds across reward uh versus cost and gain versus cost. I don't cost and gain versus cost. I don't cost and gain versus cost. I don't necessarily think this is what the end necessarily think this is what the end necessarily think this is what the end state of continual learning might look

state of continual learning might look state of continual learning might look like. You could argue that these were like. You could argue that these were like. You could argue that these were medium horizon tasks and they didn't medium horizon tasks and they didn't medium horizon tasks and they didn't push the frontiers of the incontext push the frontiers of the incontext push the frontiers of the incontext learning systems enough and I I would learning systems enough and I I would learning systems enough and I I would say that's fair and that's one of the say that's fair and that's one of the say that's fair and that's one of the things on our road map to push those things on our road map to push those things on our road map to push those even further. But it was still even further. But it was still even further. But it was still surprising that these more expensive surprising that these more expensive surprising that these more expensive context management systems perform a lot context management systems perform a lot context management systems perform a lot poorly compared to just vanilla in poorly compared to just vanilla in poorly compared to just vanilla in context learning on these sets of tasks context learning on these sets of tasks context learning on these sets of tasks where you have to do real learning. where you have to do real learning. where you have to do real learning. These are real tasks that you might These are real tasks that you might These are real tasks that you might expect a normal person to do. I think expect a normal person to do. I think expect a normal person to do. I think it's also interesting to chat briefly it's also interesting to chat briefly it's also interesting to chat briefly about what some of the failure modes about what some of the failure modes about what some of the failure modes look like when we observed in this look like when we observed in this look like when we observed in this initial version of the benchmark. I'm initial version of the benchmark. I'm initial version of the benchmark. I'm going to argue that most failure modes going to argue that most failure modes going to argue that most failure modes in continual learning fall on one side in continual learning fall on one side in continual learning fall on one side of the stability plasticity trade-off. of the stability plasticity trade-off. of the stability plasticity trade-off. Okay. Okay. And stability is your Okay. Okay. And stability is your Okay. Okay. And stability is your ability to retain new information in a ability to retain new information in a ability to retain new information in a stable way and use that for future stable way and use that for future stable way and use that for future tasks. While plasticity is your ability tasks. While plasticity is your ability tasks. While plasticity is your ability to actually learn from new information. to actually learn from new information. to actually learn from new information. We see usually that with most continual We see usually that with most continual We see usually that with most continual learning methods, any sort of failure learning methods, any sort of failure learning methods, any sort of failure mode comes from the inability to do one mode comes from the inability to do one mode comes from the inability to do one of these things. And I'll dive into of these things. And I'll dive into of these things. And I'll dive into examples from both. You don't have to examples from both. You don't have to examples from both. You don't have to read too much of the text here, but this read too much of the text here, but this read too much of the text here, but this is our sales prediction task where the is our sales prediction task where the is our sales prediction task where the model's tasked with making some model's tasked with making some model's tasked with making some predictions or forecasting for sales predictions or forecasting for sales predictions or forecasting for sales over the course of I think five years.

over the course of I think five years. over the course of I think five years. The model starts by getting feedback The model starts by getting feedback The model starts by getting feedback that it overpredicted for the last uh that it overpredicted for the last uh that it overpredicted for the last uh the last submission it set. And so what the last submission it set. And so what the last submission it set. And so what it does is that it then leads to it it does is that it then leads to it it does is that it then leads to it revises its prediction downwards to a revises its prediction downwards to a revises its prediction downwards to a lower prediction for the set of five lower prediction for the set of five lower prediction for the set of five years. And then it gets feedback that years. And then it gets feedback that years. And then it gets feedback that actually its underprediction was too actually its underprediction was too actually its underprediction was too much of an underprediction. And so as a much of an underprediction. And so as a much of an underprediction. And so as a human might naturally do you would go human might naturally do you would go human might naturally do you would go for the middle. But that's not what the for the middle. But that's not what the for the middle. But that's not what the model does. The model kind of forgot model does. The model kind of forgot model does. The model kind of forgot that it had the over prediction to begin that it had the over prediction to begin that it had the over prediction to begin with at the start and it just re jumps with at the start and it just re jumps with at the start and it just re jumps back right to the over prediction. This back right to the over prediction. This back right to the over prediction. This is kind of example of what stability is kind of example of what stability is kind of example of what stability failures look like where it's not failures look like where it's not failures look like where it's not retaining past information in improving retaining past information in improving retaining past information in improving its future experience. its future experience. its future experience. Plasticity is a bit of a different Plasticity is a bit of a different Plasticity is a bit of a different story. This is from our epidemiology story. This is from our epidemiology story. This is from our epidemiology task. And the only words I really want task. And the only words I really want task. And the only words I really want you guys to read this is a this is a you guys to read this is a this is a you guys to read this is a this is a system with with a notepad. Okay? It's a system with with a notepad. Okay? It's a system with with a notepad. Okay? It's a context management system that's given a context management system that's given a context management system that's given a notepad to externalize its memory to the notepad to externalize its memory to the notepad to externalize its memory to the highlighted text is is the important highlighted text is is the important highlighted text is is the important thing to read here. Quote, "These seem thing to read here. Quote, "These seem thing to read here. Quote, "These seem to be cohort definitions from a to be cohort definitions from a to be cohort definitions from a different study schema that doesn't different study schema that doesn't different study schema that doesn't apply here. apply here. apply here. the study schema did in fact apply here the study schema did in fact apply here the study schema did in fact apply here that the the model was unable to update that the the model was unable to update that the the model was unable to update its priors and data from new information its priors and data from new information its priors and data from new information and meaningfully improve over time. It and meaningfully improve over time. It and meaningfully improve over time. It just didn't even recognize that this is just didn't even recognize that this is just didn't even recognize that this is something that's relevant to the task something that's relevant to the task something that's relevant to the task and use that to update its priors.

and use that to update its priors. and use that to update its priors. Before I finish, I want to offer a Before I finish, I want to offer a Before I finish, I want to offer a little bit of my broader thoughts on little bit of my broader thoughts on little bit of my broader thoughts on continual learning. I alluded to this continual learning. I alluded to this continual learning. I alluded to this before that I have some opinions on what before that I have some opinions on what before that I have some opinions on what methods for continual learning might methods for continual learning might methods for continual learning might actually look like. I'm personally quite actually look like. I'm personally quite actually look like. I'm personally quite excited about the parametric methods for excited about the parametric methods for excited about the parametric methods for continual learning that look at continual learning that look at continual learning that look at alternative architectures, data, and alternative architectures, data, and alternative architectures, data, and algorithms jointly to optimize what algorithms jointly to optimize what algorithms jointly to optimize what continual learning should look like. If continual learning should look like. If continual learning should look like. If you look at language models today, the you look at language models today, the you look at language models today, the training stack used to look something training stack used to look something training stack used to look something simple like this, or maybe it never did, simple like this, or maybe it never did, simple like this, or maybe it never did, but I'm simplifying for for the sake of but I'm simplifying for for the sake of but I'm simplifying for for the sake of this pre-training, then we did this pre-training, then we did this pre-training, then we did supervised fine-tuning, and then we do supervised fine-tuning, and then we do supervised fine-tuning, and then we do RHF. But the stack kind of adapted over RHF. But the stack kind of adapted over RHF. But the stack kind of adapted over the course of the last few years. Now it the course of the last few years. Now it the course of the last few years. Now it might look something more like this might look something more like this might look something more like this where you have mid-training, you have RL where you have mid-training, you have RL where you have mid-training, you have RL for different teacher models and you for different teacher models and you for different teacher models and you finish everything off with multi finish everything off with multi finish everything off with multi multi-teer on policy distillation. multi-teer on policy distillation. multi-teer on policy distillation. A lot of the work in continual learning A lot of the work in continual learning A lot of the work in continual learning today is how do we take these models today is how do we take these models today is how do we take these models that are already trained and then kind that are already trained and then kind that are already trained and then kind of that frozen checkpoint. How do we of that frozen checkpoint. How do we of that frozen checkpoint. How do we figure out methods for continual figure out methods for continual figure out methods for continual learning that work after the fact? But learning that work after the fact? But learning that work after the fact? But these models were never designed to be these models were never designed to be these models were never designed to be continual learners to begin with. And continual learners to begin with. And continual learners to begin with. And one of my hypotheses here at least is one of my hypotheses here at least is one of my hypotheses here at least is that we're operating in a bit of a sunk that we're operating in a bit of a sunk that we're operating in a bit of a sunk cost fallacy that because we've trained cost fallacy that because we've trained cost fallacy that because we've trained models the way we are today, we need models the way we are today, we need models the way we are today, we need continual learning methods that work on continual learning methods that work on continual learning methods that work on top of that. If you were to think of top of that. If you were to think of top of that. If you were to think of designing what continual learning looks designing what continual learning looks designing what continual learning looks like as a first order principle and a like as a first order principle and a like as a first order principle and a first order requirement, it might look first order requirement, it might look first order requirement, it might look nothing like this. In the purest sense, nothing like this. In the purest sense, nothing like this. In the purest sense, continual learning might just be one continual learning might just be one continual learning might just be one set, one phase of training for continual set, one phase of training for continual set, one phase of training for continual learning and everything after that is learning and everything after that is learning and everything after that is deployment. You're just the model is deployment. You're just the model is deployment. You're just the model is interacting in the environment. It's interacting in the environment. It's interacting in the environment. It's updating its weights. there's only one updating its weights. there's only one updating its weights. there's only one phase of learning and we don't phase of learning and we don't phase of learning and we don't necessarily have to assume that

necessarily have to assume that necessarily have to assume that everything is built on top of the way we everything is built on top of the way we everything is built on top of the way we do language modeling today just because do language modeling today just because do language modeling today just because we've spent so much doing that there we've spent so much doing that there we've spent so much doing that there there are there are different ways to there are there are different ways to there are there are different ways to explore where if you think about explore where if you think about explore where if you think about continual learning as a first order continual learning as a first order continual learning as a first order requirement for your systems requirement for your systems requirement for your systems more broadly here for a second I think more broadly here for a second I think more broadly here for a second I think that continual learning is just one that continual learning is just one that continual learning is just one example of my group's broader effort to example of my group's broader effort to example of my group's broader effort to challenge the way we do things in challenge the way we do things in challenge the way we do things in training today I think that there are training today I think that there are training today I think that there are larger questions going on though and larger questions going on though and larger questions going on though and opportunities to challenge how we do AI opportunities to challenge how we do AI opportunities to challenge how we do AI research as a whole. There's a lot of research as a whole. There's a lot of research as a whole. There's a lot of questions around open science, questions around open science, questions around open science, consolidation of power, safety, what the consolidation of power, safety, what the consolidation of power, safety, what the future of things like academia or these future of things like academia or these future of things like academia or these third party institutions might look third party institutions might look third party institutions might look like. And I think that there's a huge like. And I think that there's a huge like. And I think that there's a huge opportunity for us to rethink a lot of opportunity for us to rethink a lot of opportunity for us to rethink a lot of that. And I've spent some time doing that. And I've spent some time doing that. And I've spent some time doing some more writing about that that I'd some more writing about that that I'd some more writing about that that I'd encourage people to check out. But the encourage people to check out. But the encourage people to check out. But the crux being there there are opportunities crux being there there are opportunities crux being there there are opportunities to reimagine what third party to reimagine what third party to reimagine what third party institutions for AI research look like institutions for AI research look like institutions for AI research look like and what the future of open science and what the future of open science and what the future of open science should look like that I think we all should look like that I think we all should look like that I think we all need to spend more time getting involved need to spend more time getting involved need to spend more time getting involved in. in. in. To wrap up, uh, I want to thank a lot of To wrap up, uh, I want to thank a lot of To wrap up, uh, I want to thank a lot of my contributors on continual learning my contributors on continual learning my contributors on continual learning bench. This is a collaboration across bench. This is a collaboration across bench. This is a collaboration across Berkeley, Snorkel, Udub, Madison. Uh, Berkeley, Snorkel, Udub, Madison. Uh, Berkeley, Snorkel, Udub, Madison. Uh, thanking the snorkel AI for their open thanking the snorkel AI for their open thanking the snorkel AI for their open their grant through the open benchmarks their grant through the open benchmarks their grant through the open benchmarks grant program and the LOD institute via grant program and the LOD institute via grant program and the LOD institute via their LOD slingshots program. Uh, in the their LOD slingshots program. Uh, in the their LOD slingshots program. Uh, in the road map, we have more domain specific road map, we have more domain specific road map, we have more domain specific and longer horizon tasks on the and longer horizon tasks on the and longer horizon tasks on the benchmark. We want to add more of the benchmark. We want to add more of the benchmark. We want to add more of the OSS models and the parametric OSS models and the parametric OSS models and the parametric approaches. We want to improve what approaches. We want to improve what approaches. We want to improve what simulation for task environments outside simulation for task environments outside simulation for task environments outside of easily verifiable deterministic of easily verifiable deterministic of easily verifiable deterministic domains look like. What would continual domains look like. What would continual domains look like. What would continual learning where you're personalizing to a learning where you're personalizing to a learning where you're personalizing to a user model do? I don't think we have user model do? I don't think we have user model do? I don't think we have models that are capable enough of

models that are capable enough of models that are capable enough of simulating that right now to figure it simulating that right now to figure it simulating that right now to figure it out, but I think we should. out, but I think we should. out, but I think we should. If there's one thing to take away, If there's one thing to take away, If there's one thing to take away, continual learning doesn't look like continual learning doesn't look like continual learning doesn't look like point capabilities. We need to measure point capabilities. We need to measure point capabilities. We need to measure it the right way to optimize for the it the right way to optimize for the it the right way to optimize for the right objective as a field because right objective as a field because right objective as a field because that's a history of how machine learning that's a history of how machine learning that's a history of how machine learning has progressed. Thank you guys very much has progressed. Thank you guys very much has progressed. Thank you guys very much for listening. Please feel free to email for listening. Please feel free to email for listening. Please feel free to email me or follow for updates. I would love me or follow for updates. I would love me or follow for updates. I would love to stick around, but I'm going to catch to stick around, but I'm going to catch to stick around, but I'm going to catch a flight, but I'm happy to take anything a flight, but I'm happy to take anything a flight, but I'm happy to take anything afterwards via email. Thank you guys afterwards via email. Thank you guys afterwards via email. Thank you guys very much.