The Base Model Is Dead — Varun Singh, Arcee AI

Hi everyone. My name is Varon. I'm the Hi everyone. My name is Varon. I'm the pre-training lead at RCAI. pre-training lead at RCAI. pre-training lead at RCAI. And the talk I'm going to be giving And the talk I'm going to be giving And the talk I'm going to be giving today is called the base model is dead. today is called the base model is dead. today is called the base model is dead. Um but not really. Um Um but not really. Um Um but not really. Um the idea of the base model that we have the idea of the base model that we have the idea of the base model that we have um um um kind of is like kind of is like kind of is like built on uh this idea of like training built on uh this idea of like training built on uh this idea of like training on super large-scale web text and the on super large-scale web text and the on super large-scale web text and the base model kind of being a reflection of base model kind of being a reflection of base model kind of being a reflection of like the whole like the whole like the whole knowledge of like the uh human internet. knowledge of like the uh human internet. knowledge of like the uh human internet. Um Um Um you can see in like uh you can see in like uh you can see in like uh these I've I've taken these uh from a these I've I've taken these uh from a these I've I've taken these uh from a bunch of uh different papers on like the bunch of uh different papers on like the bunch of uh different papers on like the entire LM training process. entire LM training process. entire LM training process. Um our own model RC uh training large Um our own model RC uh training large Um our own model RC uh training large thinking, um the process looked kind of thinking, um the process looked kind of thinking, um the process looked kind of like the like the like the simplified diagram on the left. I've simplified diagram on the left. I've simplified diagram on the left. I've taken the top one from uh GLM 4.5, the taken the top one from uh GLM 4.5, the taken the top one from uh GLM 4.5, the bottom one from GLM 5. bottom one from GLM 5. bottom one from GLM 5. Um Um Um all these have a pre-training phase and all these have a pre-training phase and all these have a pre-training phase and uh uh uh pre-training is like the stage where the pre-training is like the stage where the pre-training is like the stage where the model look accumulates world knowledge, model look accumulates world knowledge, model look accumulates world knowledge, builds useful representations builds useful representations builds useful representations uh all through next token prediction on uh all through next token prediction on uh all through next token prediction on um um um web text.

web text. web text. Um Um Um I've got a simplified uh transformer I've got a simplified uh transformer I've got a simplified uh transformer diagram um decoder-only transformer. diagram um decoder-only transformer. diagram um decoder-only transformer. And um And um And um a screenshot from the GPT-3 paper that a screenshot from the GPT-3 paper that a screenshot from the GPT-3 paper that talks about how talks about how talks about how um language models can um language models can um language models can uh learn how to do in-context learning uh learn how to do in-context learning uh learn how to do in-context learning through um through um through um unsupervised or self-supervised or unsupervised or self-supervised or unsupervised or self-supervised or um some some even just call it um some some even just call it um some some even just call it supervised um learning on uh through supervised um learning on uh through supervised um learning on uh through next token prediction. Um Um the way that uh older base models were the way that uh older base models were the way that uh older base models were trained was, like I said, mostly on trained was, like I said, mostly on trained was, like I said, mostly on uh things that reflected the entirety of uh things that reflected the entirety of uh things that reflected the entirety of human knowledge. human knowledge. human knowledge. Uh so, Common Crawl, uh which is like a Uh so, Common Crawl, uh which is like a Uh so, Common Crawl, uh which is like a commonly available web scrape, commonly available web scrape, commonly available web scrape, uh made up most of the training data set uh made up most of the training data set uh made up most of the training data set for GPT-3. for GPT-3. for GPT-3. Um WebText-2, another web scrape data Um WebText-2, another web scrape data Um WebText-2, another web scrape data set. Um some sources from like uh books set. Um some sources from like uh books set. Um some sources from like uh books as well. Um as well. Um as well. Um and Wikipedia as like a high-quality and Wikipedia as like a high-quality and Wikipedia as like a high-quality representation of human knowledge.

representation of human knowledge. representation of human knowledge. Um you can see that Um you can see that Um you can see that uh WebText alone here, including like uh WebText alone here, including like uh WebText alone here, including like Wikipedia, makes up like roughly 85% of Wikipedia, makes up like roughly 85% of Wikipedia, makes up like roughly 85% of the whole training mix. the whole training mix. the whole training mix. Um Um Um looking at uh the bottom with Llama 3, looking at uh the bottom with Llama 3, looking at uh the bottom with Llama 3, um um um WebText still kind of makes up a WebText still kind of makes up a WebText still kind of makes up a majority of of the um model's training majority of of the um model's training majority of of the um model's training data with like 50% of the tokens data with like 50% of the tokens data with like 50% of the tokens corresponding to general knowledge. Back then, post-training was mostly Back then, post-training was mostly shaping the model to use shaping the model to use shaping the model to use um um um the parts it to like surface the parts it to like surface the parts it to like surface uh the knowledge that it accumulates and uh the knowledge that it accumulates and uh the knowledge that it accumulates and accumulated in pre-training uh in like a accumulated in pre-training uh in like a accumulated in pre-training uh in like a chat interface. chat interface. chat interface. So, mostly allowing the model to adapt So, mostly allowing the model to adapt So, mostly allowing the model to adapt to a chat template, to the to a chat template, to the to a chat template, to the question-answer format, question-answer format, question-answer format, um and be be useful in an interaction um and be be useful in an interaction um and be be useful in an interaction that way. that way. that way. Um Um Um RL was mostly just a cherry on top, RL was mostly just a cherry on top, RL was mostly just a cherry on top, um shaping the, you know, flavor of the um shaping the, you know, flavor of the um shaping the, you know, flavor of the interactions more than conferring extra interactions more than conferring extra interactions more than conferring extra um knowledge or quality onto the base um knowledge or quality onto the base um knowledge or quality onto the base model itself.

model itself. model itself. Now, Now, Now, in in like this realm of of how language in in like this realm of of how language in in like this realm of of how language models used to be, models used to be, models used to be, pre-training um and the base model kind pre-training um and the base model kind pre-training um and the base model kind of defined how good you were able to get of defined how good you were able to get of defined how good you were able to get a model. a model. a model. Um it was like the bulk of the compute Um it was like the bulk of the compute Um it was like the bulk of the compute uh budget and uh budget and uh budget and it was it was it was um, the the like core of the training um, the the like core of the training um, the the like core of the training process. process. process. Um, Um, Um, however, however, however, uh, this kind of changed a lot last year uh, this kind of changed a lot last year uh, this kind of changed a lot last year when um, when um, when um, OpenAI I guess 2024 actually. OpenAI OpenAI I guess 2024 actually. OpenAI OpenAI I guess 2024 actually. OpenAI released 01, released 01, released 01, um, pioneering reasoning models and um, pioneering reasoning models and um, pioneering reasoning models and DeepSeek uh, also released R1 in January DeepSeek uh, also released R1 in January DeepSeek uh, also released R1 in January 2025, 2025, 2025, um, allowing the whole world to know how um, allowing the whole world to know how um, allowing the whole world to know how to build these types of language models. to build these types of language models. to build these types of language models. And And And now we have this new uh, now we have this new uh, now we have this new uh, new new new um, um, um, use for reinforcement learning, which is use for reinforcement learning, which is use for reinforcement learning, which is no longer a cherry on top, no longer a cherry on top, no longer a cherry on top, but it can dramatically improve the but it can dramatically improve the but it can dramatically improve the performance of of the model on various performance of of the model on various performance of of the model on various different tasks.

different tasks. different tasks. Um, the the famous graphs from 01 there Um, the the famous graphs from 01 there Um, the the famous graphs from 01 there talking about uh, AIME performance, um, talking about uh, AIME performance, um, talking about uh, AIME performance, um, competitive math contest. competitive math contest. competitive math contest. Um, and then even later in the year we Um, and then even later in the year we Um, and then even later in the year we we saw Cloud Code um, coming to being we saw Cloud Code um, coming to being we saw Cloud Code um, coming to being as a way for developers to easily as a way for developers to easily as a way for developers to easily um, kind of use uh, language models in a um, kind of use uh, language models in a um, kind of use uh, language models in a in a terminal to build out applications in a terminal to build out applications in a terminal to build out applications as models got stronger and stronger on as models got stronger and stronger on as models got stronger and stronger on things like function calling. things like function calling. things like function calling. Um, and then people realized you could Um, and then people realized you could Um, and then people realized you could uh, you know, RL this end-to-end. uh, you know, RL this end-to-end. uh, you know, RL this end-to-end. Um, and now models could learn how to Um, and now models could learn how to Um, and now models could learn how to interact with software environments and interact with software environments and interact with software environments and build software and build software and build software and uh, perform really useful work. uh, perform really useful work. uh, perform really useful work. And so, And so, And so, um, the question then becomes like, um, the question then becomes like, um, the question then becomes like, is your standard base model still is your standard base model still is your standard base model still uh, what the best uh, uh, what the best uh, uh, what the best uh, what will be the best um, what will be the best um, what will be the best um, prior for the for the this large-scale prior for the for the this large-scale prior for the for the this large-scale reinforcement learning phase that uh, reinforcement learning phase that uh, reinforcement learning phase that uh, research knows and research knows and research knows and um agentic models now use.

um agentic models now use. um agentic models now use. And we can kind of see like in the in a And we can kind of see like in the in a And we can kind of see like in the in a few open research papers what the trend few open research papers what the trend few open research papers what the trend is where the trend is going. is where the trend is going. is where the trend is going. Um and interestingly enough, the um it Um and interestingly enough, the um it Um and interestingly enough, the um it seems like seems like seems like not super clear yet. Um not super clear yet. Um not super clear yet. Um I have my opinions on like synthetic I have my opinions on like synthetic I have my opinions on like synthetic data being the way forward, but I'm I've data being the way forward, but I'm I've data being the way forward, but I'm I've got like two contrasting uh perspectives got like two contrasting uh perspectives got like two contrasting uh perspectives here kind of in the slide. here kind of in the slide. here kind of in the slide. The top image is from the MEI uh The top image is from the MEI uh The top image is from the MEI uh Thinking 1 paper Thinking 1 paper Thinking 1 paper where they make a make it really large where they make a make it really large where they make a make it really large point to not use any synthetic data or point to not use any synthetic data or point to not use any synthetic data or any uh data from any other language any uh data from any other language any uh data from any other language model. model. model. Um and they really try to, you know, Um and they really try to, you know, Um and they really try to, you know, filter their web scripts for this as filter their web scripts for this as filter their web scripts for this as well. well. well. Um in order uh to kind of adhere to like Um in order uh to kind of adhere to like Um in order uh to kind of adhere to like the previous the previous the previous um paradigm of like um paradigm of like um paradigm of like uh using human knowledge as a way to uh using human knowledge as a way to uh using human knowledge as a way to bootstrap uh model representations and bootstrap uh model representations and bootstrap uh model representations and like like like uh capabilities. uh capabilities. uh capabilities. Um Um Um But I would say that this is also, like But I would say that this is also, like But I would say that this is also, like even even though they stuck with no even even though they stuck with no even even though they stuck with no synthetic data, synthetic data, synthetic data, the data mix that they've chosen here is the data mix that they've chosen here is the data mix that they've chosen here is still um still um still um totally different from what you'd expect totally different from what you'd expect totally different from what you'd expect in like a in like a in like a um in a in a classical uh language um in a in a classical uh language um in a in a classical uh language model.

model. model. And I mean, the the main reason for that And I mean, the the main reason for that And I mean, the the main reason for that is that web text, which used to make up is that web text, which used to make up is that web text, which used to make up like up to 85% of the train data in GPT like up to 85% of the train data in GPT like up to 85% of the train data in GPT uh 3, uh 3, uh 3, is now all the way down at 15%. is now all the way down at 15%. is now all the way down at 15%. And And And that I mean, that just shows that uh the that I mean, that just shows that uh the that I mean, that just shows that uh the value of like web text contributing to value of like web text contributing to value of like web text contributing to like the downstream like the downstream like the downstream um um um performance of like the models on RL and performance of like the models on RL and performance of like the models on RL and stuff is stuff is stuff is kind of kind of kind of uh uh uh it it's still important, but taking a it it's still important, but taking a it it's still important, but taking a backseat to things like code and stem backseat to things like code and stem backseat to things like code and stem abilities as the models kind of gain abilities as the models kind of gain abilities as the models kind of gain more real-world use cases related to to more real-world use cases related to to more real-world use cases related to to those. those. those. Um the other approach uh is to bring um Um the other approach uh is to bring um Um the other approach uh is to bring um post-training data and large-scale post-training data and large-scale post-training data and large-scale synthetic data back uh through pull it synthetic data back uh through pull it synthetic data back uh through pull it back through the process into the back through the process into the back through the process into the pre-training phase. pre-training phase. pre-training phase. Um the the bottom chart I've taken from Um the the bottom chart I've taken from Um the the bottom chart I've taken from NeMo Tron 3 Ultra. Um what they reveal NeMo Tron 3 Ultra. Um what they reveal NeMo Tron 3 Ultra. Um what they reveal there um there um there um the data recipe and not sure how the data recipe and not sure how the data recipe and not sure how readable it is, but these top three um readable it is, but these top three um readable it is, but these top three um on the left uh pie chart, the top three on the left uh pie chart, the top three on the left uh pie chart, the top three on the kind of right right side of it, on the kind of right right side of it, on the kind of right right side of it, uh they're all labeled SFT uh they're all labeled SFT uh they're all labeled SFT with with SFT as a prefix.

with with SFT as a prefix. with with SFT as a prefix. And And And that's the type of question and answer that's the type of question and answer that's the type of question and answer kind of chat data set that you'd you'd kind of chat data set that you'd you'd kind of chat data set that you'd you'd expect to see only in post-training, expect to see only in post-training, expect to see only in post-training, but by pulling it back into the process, but by pulling it back into the process, but by pulling it back into the process, they're able to like get the model to they're able to like get the model to they're able to like get the model to learn um learn um learn um the shape of these conversations and the shape of these conversations and the shape of these conversations and what kind of tasks they might be what kind of tasks they might be what kind of tasks they might be expected to do downstream expected to do downstream expected to do downstream um from the very beginning of the um from the very beginning of the um from the very beginning of the pre-training process. pre-training process. pre-training process. Um and uh this follows like uh a similar Um and uh this follows like uh a similar Um and uh this follows like uh a similar um um um trend in like diminishing uh the amount trend in like diminishing uh the amount trend in like diminishing uh the amount of web text used of web text used of web text used in the model. in the model. in the model. Um Yeah. Yeah. Um it's really interesting to see the Um it's really interesting to see the Um it's really interesting to see the the NeMo Tron series leans so heavily the NeMo Tron series leans so heavily the NeMo Tron series leans so heavily into synthetic data, but uh MAI Thinking into synthetic data, but uh MAI Thinking into synthetic data, but uh MAI Thinking 1 kind of lean in the opposite 1 kind of lean in the opposite 1 kind of lean in the opposite direction. direction. direction. Um I've I've I've just got this slide here I've I've I've just got this slide here as like a con uh easy contrast that as like a con uh easy contrast that as like a con uh easy contrast that people can see on the like amount of web people can see on the like amount of web people can see on the like amount of web text and the amount of books and stuff text and the amount of books and stuff text and the amount of books and stuff um being less of a percentage here.

um being less of a percentage here. um being less of a percentage here. And GPT-3 didn't even used to have any And GPT-3 didn't even used to have any And GPT-3 didn't even used to have any specific code data sets, but now code is specific code data sets, but now code is specific code data sets, but now code is like the dominating like the dominating like the dominating um um um data data subset that we have in uh data data subset that we have in uh data data subset that we have in uh pre-training recipes. Um so I mentioned synthetic data, but Um so I mentioned synthetic data, but what is actually uh like how is what is actually uh like how is what is actually uh like how is synthetic data used? There's a lot of uh synthetic data used? There's a lot of uh synthetic data used? There's a lot of uh talk around synthetic data that, you talk around synthetic data that, you talk around synthetic data that, you know, blindly tossing it into a model know, blindly tossing it into a model know, blindly tossing it into a model can cause the model to collapse and uh can cause the model to collapse and uh can cause the model to collapse and uh and performance to tank, but and performance to tank, but and performance to tank, but there's been a lot of work and uh even there's been a lot of work and uh even there's been a lot of work and uh even at like a large scale, at like a large scale, at like a large scale, you know, example of this uh turning out you know, example of this uh turning out you know, example of this uh turning out really well. really well. really well. Um Um Um so in our own in our in our own uh model so in our own in our in our own uh model so in our own in our in our own uh model Trinity Lodge, we uh Trinity Lodge, we uh Trinity Lodge, we uh had a large amount of uh web uh web had a large amount of uh web uh web had a large amount of uh web uh web scale synthetic data um mostly through scale synthetic data um mostly through scale synthetic data um mostly through rephrasing, rephrasing, rephrasing, where you take a seed data item and you where you take a seed data item and you where you take a seed data item and you sort of upsample it in the mix by uh sort of upsample it in the mix by uh sort of upsample it in the mix by uh generating synthetic rephrases of the generating synthetic rephrases of the generating synthetic rephrases of the same information. So, the model sees the same information. So, the model sees the same information. So, the model sees the same information in like multiple ways.

same information in like multiple ways. same information in like multiple ways. Um the bottom two uh screenshots are Um the bottom two uh screenshots are Um the bottom two uh screenshots are from Kimikay2, from Kimikay2, from Kimikay2, um um um an even larger scale model that uh an even larger scale model that uh an even larger scale model that uh broadly used this um broadly used this um broadly used this um across the whole pre-training data set. across the whole pre-training data set. across the whole pre-training data set. Um Um Um the top uh top right um screenshot is the top uh top right um screenshot is the top uh top right um screenshot is from a paper that uh from a paper that uh from a paper that uh resulted in the data sets Swallow Code resulted in the data sets Swallow Code resulted in the data sets Swallow Code and Swallow Math, which are early and Swallow Math, which are early and Swallow Math, which are early examples of this. examples of this. examples of this. But, the trend seems to be that um But, the trend seems to be that um But, the trend seems to be that um synthetic data not only allows you to synthetic data not only allows you to synthetic data not only allows you to get more and more tokens, uh but also, get more and more tokens, uh but also, get more and more tokens, uh but also, you know, clean up tokens, get higher you know, clean up tokens, get higher you know, clean up tokens, get higher quality tokens, and have um quality tokens, and have um quality tokens, and have um tokens that are shaped more like tokens that are shaped more like tokens that are shaped more like instruct or agentic tasks all the way instruct or agentic tasks all the way instruct or agentic tasks all the way back in pre-training and back in pre-training and back in pre-training and uh allowing the model to like learn uh allowing the model to like learn uh allowing the model to like learn those task representations from the very those task representations from the very those task representations from the very beginning. Um another reason that it's uh Um another reason that it's uh beneficial to add post-training data beneficial to add post-training data beneficial to add post-training data early in pre-training is now with MOEs, early in pre-training is now with MOEs, early in pre-training is now with MOEs, um one of the biggest uh pain points in um one of the biggest uh pain points in um one of the biggest uh pain points in training an MOE is dealing with load training an MOE is dealing with load training an MOE is dealing with load balancing.

balancing. balancing. Um, where experts can specialize over Um, where experts can specialize over Um, where experts can specialize over the course of training the course of training the course of training and um and um and um they uh and load balancing objectives they uh and load balancing objectives they uh and load balancing objectives aim to uh achieve broadly equal aim to uh achieve broadly equal aim to uh achieve broadly equal utilization of the experts utilization of the experts utilization of the experts um in a given batch or sequence um in a given batch or sequence um in a given batch or sequence depending on the objective. depending on the objective. depending on the objective. Um Um Um without uh post-training data in uh without uh post-training data in uh without uh post-training data in uh early in pre-training uh with an MOE one early in pre-training uh with an MOE one early in pre-training uh with an MOE one really easy pitfall uh that we can fall really easy pitfall uh that we can fall really easy pitfall uh that we can fall into is this is kind of illustrated in into is this is kind of illustrated in into is this is kind of illustrated in the MAI thinking one report, which is the MAI thinking one report, which is the MAI thinking one report, which is that the data distribution that the that the data distribution that the that the data distribution that the model sees uh in post-training is really model sees uh in post-training is really model sees uh in post-training is really really different uh compared to what it really different uh compared to what it really different uh compared to what it sees in pre-training. sees in pre-training. sees in pre-training. And this can cause massive imbalances And this can cause massive imbalances And this can cause massive imbalances and um and um and um MAI overcame it by MAI overcame it by MAI overcame it by uh really cranking up the load balancing uh really cranking up the load balancing uh really cranking up the load balancing coefficient during the SFT stages. coefficient during the SFT stages. coefficient during the SFT stages. Um Um Um but I mean, ideally you don't want to but I mean, ideally you don't want to but I mean, ideally you don't want to mess with the mess with the mess with the balance that far into training and the balance that far into training and the balance that far into training and the model should learn stable model should learn stable model should learn stable representations from really early on.

Another uh interesting thing that um Another uh interesting thing that um is changing in base models now is that is changing in base models now is that is changing in base models now is that you is this whole advent of you is this whole advent of you is this whole advent of mid-training, uh which is mid-training, uh which is mid-training, uh which is exposing the model to the distribution exposing the model to the distribution exposing the model to the distribution that it would see during post-training that it would see during post-training that it would see during post-training in RL and at a longer context, so for in RL and at a longer context, so for in RL and at a longer context, so for things like agentic traces to be allowed things like agentic traces to be allowed things like agentic traces to be allowed into the mix and uh to kind of help into the mix and uh to kind of help into the mix and uh to kind of help prepare the model that way. prepare the model that way. prepare the model that way. Um Um Um a lot of models though are training with a lot of models though are training with a lot of models though are training with much longer context in pre-training and much longer context in pre-training and much longer context in pre-training and there's no reason that these data sets there's no reason that these data sets there's no reason that these data sets can't be pulled back into the mix to can't be pulled back into the mix to can't be pulled back into the mix to allow for more stable representations allow for more stable representations allow for more stable representations from the very beginning. from the very beginning. from the very beginning. Um Um Um I think uh a better uh better way to I think uh a better uh better way to I think uh a better uh better way to understand the current uh phase of LM understand the current uh phase of LM understand the current uh phase of LM training uh isn't so much like training uh isn't so much like training uh isn't so much like pre-training, mid-training, pre-training, mid-training, pre-training, mid-training, post-training, RL. It all gets a bit post-training, RL. It all gets a bit post-training, RL. It all gets a bit muddy that way, but there's two broad muddy that way, but there's two broad muddy that way, but there's two broad paradigms that are like that really help paradigms that are like that really help paradigms that are like that really help build a LM today, and that's supervised build a LM today, and that's supervised build a LM today, and that's supervised learning learning learning to next token prediction to next token prediction to next token prediction and RL. and RL. and RL. Um and RL is becoming more and more Um and RL is becoming more and more Um and RL is becoming more and more important. Um important. Um important. Um The bottom thing is a screenshot from The bottom thing is a screenshot from The bottom thing is a screenshot from interview um interview um interview um with uh with uh with uh the head of Xiaomi's Mimo Labs, the head of Xiaomi's Mimo Labs, the head of Xiaomi's Mimo Labs, uh where she talks about how they uh where she talks about how they uh where she talks about how they allocate compute between research, allocate compute between research, allocate compute between research, pre-training, and post-training. And pre-training, and post-training. And pre-training, and post-training. And pre-training and post-training in the pre-training and post-training in the pre-training and post-training in the final model have a roughly equal compute final model have a roughly equal compute final model have a roughly equal compute allocation.

allocation. allocation. Um Composer 2.5 takes us to the extreme Um Composer 2.5 takes us to the extreme Um Composer 2.5 takes us to the extreme where where where um Kosar really um Kosar really um Kosar really uh sank much, much more RL compute into uh sank much, much more RL compute into uh sank much, much more RL compute into the model than the model had ever seen the model than the model had ever seen the model than the model had ever seen in supervised learning. in supervised learning. in supervised learning. But But But with RL dominating such a massive amount with RL dominating such a massive amount with RL dominating such a massive amount of the compute budget, uh it makes sense of the compute budget, uh it makes sense of the compute budget, uh it makes sense to view supervised learning as to view supervised learning as to view supervised learning as a way specifically to prepare the model a way specifically to prepare the model a way specifically to prepare the model for for for to build useful representations for for to build useful representations for for to build useful representations for for RL instead of it being the bulk of like RL instead of it being the bulk of like RL instead of it being the bulk of like um um um what the model would be used for like what the model would be used for like what the model would be used for like previously. Um there's been some some work on how Um there's been some some work on how supervised learning affects RL. supervised learning affects RL. supervised learning affects RL. Um I really like this one paper Um I really like this one paper Um I really like this one paper where the main takeaways are basically where the main takeaways are basically where the main takeaways are basically that that that uh the base model needs to have some uh the base model needs to have some uh the base model needs to have some exposure to like exposure to like exposure to like uh like the atomic skills that it would uh like the atomic skills that it would uh like the atomic skills that it would need to compose during RL, and um the need to compose during RL, and um the need to compose during RL, and um the model can learn to extrapolate from model can learn to extrapolate from model can learn to extrapolate from there there there during RL given like during RL given like during RL given like the environment has a sufficient level the environment has a sufficient level the environment has a sufficient level of difficulty.

of difficulty. of difficulty. Um Um Um I had to put in the classic AlphaGo I had to put in the classic AlphaGo I had to put in the classic AlphaGo graph there where RL eventually graph there where RL eventually graph there where RL eventually overtakes supervised learning. overtakes supervised learning. overtakes supervised learning. Uh it's unclear if we'll see something Uh it's unclear if we'll see something Uh it's unclear if we'll see something like for language models because of like for language models because of like for language models because of course, you know, uh, course, you know, uh, course, you know, uh, human language is such an insane human language is such an insane human language is such an insane distribution to have to like learn distribution to have to like learn distribution to have to like learn through reinforcement learning alone. through reinforcement learning alone. through reinforcement learning alone. Um, Um, Um, but it's definitely possible that we but it's definitely possible that we but it's definitely possible that we might see diminished supervised learning might see diminished supervised learning might see diminished supervised learning in more and more RL, uh, which makes in more and more RL, uh, which makes in more and more RL, uh, which makes this kind of this kind of this kind of thinking of a base model as, um, thinking of a base model as, um, thinking of a base model as, um, atomic skills for RL more and more atomic skills for RL more and more atomic skills for RL more and more valuable. valuable. valuable. Another thing that some labs are doing, Another thing that some labs are doing, Another thing that some labs are doing, um, is kind of um, is kind of um, is kind of introducing novel data, uh, during introducing novel data, uh, during introducing novel data, uh, during supervised learning. And by novel, I supervised learning. And by novel, I supervised learning. And by novel, I mean something that the model really mean something that the model really mean something that the model really wouldn't have seen the shape of before. wouldn't have seen the shape of before. wouldn't have seen the shape of before. Uh, easy example is, you know, reasoning Uh, easy example is, you know, reasoning Uh, easy example is, you know, reasoning traces. They don't really look like a traces. They don't really look like a traces. They don't really look like a like a ton of what, like a ton of what, like a ton of what, um, um, um, humans output. humans output. humans output. And, um, And, um, And, um, another interesting thing is like another interesting thing is like another interesting thing is like training for test time compute, training for test time compute, training for test time compute, um, um, um, schemes, um, by kind of warming the schemes, um, by kind of warming the schemes, um, by kind of warming the model up to them during SFT, um, model up to them during SFT, um, model up to them during SFT, um, or even pre-training itself.

or even pre-training itself. or even pre-training itself. Um, this These screenshots were taken Um, this These screenshots were taken Um, this These screenshots were taken from, uh, Zephyr's Ziya 1 paper. from, uh, Zephyr's Ziya 1 paper. from, uh, Zephyr's Ziya 1 paper. Um, and I think if they're, uh, Um, and I think if they're, uh, Um, and I think if they're, uh, very interesting ways of thinking about very interesting ways of thinking about very interesting ways of thinking about how data can affect, um, like, uh, the skills needed to explore like, uh, the skills needed to explore well in RL. well in RL. well in RL. Um, in conclusion, Um, in conclusion, Um, in conclusion, um, base models have kind of moved from um, base models have kind of moved from um, base models have kind of moved from general, uh, human knowledge and world general, uh, human knowledge and world general, uh, human knowledge and world priors to reasoning and agentic behavior priors to reasoning and agentic behavior priors to reasoning and agentic behavior priors. priors. priors. Um, of course, that's kind of, uh, Um, of course, that's kind of, uh, Um, of course, that's kind of, uh, reductive in the in the in a way that reductive in the in the in a way that reductive in the in the in a way that reasoning reasoners and agents are like reasoning reasoners and agents are like reasoning reasoners and agents are like the main way we see, uh, bots, uh, the main way we see, uh, bots, uh, the main way we see, uh, bots, uh, the main way we see these chatbots used the main way we see these chatbots used the main way we see these chatbots used now. now. now. But, But, But, if a new paradigm were to take off, like if a new paradigm were to take off, like if a new paradigm were to take off, like a new way of interacting with the a new way of interacting with the a new way of interacting with the models, models, models, um, it makes sense to like think of a um, it makes sense to like think of a um, it makes sense to like think of a base model as building a prior for that base model as building a prior for that base model as building a prior for that instead of, um, instead of, um, instead of, um, just building off like a a massive, uh, just building off like a a massive, uh, just building off like a a massive, uh, script of web text. script of web text. script of web text. And yeah.

And yeah. And yeah. Thanks for Thanks for listening. Thanks Thanks for Thanks for listening. Thanks Thanks for Thanks for listening. Thanks for your time. for your time. for your time. And And And >> [applause]