Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

Uh thanks all for coming to AI engineer Uh thanks all for coming to AI engineer and checking out the post training and checking out the post training and checking out the post training session. Um hopefully lots of fun stuff session. Um hopefully lots of fun stuff session. Um hopefully lots of fun stuff today and throughout the conference. today and throughout the conference. today and throughout the conference. Um I'm Will Brown. I lead applied Um I'm Will Brown. I lead applied Um I'm Will Brown. I lead applied research at Primordial AI and today I research at Primordial AI and today I research at Primordial AI and today I want to talk about reinforcement want to talk about reinforcement want to talk about reinforcement learning without verifiable rewards. And learning without verifiable rewards. And learning without verifiable rewards. And so many of people may have been learning so many of people may have been learning so many of people may have been learning about RLVR over the past uh year or so about RLVR over the past uh year or so about RLVR over the past uh year or so uh year and a half as this stuff has uh year and a half as this stuff has uh year and a half as this stuff has really taken off and become the main way really taken off and become the main way really taken off and become the main way that we think about scaling that we think about scaling that we think about scaling reinforcement learning. Um but often we reinforcement learning. Um but often we reinforcement learning. Um but often we don't actually have verifiable rewards. don't actually have verifiable rewards. don't actually have verifiable rewards. And so messy real world tasks often And so messy real world tasks often And so messy real world tasks often we're kind of figuring out as we go. we're kind of figuring out as we go. we're kind of figuring out as we go. We're having our agents run around and We're having our agents run around and We're having our agents run around and we kind of in hindsight maybe can like we kind of in hindsight maybe can like we kind of in hindsight maybe can like look at what they did and say like okay look at what they did and say like okay look at what they did and say like okay this is good this is bad. Um but this is good this is bad. Um but this is good this is bad. Um but sometimes sitting down and just like sometimes sitting down and just like sometimes sitting down and just like specifying okay this is the rule this is specifying okay this is the rule this is specifying okay this is the rule this is the goal is not always so the goal is not always so the goal is not always so straightforward. And so this is going to straightforward. And so this is going to straightforward. And so this is going to be uh synthesizing a lot of work we've be uh synthesizing a lot of work we've be uh synthesizing a lot of work we've been doing as well as from the broader been doing as well as from the broader been doing as well as from the broader research literature and some of the research literature and some of the research literature and some of the things we're building to kind of support things we're building to kind of support things we're building to kind of support extending RL into more messy real world extending RL into more messy real world extending RL into more messy real world tasks. tasks. tasks. And so recap quickly of how And so recap quickly of how And so recap quickly of how reinforcement learning works. I would reinforcement learning works. I would reinforcement learning works. I would imagine if you're in the post training imagine if you're in the post training imagine if you're in the post training session here you've probably heard a session here you've probably heard a session here you've probably heard a little bit about RL but for those of you little bit about RL but for those of you little bit about RL but for those of you at home and uh for those uh who are kind at home and uh for those uh who are kind at home and uh for those uh who are kind of still just kind of looking for the of still just kind of looking for the of still just kind of looking for the crash course generally we have an agent crash course generally we have an agent crash course generally we have an agent which we're going to call a model plus a which we're going to call a model plus a which we're going to call a model plus a harness which we place into an harness which we place into an harness which we place into an environment. And so an environment we're environment. And so an environment we're environment. And so an environment we're going to call a task plus a world. A going to call a task plus a world. A going to call a task plus a world. A world you could think of as maybe it's a world you could think of as maybe it's a world you could think of as maybe it's a docker image maybe it's a code base docker image maybe it's a code base docker image maybe it's a code base maybe it is a collection of uh maybe it is a collection of uh maybe it is a collection of uh task specific tools maybe it's some task specific tools maybe it's some task specific tools maybe it's some skills maybe it is a bunch of

skills maybe it is a bunch of skills maybe it is a bunch of applications or browser tabs or things applications or browser tabs or things applications or browser tabs or things like this as well as a scoring rule uh like this as well as a scoring rule uh like this as well as a scoring rule uh verifiers or rewards whatever you want verifiers or rewards whatever you want verifiers or rewards whatever you want to call them uh and the agent in the to call them uh and the agent in the to call them uh and the agent in the environment are going to interact in a environment are going to interact in a environment are going to interact in a loop and at the end we'll have some loop and at the end we'll have some loop and at the end we'll have some reward of how well the agent did in the reward of how well the agent did in the reward of how well the agent did in the environment for this task. environment for this task. environment for this task. And then reinforcement learning is all And then reinforcement learning is all And then reinforcement learning is all about creating an advantage, and the about creating an advantage, and the about creating an advantage, and the advantage is really about taking the advantage is really about taking the advantage is really about taking the reward minusing some baseline, and maybe reward minusing some baseline, and maybe reward minusing some baseline, and maybe doing some scaling, and then doing some scaling, and then doing some scaling, and then [clears throat] now you have a set of [clears throat] now you have a set of [clears throat] now you have a set of rollouts from the agent in the rollouts from the agent in the rollouts from the agent in the environment that you can use to then environment that you can use to then environment that you can use to then update the policy. The policy here is update the policy. The policy here is update the policy. The policy here is just the model weights themselves, and just the model weights themselves, and just the model weights themselves, and the goal here is to take a gradient the goal here is to take a gradient the goal here is to take a gradient which nudges the model towards getting which nudges the model towards getting which nudges the model towards getting higher reward. And so all the RL stuff higher reward. And so all the RL stuff higher reward. And so all the RL stuff people talk about whether it's GRPO or people talk about whether it's GRPO or people talk about whether it's GRPO or reinforce or SISPO or any of the other reinforce or SISPO or any of the other reinforce or SISPO or any of the other new algorithms people come up with, new algorithms people come up with, new algorithms people come up with, they're all kind of in this policy they're all kind of in this policy they're all kind of in this policy gradient framework, which is just about gradient framework, which is just about gradient framework, which is just about saying, "Okay, how do I make the model saying, "Okay, how do I make the model saying, "Okay, how do I make the model do things that have higher reward?" do things that have higher reward?" do things that have higher reward?" >> [clears throat] >> [clears throat] >> [clears throat] >> And so at Primary and Like we build a >> And so at Primary and Like we build a >> And so at Primary and Like we build a lot of tooling to power all this at lot of tooling to power all this at lot of tooling to power all this at every player. We kind of go both from every player. We kind of go both from every player. We kind of go both from the we start at the compute layer and do the we start at the compute layer and do the we start at the compute layer and do lots of large-scale GPU orchestration.

lots of large-scale GPU orchestration. lots of large-scale GPU orchestration. We build the Primary RL training We build the Primary RL training We build the Primary RL training framework, which powers all of our framework, which powers all of our framework, which powers all of our large-scale reinforcement learning and large-scale reinforcement learning and large-scale reinforcement learning and other algorithms running. We build other algorithms running. We build other algorithms running. We build environments. We have task sets and environments. We have task sets and environments. We have task sets and harnesses and verifiers as tools that harnesses and verifiers as tools that harnesses and verifiers as tools that you can mix and match to assemble you can mix and match to assemble you can mix and match to assemble complex worlds for agents to learn from complex worlds for agents to learn from complex worlds for agents to learn from real-world feedback. We have a training real-world feedback. We have a training real-world feedback. We have a training platform called Lab, which is anchored platform called Lab, which is anchored platform called Lab, which is anchored around environments where we do both around environments where we do both around environments where we do both hosted training and evaluations as well hosted training and evaluations as well hosted training and evaluations as well as inference. And this is to allow as inference. And this is to allow as inference. And this is to allow people to monitor their experiments and people to monitor their experiments and people to monitor their experiments and manage the training runs and iterate on manage the training runs and iterate on manage the training runs and iterate on their evals and deploy these models. And their evals and deploy these models. And their evals and deploy these models. And ultimately the models are starting ultimately the models are starting ultimately the models are starting generally from some open-source base generally from some open-source base generally from some open-source base model, model, model, and you're optimizing it for your task. and you're optimizing it for your task. and you're optimizing it for your task. And the goal that we're really trying to And the goal that we're really trying to And the goal that we're really trying to enable is more people to be able to enable is more people to be able to enable is more people to be able to become their own research lab, become become their own research lab, become become their own research lab, become uh uh uh take ownership over the intelligence of take ownership over the intelligence of take ownership over the intelligence of their own model weights and that their own model weights and that their own model weights and that optimize for the task that they care optimize for the task that they care optimize for the task that they care about with themselves as the experts about with themselves as the experts about with themselves as the experts steering the model, which means we need steering the model, which means we need steering the model, which means we need to make it way easier for people to do to make it way easier for people to do to make it way easier for people to do this. Currently, for a lot of people, this. Currently, for a lot of people, this. Currently, for a lot of people, it's still really hard. I think you can it's still really hard. I think you can it's still really hard. I think you can go like we're all here learning more go like we're all here learning more go like we're all here learning more about how it works because it's hard. We about how it works because it's hard. We about how it works because it's hard. We don't know how it all works and we're don't know how it all works and we're don't know how it all works and we're figuring it out as we go in many cases, figuring it out as we go in many cases, figuring it out as we go in many cases, but we've spent a lot of effort and a but we've spent a lot of effort and a but we've spent a lot of effort and a lot of time building stuff that lot of time building stuff that lot of time building stuff that hopefully makes this a bit easier for hopefully makes this a bit easier for hopefully makes this a bit easier for people.

people. people. Um and so what's an environment? An Um and so what's an environment? An Um and so what's an environment? An environment is tasks, harness, and environment is tasks, harness, and environment is tasks, harness, and rewards. Uh but it's not just for RL. So rewards. Uh but it's not just for RL. So rewards. Uh but it's not just for RL. So I think a lot of people think RL when I think a lot of people think RL when I think a lot of people think RL when they think environment, but environments they think environment, but environments they think environment, but environments and evals are really the same thing. You and evals are really the same thing. You and evals are really the same thing. You can use these same objects for can use these same objects for can use these same objects for generating static data, which then you generating static data, which then you generating static data, which then you can use for SFT. You can do RL or you can use for SFT. You can do RL or you can use for SFT. You can do RL or you can do algorithms like on-policy can do algorithms like on-policy can do algorithms like on-policy distillation. You can do prompt distillation. You can do prompt distillation. You can do prompt optimization like Jeph I. You can use it optimization like Jeph I. You can use it optimization like Jeph I. You can use it as a kind of scientific test bed to as a kind of scientific test bed to as a kind of scientific test bed to iterate on your agents and your iterate on your agents and your iterate on your agents and your harnesses. Um and verifiable rewards are harnesses. Um and verifiable rewards are harnesses. Um and verifiable rewards are the easy case where we just kind of can the easy case where we just kind of can the easy case where we just kind of can check exactly was something done check exactly was something done check exactly was something done correctly or not. And so for math, often correctly or not. And so for math, often correctly or not. And so for math, often if you have a numerical answer, you can if you have a numerical answer, you can if you have a numerical answer, you can just parse this out of like a box in the just parse this out of like a box in the just parse this out of like a box in the answer from the model and check. For answer from the model and check. For answer from the model and check. For code, maybe you want to use test cases code, maybe you want to use test cases code, maybe you want to use test cases or a linter or something like this. For or a linter or something like this. For or a linter or something like this. For tool use, often you have some database tool use, often you have some database tool use, often you have some database state, which you kind of know what you state, which you kind of know what you state, which you kind of know what you what you're expecting at the end and you what you're expecting at the end and you what you're expecting at the end and you can just kind of like check this can just kind of like check this can just kind of like check this deterministically. And so these are kind deterministically. And so these are kind deterministically. And so these are kind of the easy cases where of the easy cases where of the easy cases where the the the the the the uh reward design problem is not so uh reward design problem is not so uh reward design problem is not so difficult. Um but most real-world tasks difficult. Um but most real-world tasks difficult. Um but most real-world tasks are not this verifiable. Uh for a lot of are not this verifiable. Uh for a lot of are not this verifiable. Uh for a lot of real agent work tasks, we're having real agent work tasks, we're having real agent work tasks, we're having agents do things like write reports that agents do things like write reports that agents do things like write reports that maybe are analyzing a bunch of documents maybe are analyzing a bunch of documents maybe are analyzing a bunch of documents or research. Um we're maybe asked to do or research. Um we're maybe asked to do or research. Um we're maybe asked to do things like book flights or buy things, things like book flights or buy things, things like book flights or buy things, but there isn't always like a clean best but there isn't always like a clean best but there isn't always like a clean best answer here. Uh and there's also notions answer here. Uh and there's also notions answer here. Uh and there's also notions that are fuzzier like interacting with that are fuzzier like interacting with that are fuzzier like interacting with users like handling a refund. Like what users like handling a refund. Like what users like handling a refund. Like what does it mean to handle this well? Um and does it mean to handle this well? Um and does it mean to handle this well? Um and so here the signal is less clear and so here the signal is less clear and so here the signal is less clear and there's a lot of different tricks we there's a lot of different tricks we there's a lot of different tricks we might want to explore and techniques we might want to explore and techniques we might want to explore and techniques we want to develop to ensure that this can want to develop to ensure that this can want to develop to ensure that this can be done reliably and scalably.

be done reliably and scalably. be done reliably and scalably. Uh and making evals is hard because Uh and making evals is hard because Uh and making evals is hard because often times the benchmarks out there often times the benchmarks out there often times the benchmarks out there that we might like look at in the kind that we might like look at in the kind that we might like look at in the kind of uh new model releases, it's like a of uh new model releases, it's like a of uh new model releases, it's like a set of a few hundred tasks that a bunch set of a few hundred tasks that a bunch set of a few hundred tasks that a bunch of researchers spend months kind of of researchers spend months kind of of researchers spend months kind of handcrafting and talking to experts. handcrafting and talking to experts. handcrafting and talking to experts. Maybe they worked with data vendors and Maybe they worked with data vendors and Maybe they worked with data vendors and kind of like spent lots and lots of kind of like spent lots and lots of kind of like spent lots and lots of money kind of getting these to be very money kind of getting these to be very money kind of getting these to be very like precisely refined. And and this like precisely refined. And and this like precisely refined. And and this isn't very scalable out of the box isn't very scalable out of the box isn't very scalable out of the box um um um especially for things that are more especially for things that are more especially for things that are more open-ended where there's no kind of open-ended where there's no kind of open-ended where there's no kind of clean check for what's good or not. clean check for what's good or not. clean check for what's good or not. Um often the the the real-world Um often the the the real-world Um often the the the real-world situations can be unbound. You don't situations can be unbound. You don't situations can be unbound. You don't always know what things are going to be always know what things are going to be always know what things are going to be in the distribution. Like a lot of these in the distribution. Like a lot of these in the distribution. Like a lot of these cases we are figuring out the cases we are figuring out the cases we are figuring out the distribution as we go. And classical distribution as we go. And classical distribution as we go. And classical machine learning will tell you you can machine learning will tell you you can machine learning will tell you you can train for the distribution, but train for the distribution, but train for the distribution, but generalizing outside of the distribution generalizing outside of the distribution generalizing outside of the distribution is kind of an undefined problem. And is kind of an undefined problem. And is kind of an undefined problem. And then especially with RL, we have to be then especially with RL, we have to be then especially with RL, we have to be very careful about reward hacking. And very careful about reward hacking. And very careful about reward hacking. And so reward hacking is when you have a so reward hacking is when you have a so reward hacking is when you have a kind of loose proxy for your objective kind of loose proxy for your objective kind of loose proxy for your objective that is undefined at the boundaries. And that is undefined at the boundaries. And that is undefined at the boundaries. And then models, if you train with RL, they then models, if you train with RL, they then models, if you train with RL, they can learn to exploit this and find can learn to exploit this and find can learn to exploit this and find weaknesses where there's some path weaknesses where there's some path weaknesses where there's some path towards a climbing the the reward that towards a climbing the the reward that towards a climbing the the reward that doesn't actually give you what you want.

doesn't actually give you what you want. doesn't actually give you what you want. And so really the goal of what we would And so really the goal of what we would And so really the goal of what we would hope all of this builds into is hope all of this builds into is hope all of this builds into is continual learning, which is a big continual learning, which is a big continual learning, which is a big buzzword that I think a lot of people buzzword that I think a lot of people buzzword that I think a lot of people like talking about in many different like talking about in many different like talking about in many different ways, but I'm going to use it to mean a ways, but I'm going to use it to mean a ways, but I'm going to use it to mean a very particular thing, which is that very particular thing, which is that very particular thing, which is that we want models to be deployed in we want models to be deployed in we want models to be deployed in relatively realistic complex, messy relatively realistic complex, messy relatively realistic complex, messy settings and to be able to learn as they settings and to be able to learn as they settings and to be able to learn as they go where they are doing things, they are go where they are doing things, they are go where they are doing things, they are making mistakes, they are then able to making mistakes, they are then able to making mistakes, they are then able to observe and catch these mistakes after observe and catch these mistakes after observe and catch these mistakes after they happen, and use this to not do the they happen, and use this to not do the they happen, and use this to not do the same thing again. In some cases people same thing again. In some cases people same thing again. In some cases people want to try to do this at the harness want to try to do this at the harness want to try to do this at the harness layer of the front layer. layer of the front layer. layer of the front layer. But ultimately you want a system that But ultimately you want a system that But ultimately you want a system that can evolve autonomously to be able to can evolve autonomously to be able to can evolve autonomously to be able to get better over time with humans in the get better over time with humans in the get better over time with humans in the loop at that right level of abstraction. loop at that right level of abstraction. loop at that right level of abstraction. And I think currently the level of And I think currently the level of And I think currently the level of abstraction for doing this is far too abstraction for doing this is far too abstraction for doing this is far too low for it to be practical for most low for it to be practical for most low for it to be practical for most people. And so this means we need new people. And so this means we need new people. And so this means we need new methods to be able to uh automatize as methods to be able to uh automatize as methods to be able to uh automatize as much of the difficult processes as much of the difficult processes as much of the difficult processes as possible. And many of these actually are possible. And many of these actually are possible. And many of these actually are automatizable. They just are kind of automatizable. They just are kind of automatizable. They just are kind of difficult problems to solve. And so difficult problems to solve. And so difficult problems to solve. And so there's a few techniques you can you can there's a few techniques you can you can there's a few techniques you can you can use to kind of start making progress use to kind of start making progress use to kind of start making progress here, but one of the goals here is to do here, but one of the goals here is to do here, but one of the goals here is to do online reinforcement learning so that online reinforcement learning so that online reinforcement learning so that you can kind of iterate on this process you can kind of iterate on this process you can kind of iterate on this process as you go, as well as beyond just RL.

as you go, as well as beyond just RL. as you go, as well as beyond just RL. There are other things where you might There are other things where you might There are other things where you might want to incorporate world knowledge into want to incorporate world knowledge into want to incorporate world knowledge into the model itself, not just in terms of the model itself, not just in terms of the model itself, not just in terms of skill refinement. RL's great for skill refinement. RL's great for skill refinement. RL's great for refining skills, but less so for refining skills, but less so for refining skills, but less so for incorporating like dense new knowledge. incorporating like dense new knowledge. incorporating like dense new knowledge. And so blending these two together is And so blending these two together is And so blending these two together is also an important goal. And ultimately also an important goal. And ultimately also an important goal. And ultimately the what we want from this is to be able the what we want from this is to be able the what we want from this is to be able to deploy agents into production and to deploy agents into production and to deploy agents into production and have them improve as they go and have have them improve as they go and have have them improve as they go and have the these experiments be monitorable and the these experiments be monitorable and the these experiments be monitorable and traceable and replayable so that we can traceable and replayable so that we can traceable and replayable so that we can kind of treat kind of treat kind of treat model optimization very much as a model optimization very much as a model optimization very much as a science and make this science accessible science and make this science accessible science and make this science accessible to people who have a very wide variety to people who have a very wide variety to people who have a very wide variety of use cases they want to deploy agents of use cases they want to deploy agents of use cases they want to deploy agents for which don't all live in the training for which don't all live in the training for which don't all live in the training distributions of the big models. And so distributions of the big models. And so distributions of the big models. And so we want to be able to do this on top of we want to be able to do this on top of we want to be able to do this on top of the best and biggest open models in the the best and biggest open models in the the best and biggest open models in the world world world and make this accessible. And so how do and make this accessible. And so how do and make this accessible. And so how do you manufacture signal? There's a bunch you manufacture signal? There's a bunch you manufacture signal? There's a bunch of techniques that we've found very of techniques that we've found very of techniques that we've found very useful. useful. useful. One is grounding. And so grounding One is grounding. And so grounding One is grounding. And so grounding roughly means that you have some source roughly means that you have some source roughly means that you have some source material and in material and in material and in machine learning generally you want to machine learning generally you want to machine learning generally you want to have some notion of supervision. There's have some notion of supervision. There's have some notion of supervision. There's something you're learning from. something you're learning from. something you're learning from. And in messy situations we don't And in messy situations we don't And in messy situations we don't necessarily always have clean necessarily always have clean necessarily always have clean supervision, but we can get pretty supervision, but we can get pretty supervision, but we can get pretty reliable supervision if we kind of are reliable supervision if we kind of are reliable supervision if we kind of are careful about techniques we use. And so careful about techniques we use. And so careful about techniques we use. And so grounding is one where you have some grounding is one where you have some grounding is one where you have some source material and the ability to do an source material and the ability to do an source material and the ability to do an AB test of like with and without is a AB test of like with and without is a AB test of like with and without is a very useful way of creating this kind of very useful way of creating this kind of very useful way of creating this kind of capability gap where a model will do capability gap where a model will do capability gap where a model will do better if it has something in context.

better if it has something in context. better if it has something in context. And this gap is something we can exploit And this gap is something we can exploit And this gap is something we can exploit to create signal that we can then learn to create signal that we can then learn to create signal that we can then learn from. Judges are also really useful. from. Judges are also really useful. from. Judges are also really useful. We're relying on the fact that LLMs are We're relying on the fact that LLMs are We're relying on the fact that LLMs are already really powerful general already really powerful general already really powerful general reasoners for many things. And if we reasoners for many things. And if we reasoners for many things. And if we assign the if we set these judges up in assign the if we set these judges up in assign the if we set these judges up in the right way, then they can spend the right way, then they can spend the right way, then they can spend computes to make decisions about whether computes to make decisions about whether computes to make decisions about whether an action was good or bad. We also want an action was good or bad. We also want an action was good or bad. We also want to be scaling search. So search in many to be scaling search. So search in many to be scaling search. So search in many ways is something we can apply at many ways is something we can apply at many ways is something we can apply at many different layers of the pipeline both in different layers of the pipeline both in different layers of the pipeline both in terms of creating tasks as well as the terms of creating tasks as well as the terms of creating tasks as well as the worlds as well as the criteria for which worlds as well as the criteria for which worlds as well as the criteria for which we want to be giving judges for we want to be giving judges for we want to be giving judges for answering questions about the quality of answering questions about the quality of answering questions about the quality of a rollout. a rollout. a rollout. And so for source material, um one very And so for source material, um one very And so for source material, um one very useful version of this, especially for useful version of this, especially for useful version of this, especially for this continual learning goal, is this continual learning goal, is this continual learning goal, is production traces themselves. And so production traces themselves. And so production traces themselves. And so what we found is super helpful is taking what we found is super helpful is taking what we found is super helpful is taking existing traces from a deployed agent existing traces from a deployed agent existing traces from a deployed agent and treating these as the source and treating these as the source and treating these as the source material where we don't necessarily know material where we don't necessarily know material where we don't necessarily know up front what the distribution of tasks up front what the distribution of tasks up front what the distribution of tasks is, but as an agent is deployed, you is, but as an agent is deployed, you is, but as an agent is deployed, you start collecting more and more examples start collecting more and more examples start collecting more and more examples of let's say user prompts or uh of let's say user prompts or uh of let's say user prompts or uh uh calls from an orchestrator agent down uh calls from an orchestrator agent down uh calls from an orchestrator agent down into a sub-agent. And this starts into a sub-agent. And this starts into a sub-agent. And this starts becoming the distribution. We don't have becoming the distribution. We don't have becoming the distribution. We don't have labels yet, but it tells us at least labels yet, but it tells us at least labels yet, but it tells us at least what we want to look for. Um and so this what we want to look for. Um and so this what we want to look for. Um and so this is one very useful category, especially is one very useful category, especially is one very useful category, especially for other things like uh search or for for other things like uh search or for for other things like uh search or for code. Uh you have doc corpora corpora code. Uh you have doc corpora corpora code. Uh you have doc corpora corpora corpora of documents, you have repos corpora of documents, you have repos corpora of documents, you have repos that are also very useful for kind of that are also very useful for kind of that are also very useful for kind of anchoring your um anchoring your um anchoring your um your uh your learning as well. That are your uh your learning as well. That are your uh your learning as well. That are And so uh taking these sources, these And so uh taking these sources, these And so uh taking these sources, these raw materials, as places to kind of raw materials, as places to kind of raw materials, as places to kind of search for tasks from is a very useful search for tasks from is a very useful search for tasks from is a very useful way of starting to kind of create this way of starting to kind of create this way of starting to kind of create this environment out of nothing. Well, it's environment out of nothing. Well, it's environment out of nothing. Well, it's not nothing, it's something from the

not nothing, it's something from the not nothing, it's something from the real world. And because you have the real world. And because you have the real world. And because you have the real world, you want to use that, the real world, you want to use that, the real world, you want to use that, the real world, your production environment, real world, your production environment, real world, your production environment, your agent traces, as the source from your agent traces, as the source from your agent traces, as the source from which you want to learn, even if you which you want to learn, even if you which you want to learn, even if you don't have supervision yet. don't have supervision yet. don't have supervision yet. And so one thing you have to do here is And so one thing you have to do here is And so one thing you have to do here is get tasks. And so documents are actually get tasks. And so documents are actually get tasks. And so documents are actually a pretty easy version of this where you a pretty easy version of this where you a pretty easy version of this where you can just sample documents, you can have can just sample documents, you can have can just sample documents, you can have models generate question-answer pairs models generate question-answer pairs models generate question-answer pairs grounded in the documents, you can grounded in the documents, you can grounded in the documents, you can verify that those answer those questions verify that those answer those questions verify that those answer those questions are answerable with other models that are answerable with other models that are answerable with other models that are still grounded in the documents, and are still grounded in the documents, and are still grounded in the documents, and then the actual task at hand involves then the actual task at hand involves then the actual task at hand involves throwing away the initial search. And so throwing away the initial search. And so throwing away the initial search. And so you kind of get to work backwards. And you kind of get to work backwards. And you kind of get to work backwards. And so this general principle of like so this general principle of like so this general principle of like working backwards is starting from the working backwards is starting from the working backwards is starting from the solution or something that's close to solution or something that's close to solution or something that's close to the solution and have where your real the solution and have where your real the solution and have where your real task is like further upstream. This is a task is like further upstream. This is a task is like further upstream. This is a very useful way of kind of having a very useful way of kind of having a very useful way of kind of having a You can verify the easy problem and then You can verify the easy problem and then You can verify the easy problem and then learn on the hard problem. And so learn on the hard problem. And so learn on the hard problem. And so anything where you can move backwards anything where you can move backwards anything where you can move backwards like this is super useful for kind of like this is super useful for kind of like this is super useful for kind of getting supervision for free. getting supervision for free. getting supervision for free. Um in code, you can have uh you can use Um in code, you can have uh you can use Um in code, you can have uh you can use uh real-world PRs, the diffs, the the uh real-world PRs, the diffs, the the uh real-world PRs, the diffs, the the descriptions, the test cases, uh descriptions, the test cases, uh descriptions, the test cases, uh removing different pieces of different removing different pieces of different removing different pieces of different files to be able to have models start files to be able to have models start files to be able to have models start learning over code bases because you can learning over code bases because you can learning over code bases because you can kind of take something that is a kind of take something that is a kind of take something that is a completed artifact and start breaking it completed artifact and start breaking it completed artifact and start breaking it down into smaller pieces and then have down into smaller pieces and then have down into smaller pieces and then have replaying these pieces of getting to an replaying these pieces of getting to an replaying these pieces of getting to an end state that you know is reachable uh end state that you know is reachable uh end state that you know is reachable uh be a task that you train on. And so, be a task that you train on. And so, be a task that you train on. And so, this idea of wanting to know that an end this idea of wanting to know that an end this idea of wanting to know that an end state is reachable and that you can then state is reachable and that you can then state is reachable and that you can then take steps back, throw away the take steps back, throw away the take steps back, throw away the solution, and then learn to find it solution, and then learn to find it solution, and then learn to find it again uh can be applied more generally again uh can be applied more generally again uh can be applied more generally beyond code as well. Uh and so, we talk beyond code as well. Uh and so, we talk beyond code as well. Uh and so, we talk about world simulators broadly as the

about world simulators broadly as the about world simulators broadly as the the sort of thing we might want to do in the sort of thing we might want to do in the sort of thing we might want to do in messier environments which are not just messier environments which are not just messier environments which are not just uh production uh like which are not just uh production uh like which are not just uh production uh like which are not just doc search or code. And so, a lot of the doc search or code. And so, a lot of the doc search or code. And so, a lot of the ones that we've been working on at Prime ones that we've been working on at Prime ones that we've been working on at Prime and Elect are related to things like and Elect are related to things like and Elect are related to things like tool use and web applications where we tool use and web applications where we tool use and web applications where we don't actually have full controllability don't actually have full controllability don't actually have full controllability of the back-end state. There are some of the back-end state. There are some of the back-end state. There are some MCP tools or CLI tools or websites or MCP tools or CLI tools or websites or MCP tools or CLI tools or websites or applications where we can't actually applications where we can't actually applications where we can't actually program them yet. And so, what we want program them yet. And so, what we want program them yet. And so, what we want to do is learn to simulate them. And so, to do is learn to simulate them. And so, to do is learn to simulate them. And so, we found that using combinations of we found that using combinations of we found that using combinations of uh uh uh universal back-end infrastructure and universal back-end infrastructure and universal back-end infrastructure and test time scaling and search and kind of test time scaling and search and kind of test time scaling and search and kind of iterating between the simulator and the iterating between the simulator and the iterating between the simulator and the real real real uh uh uh and the real, we can kind of ground in and the real, we can kind of ground in and the real, we can kind of ground in these production traces. This then these production traces. This then these production traces. This then allows us to create really high-fidelity allows us to create really high-fidelity allows us to create really high-fidelity simulators. And so, we found that these simulators. And so, we found that these simulators. And so, we found that these simulators are actually really great for simulators are actually really great for simulators are actually really great for RL because you can one, if you have RL because you can one, if you have RL because you can one, if you have production data, you can make your production data, you can make your production data, you can make your simulator better and better over time, simulator better and better over time, simulator better and better over time, but also you have full controllability but also you have full controllability but also you have full controllability over the back-end. And so, you can over the back-end. And so, you can over the back-end. And so, you can actually do this reverse engineering actually do this reverse engineering actually do this reverse engineering where you get to kind of plant the where you get to kind of plant the where you get to kind of plant the answer. You can start from the end and answer. You can start from the end and answer. You can start from the end and work backwards uh and so that you you work backwards uh and so that you you work backwards uh and so that you you have this verifiability baked into the have this verifiability baked into the have this verifiability baked into the simulator even if you don't have it in simulator even if you don't have it in simulator even if you don't have it in the real-world production deployment the real-world production deployment the real-world production deployment because you don't know in advance if a because you don't know in advance if a because you don't know in advance if a task was solvable at the time that you task was solvable at the time that you task was solvable at the time that you are being asked it. Um are being asked it. Um are being asked it. Um and in terms of doing this, a very and in terms of doing this, a very and in terms of doing this, a very useful thing is scaling judges. And so, useful thing is scaling judges. And so, useful thing is scaling judges. And so, a lot of times we will have a model that a lot of times we will have a model that a lot of times we will have a model that does something and it will make mistakes does something and it will make mistakes does something and it will make mistakes along the way and it's easier to tell along the way and it's easier to tell along the way and it's easier to tell what went wrong in hindsight. And so, what went wrong in hindsight. And so, what went wrong in hindsight. And so, the fact that you've already seen the the fact that you've already seen the the fact that you've already seen the chain of events after and you can look chain of events after and you can look chain of events after and you can look backwards and say, "Okay, the model made backwards and say, "Okay, the model made backwards and say, "Okay, the model made a mistake here. This thing doesn't feel

a mistake here. This thing doesn't feel a mistake here. This thing doesn't feel quite right." Or we asked seven quite right." Or we asked seven quite right." Or we asked seven different models and they all kind of different models and they all kind of different models and they all kind of agree this thing is wrong. This is a agree this thing is wrong. This is a agree this thing is wrong. This is a very useful way of kind of spending very useful way of kind of spending very useful way of kind of spending compute to do search to then extract compute to do search to then extract compute to do search to then extract rubrics. These rubric questions are are rubrics. These rubric questions are are rubrics. These rubric questions are are are very effective at like kind of are very effective at like kind of are very effective at like kind of distilling down the the search into distilling down the the search into distilling down the the search into something that we can then use to more something that we can then use to more something that we can then use to more cheaply audit and kind of also ground, cheaply audit and kind of also ground, cheaply audit and kind of also ground, once we have these rubrics, as a a way once we have these rubrics, as a a way once we have these rubrics, as a a way of saying, "Okay, we need tasks that of saying, "Okay, we need tasks that of saying, "Okay, we need tasks that target these kinds of failure modes as target these kinds of failure modes as target these kinds of failure modes as well." Um all of this is under the well." Um all of this is under the well." Um all of this is under the umbrella of scaling search with umbrella of scaling search with umbrella of scaling search with test-time compute. And so, we can scale test-time compute. And so, we can scale test-time compute. And so, we can scale search for mining traces by if we have search for mining traces by if we have search for mining traces by if we have offline production traces, we can just offline production traces, we can just offline production traces, we can just look at them more and think about them look at them more and think about them look at them more and think about them more and have more models play with more and have more models play with more and have more models play with them. We can calibrate difficulty. So, them. We can calibrate difficulty. So, them. We can calibrate difficulty. So, RL to have the advantage gap that we RL to have the advantage gap that we RL to have the advantage gap that we mentioned needs to have a a separation mentioned needs to have a a separation mentioned needs to have a a separation between what one model will do once and between what one model will do once and between what one model will do once and what a model what a collection of what a model what a collection of what a model what a collection of rollouts will do. And so, you want tasks rollouts will do. And so, you want tasks rollouts will do. And so, you want tasks that are not too easy, not too hard and that are not too easy, not too hard and that are not too easy, not too hard and you want to be searching for these and you want to be searching for these and you want to be searching for these and iterating on generating more of them. iterating on generating more of them. iterating on generating more of them. And so, this is another area where you And so, this is another area where you And so, this is another area where you can spend compute to um refine the can spend compute to um refine the can spend compute to um refine the difficulty of your your task difficulty of your your task difficulty of your your task distributions, your task sets. Um for distributions, your task sets. Um for distributions, your task sets. Um for simulators, when you're building web simulators, when you're building web simulators, when you're building web applications or tools that need to applications or tools that need to applications or tools that need to simulate complex behavior, you can spend simulate complex behavior, you can spend simulate complex behavior, you can spend compute on searching these and then you compute on searching these and then you compute on searching these and then you can have agents refine the can have agents refine the can have agents refine the implementations of them and this allows implementations of them and this allows implementations of them and this allows for increasingly high-fidelity for increasingly high-fidelity for increasingly high-fidelity environments. You can do this on environments. You can do this on environments. You can do this on verification both at train time as well verification both at train time as well verification both at train time as well as as as offline when you're kind of creating offline when you're kind of creating offline when you're kind of creating these rubrics. You can do things like these rubrics. You can do things like these rubrics. You can do things like red teaming with adversarial prompt red teaming with adversarial prompt red teaming with adversarial prompt optimization to to kind of explore for optimization to to kind of explore for optimization to to kind of explore for backdoors. Then you can look for traces

backdoors. Then you can look for traces backdoors. Then you can look for traces and spend compute mining these traces and spend compute mining these traces and spend compute mining these traces for understanding was this the reward a for understanding was this the reward a for understanding was this the reward a reward hacker? Was this actually kind of reward hacker? Was this actually kind of reward hacker? Was this actually kind of in the spirit of the task? And I think in the spirit of the task? And I think in the spirit of the task? And I think these things can kind of feel like these things can kind of feel like these things can kind of feel like reward hacking can kind of sneak up on reward hacking can kind of sneak up on reward hacking can kind of sneak up on you if you're not careful for it, but in you if you're not careful for it, but in you if you're not careful for it, but in many cases many cases many cases the basic simple things actually work the basic simple things actually work the basic simple things actually work quite well where if the reward hacks are quite well where if the reward hacks are quite well where if the reward hacks are the sorts of things where a human can the sorts of things where a human can the sorts of things where a human can look at them and be like, oh yeah, look at them and be like, oh yeah, look at them and be like, oh yeah, that's a reward hack. that's a reward hack. that's a reward hack. Judges are often not quite good at doing Judges are often not quite good at doing Judges are often not quite good at doing this as well. They just don't this as well. They just don't this as well. They just don't necessarily necessarily necessarily tell if you tell the model not to do tell if you tell the model not to do tell if you tell the model not to do this, it won't necessarily do it this, it won't necessarily do it this, it won't necessarily do it in the rollout, but in hindsight you can in the rollout, but in hindsight you can in the rollout, but in hindsight you can reflect on this and spend compute to reflect on this and spend compute to reflect on this and spend compute to kind of especially if you are collecting kind of especially if you are collecting kind of especially if you are collecting these over time and you are building up these over time and you are building up these over time and you are building up your corpus of examples of reward hacks, your corpus of examples of reward hacks, your corpus of examples of reward hacks, you can understand the sorts of things you can understand the sorts of things you can understand the sorts of things that go wrong that go wrong that go wrong and address this by kind of again and address this by kind of again and address this by kind of again spending inference compute on refining spending inference compute on refining spending inference compute on refining your implementation, refining your your implementation, refining your your implementation, refining your rewards. rewards. rewards. As well as validating these by training. As well as validating these by training. As well as validating these by training. And so we find that in many cases you And so we find that in many cases you And so we find that in many cases you can do a lot up front, but also also can do a lot up front, but also also can do a lot up front, but also also there are things that don't show up there are things that don't show up there are things that don't show up until you actually like start doing RL. until you actually like start doing RL. until you actually like start doing RL. And so part of this is folding in And so part of this is folding in And so part of this is folding in training experiments themselves into the training experiments themselves into the training experiments themselves into the process of environment design where you process of environment design where you process of environment design where you can you can do small runs with can you can do small runs with can you can do small runs with individual models on like one individual models on like one individual models on like one environment and and see what happens and environment and and see what happens and environment and and see what happens and you can understand that the behavior you can understand that the behavior you can understand that the behavior changes. You can have metrics that log changes. You can have metrics that log changes. You can have metrics that log the the types of tool calls that are the the types of tool calls that are the the types of tool calls that are being done that are like judges asking being done that are like judges asking being done that are like judges asking questions about the the traces to questions about the the traces to questions about the the traces to understand how behavioral patterns are understand how behavioral patterns are understand how behavioral patterns are changing. And all of these are very changing. And all of these are very changing. And all of these are very useful ways of kind of getting something useful ways of kind of getting something useful ways of kind of getting something from nothing and using compute as the from nothing and using compute as the from nothing and using compute as the thing that allows you to refine your thing that allows you to refine your thing that allows you to refine your understanding. And ultimately what you understanding. And ultimately what you understanding. And ultimately what you want is to surface the most important want is to surface the most important want is to surface the most important pieces up to the human, the the highest pieces up to the human, the the highest pieces up to the human, the the highest level of

level of level of questions about what is actually going questions about what is actually going questions about what is actually going on, what is the goal so that the all on, what is the goal so that the all on, what is the goal so that the all this is deferring to the human for the this is deferring to the human for the this is deferring to the human for the most important pieces of like actually most important pieces of like actually most important pieces of like actually employing expert to say this is good, employing expert to say this is good, employing expert to say this is good, this is bad, this is what I want, this this is bad, this is what I want, this this is bad, this is what I want, this is not what I want. And uh is not what I want. And uh is not what I want. And uh these are all the ways that we kind of these are all the ways that we kind of these are all the ways that we kind of gain confidence in the environments. gain confidence in the environments. gain confidence in the environments. I know we're running a little short on I know we're running a little short on I know we're running a little short on time, so I wanted to break up a couple time, so I wanted to break up a couple time, so I wanted to break up a couple blogs that we've put out recently that blogs that we've put out recently that blogs that we've put out recently that are kind of demonstrating pieces of are kind of demonstrating pieces of are kind of demonstrating pieces of this. We have a blog called general this. We have a blog called general this. We have a blog called general agent agent agent which is demonstrating this for tool which is demonstrating this for tool which is demonstrating this for tool use, this online loop of generating, use, this online loop of generating, use, this online loop of generating, solving, and synthesizing new tasks and solving, and synthesizing new tasks and solving, and synthesizing new tasks and gating based on this pass rate which gating based on this pass rate which gating based on this pass rate which then we train on and we see a great then we train on and we see a great then we train on and we see a great uplift on popular benchmarks for tool uplift on popular benchmarks for tool uplift on popular benchmarks for tool use. use. use. Um additionally, beyond just RL, we Um additionally, beyond just RL, we Um additionally, beyond just RL, we found that it's quite important to think found that it's quite important to think found that it's quite important to think about cases where there's information in about cases where there's information in about cases where there's information in the world that RL alone will not the world that RL alone will not the world that RL alone will not explore. And so, there's this great work explore. And so, there's this great work explore. And so, there's this great work Echo from some researchers that we are Echo from some researchers that we are Echo from some researchers that we are friends with and have been collaborating friends with and have been collaborating friends with and have been collaborating with. And then we did our own kind of with. And then we did our own kind of with. And then we did our own kind of deep dive into this as well to look at deep dive into this as well to look at deep dive into this as well to look at what happens when you have an agent that what happens when you have an agent that what happens when you have an agent that is not just training with reinforcement is not just training with reinforcement is not just training with reinforcement learning, but is also getting supervised learning, but is also getting supervised learning, but is also getting supervised learning signal from the environment learning signal from the environment learning signal from the environment itself. Which then allows the model to itself. Which then allows the model to itself. Which then allows the model to understand things like having a native understand things like having a native understand things like having a native world model of the environment, world model of the environment, world model of the environment, understanding what to expect because it understanding what to expect because it understanding what to expect because it has a likelihood model of the tokens has a likelihood model of the tokens has a likelihood model of the tokens that the environment itself will that the environment itself will that the environment itself will generate. And these are the sorts of generate. And these are the sorts of generate. And these are the sorts of things that in many cases allow the things that in many cases allow the things that in many cases allow the model itself to kind of more adaptively model itself to kind of more adaptively model itself to kind of more adaptively navigate the world and not just like navigate the world and not just like navigate the world and not just like refine its skill, but like get new refine its skill, but like get new refine its skill, but like get new information into its weights over time information into its weights over time information into its weights over time as well.

as well. as well. Um Um Um and so all of this is in spirit of and so all of this is in spirit of and so all of this is in spirit of making post-training easier, making making post-training easier, making making post-training easier, making continual learning easier, giving people continual learning easier, giving people continual learning easier, giving people the ability to create agents and not the ability to create agents and not the ability to create agents and not worry too much about having to to fuss worry too much about having to to fuss worry too much about having to to fuss with the research pieces and fine-tune with the research pieces and fine-tune with the research pieces and fine-tune all the the small details. Today we all the the small details. Today we all the the small details. Today we still do, but we're kind of seeing paths still do, but we're kind of seeing paths still do, but we're kind of seeing paths forward of how we start automating this forward of how we start automating this forward of how we start automating this more and more by like having more and more by like having more and more by like having environments as the anchor, which we can environments as the anchor, which we can environments as the anchor, which we can then spend compute on refining. We can then spend compute on refining. We can then spend compute on refining. We can kind of use compute to mine the data we kind of use compute to mine the data we kind of use compute to mine the data we have from the real world to refine the have from the real world to refine the have from the real world to refine the the signals where the humans are kind of the signals where the humans are kind of the signals where the humans are kind of just just just in the same way that with coding agents in the same way that with coding agents in the same way that with coding agents we're kind of going to higher levels of we're kind of going to higher levels of we're kind of going to higher levels of abstraction, we can do this with abstraction, we can do this with abstraction, we can do this with environment and reward design as well. environment and reward design as well. environment and reward design as well. Um and all of this is well allows us to Um and all of this is well allows us to Um and all of this is well allows us to ultimately close the loop where models ultimately close the loop where models ultimately close the loop where models are then able to are then able to are then able to stay within the guardrails we give them, stay within the guardrails we give them, stay within the guardrails we give them, they go find the issues in production, they go find the issues in production, they go find the issues in production, and then they turn these back into new and then they turn these back into new and then they turn these back into new tasks that can then be trained on for tasks that can then be trained on for tasks that can then be trained on for getting better in the real world. Um we getting better in the real world. Um we getting better in the real world. Um we work hands-on with startup enterprises work hands-on with startup enterprises work hands-on with startup enterprises to help them train their models. We are to help them train their models. We are to help them train their models. We are also hiring quite a lot. also hiring quite a lot. also hiring quite a lot. If you want to get in touch for either If you want to get in touch for either If you want to get in touch for either of these, find me after the talk. Thanks of these, find me after the talk. Thanks of these, find me after the talk. Thanks a bunch.

a bunch. a bunch. >> [applause]