Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

Yeah, thank you, Jack. Um, really Yeah, thank you, Jack. Um, really grateful for the opportunity to speak grateful for the opportunity to speak grateful for the opportunity to speak here. here. here. Today, I'm going to be sharing Today, I'm going to be sharing Today, I'm going to be sharing some of our frontier work on post some of our frontier work on post some of our frontier work on post training and how we envision a future training and how we envision a future training and how we envision a future where agents can learn new skills on the where agents can learn new skills on the where agents can learn new skills on the job. job. job. So, over the last year or so, we've seen So, over the last year or so, we've seen So, over the last year or so, we've seen agents develop really strong reasoning agents develop really strong reasoning agents develop really strong reasoning skills. skills. skills. And they've learned to use, uh, agentic And they've learned to use, uh, agentic And they've learned to use, uh, agentic harnesses to solve longer and longer harnesses to solve longer and longer harnesses to solve longer and longer horizon tasks, which involve many turns horizon tasks, which involve many turns horizon tasks, which involve many turns and tool calls on and tool calls on and tool calls on complicated environment states. complicated environment states. complicated environment states. We're seeing an increasing, uh, demand We're seeing an increasing, uh, demand We're seeing an increasing, uh, demand for for for agents that can just be deployed in a agents that can just be deployed in a agents that can just be deployed in a plug-and-play way into plug-and-play way into plug-and-play way into how enterprises use the agents, um, how enterprises use the agents, um, how enterprises use the agents, um, so, for instance, if they already have so, for instance, if they already have so, for instance, if they already have some method of some method of some method of calling the agent to do a task, to do a calling the agent to do a task, to do a calling the agent to do a task, to do a task, task, task, they would want to be able to train a they would want to be able to train a they would want to be able to train a custom model to do that task instead. custom model to do that task instead. custom model to do that task instead. And And And that requires new ways of looking at that requires new ways of looking at that requires new ways of looking at post training that allow you to, um, post training that allow you to, um, post training that allow you to, um, adapt to any harness, including ones adapt to any harness, including ones adapt to any harness, including ones that you don't necessarily have access that you don't necessarily have access that you don't necessarily have access to the source code of.

So, I wanted to talk about a few So, I wanted to talk about a few different levels of post training, where different levels of post training, where different levels of post training, where each one builds on top of the last. each one builds on top of the last. each one builds on top of the last. One way that we kind of think of this, One way that we kind of think of this, One way that we kind of think of this, uh, is a framework comparing it to how uh, is a framework comparing it to how uh, is a framework comparing it to how humans do learning, where humans do learning, where humans do learning, where you learn simple tasks first, and you you learn simple tasks first, and you you learn simple tasks first, and you can sort of compound your understanding can sort of compound your understanding can sort of compound your understanding to more and more complicated tasks. to more and more complicated tasks. to more and more complicated tasks. So, So, So, over the last year, we've sort of, over the last year, we've sort of, over the last year, we've sort of, I would say, mastered or gotten a lot of I would say, mastered or gotten a lot of I would say, mastered or gotten a lot of reps with these simple single-turn Q&A reps with these simple single-turn Q&A reps with these simple single-turn Q&A tasks and um some longer horizon tasks and um some longer horizon tasks and um some longer horizon synthetic environment tasks. synthetic environment tasks. synthetic environment tasks. But, what we're increasingly seeing is But, what we're increasingly seeing is But, what we're increasingly seeing is we want to be able to adapt to custom we want to be able to adapt to custom we want to be able to adapt to custom harnesses and harnesses and harnesses and be able to be able to be able to train directly on those instead. And we train directly on those instead. And we train directly on those instead. And we kind of think of those kind of like kind of think of those kind of like kind of think of those kind of like internships where internships where internships where you want to the model to do a specific you want to the model to do a specific you want to the model to do a specific task, but you don't necessarily know how task, but you don't necessarily know how task, but you don't necessarily know how exactly the task will play out because exactly the task will play out because exactly the task will play out because you don't own the harness. you don't own the harness. you don't own the harness. And then finally, I would want to share And then finally, I would want to share And then finally, I would want to share some visions we have for the future some visions we have for the future some visions we have for the future of the custom model training space, of the custom model training space, of the custom model training space, where we think that there will be these where we think that there will be these where we think that there will be these kind of agentic citizens, which you can kind of agentic citizens, which you can kind of agentic citizens, which you can just deploy once, and they'll be able to just deploy once, and they'll be able to just deploy once, and they'll be able to adapt to many different types of out of adapt to many different types of out of adapt to many different types of out of distribution tasks and learn from their distribution tasks and learn from their distribution tasks and learn from their interactions.

So first, I just want to talk about the So first, I just want to talk about the training setup for these simple Q&A training setup for these simple Q&A training setup for these simple Q&A tasks. We have something that looks like this, We have something that looks like this, where you have an orchestrator, and the where you have an orchestrator, and the where you have an orchestrator, and the orchestrator is in charge of driving the orchestrator is in charge of driving the orchestrator is in charge of driving the rollouts. rollouts. rollouts. The orchestrator holds a task back, The orchestrator holds a task back, The orchestrator holds a task back, which you can think of for now is just a which you can think of for now is just a which you can think of for now is just a simple prompt and answer. So, something simple prompt and answer. So, something simple prompt and answer. So, something like a math question and a corresponding like a math question and a corresponding like a math question and a corresponding numerical answer. numerical answer. numerical answer. Um the orchestrator will send this Um the orchestrator will send this Um the orchestrator will send this prompt to a model and then get an answer prompt to a model and then get an answer prompt to a model and then get an answer back. back. back. Then it will send the answer to a grader Then it will send the answer to a grader Then it will send the answer to a grader and have it be graded. So, once all of and have it be graded. So, once all of and have it be graded. So, once all of this is done, we want to improve our this is done, we want to improve our this is done, we want to improve our model based on that interaction or maybe model based on that interaction or maybe model based on that interaction or maybe like a batch of interactions. And the like a batch of interactions. And the like a batch of interactions. And the way we do that is through a training way we do that is through a training way we do that is through a training engine, engine, engine, which takes in the graded chats and which takes in the graded chats and which takes in the graded chats and produces a weight update. produces a weight update. produces a weight update. That weight update is then synced to That weight update is then synced to That weight update is then synced to some inference engines. And once those some inference engines. And once those some inference engines. And once those inference engines are updated, then we inference engines are updated, then we inference engines are updated, then we can start this entire process over can start this entire process over can start this entire process over again, where the orchestrator will have again, where the orchestrator will have again, where the orchestrator will have like new problems to like new problems to like new problems to um um um send to the model completion endpoint, send to the model completion endpoint, send to the model completion endpoint, and then you'll be able to get more and then you'll be able to get more and then you'll be able to get more chats, grade them, and train again.

chats, grade them, and train again. chats, grade them, and train again. The key thing to note here is that the The key thing to note here is that the The key thing to note here is that the only thing you need for improving your only thing you need for improving your only thing you need for improving your model is the graded chats in some model is the graded chats in some model is the graded chats in some format, and once you have those, the format, and once you have those, the format, and once you have those, the training engine can compute weight training engine can compute weight training engine can compute weight updates to improve your model. updates to improve your model. updates to improve your model. Uh what's important here is that the Uh what's important here is that the Uh what's important here is that the chats are in a very specific format chats are in a very specific format chats are in a very specific format because we're sort of constraining because we're sort of constraining because we're sort of constraining everything to be inside of our training everything to be inside of our training everything to be inside of our training stack. So, in this simple setup for Q&A, stack. So, in this simple setup for Q&A, stack. So, in this simple setup for Q&A, you don't have anything living outside you don't have anything living outside you don't have anything living outside of the training stack. You You basically of the training stack. You You basically of the training stack. You You basically have the code of how to run the rollout have the code of how to run the rollout have the code of how to run the rollout and how everything is formatted, so it's and how everything is formatted, so it's and how everything is formatted, so it's like in a very controlled environment. like in a very controlled environment. like in a very controlled environment. However, However, However, this is kind of limited because this is kind of limited because this is kind of limited because we can only kind of do single-turn tasks we can only kind of do single-turn tasks we can only kind of do single-turn tasks in this way. If we want to do longer and in this way. If we want to do longer and in this way. If we want to do longer and long-horizon tasks, and we want to build long-horizon tasks, and we want to build long-horizon tasks, and we want to build like higher-order skills into our like higher-order skills into our like higher-order skills into our models, models, models, we need to also increase the complexity we need to also increase the complexity we need to also increase the complexity of our environment. So, of our environment. So, of our environment. So, with synthetic environments, we have a with synthetic environments, we have a with synthetic environments, we have a very similar setup, but we offload a lot very similar setup, but we offload a lot very similar setup, but we offload a lot of the environment state um outside of of the environment state um outside of of the environment state um outside of the training stack. So, you still have the training stack. So, you still have the training stack. So, you still have the same orchestrator um from before, the same orchestrator um from before, the same orchestrator um from before, but but but um um um the task back is maybe a little bit more the task back is maybe a little bit more the task back is maybe a little bit more complicated, and the environment state complicated, and the environment state complicated, and the environment state is uh living outside of the training is uh living outside of the training is uh living outside of the training stack. So, the task back might now stack. So, the task back might now stack. So, the task back might now include things like tool cost specs or include things like tool cost specs or include things like tool cost specs or like maybe an initial state for your like maybe an initial state for your like maybe an initial state for your environment, like a file system.

environment, like a file system. environment, like a file system. And the orchestrator is now in charge of And the orchestrator is now in charge of And the orchestrator is now in charge of running many turns in a in series, where running many turns in a in series, where running many turns in a in series, where maybe first they ask the model for how maybe first they ask the model for how maybe first they ask the model for how it wants to respond, and then if the it wants to respond, and then if the it wants to respond, and then if the model wants to call some tools, it'll model wants to call some tools, it'll model wants to call some tools, it'll then call the sandbox to actually like then call the sandbox to actually like then call the sandbox to actually like modify the environment state or read the modify the environment state or read the modify the environment state or read the environment state, and then return those environment state, and then return those environment state, and then return those results back to the model. results back to the model. results back to the model. After all that is said and done, you get After all that is said and done, you get After all that is said and done, you get a full task trace out of this, um, and a full task trace out of this, um, and a full task trace out of this, um, and that task trace is then sent to a grader that task trace is then sent to a grader that task trace is then sent to a grader for grading. for grading. for grading. And very similar to what we had before, And very similar to what we had before, And very similar to what we had before, you'll be able to take the graded chats, you'll be able to take the graded chats, you'll be able to take the graded chats, you'll be able to then use them to do a you'll be able to then use them to do a you'll be able to then use them to do a weight update. weight update. weight update. The main thing to highlight here is that The main thing to highlight here is that The main thing to highlight here is that this orchestrator and sandbox setup is this orchestrator and sandbox setup is this orchestrator and sandbox setup is replayable, which is replayable, which is replayable, which is basically just saying that for any basically just saying that for any basically just saying that for any specific prompt, you can always like specific prompt, you can always like specific prompt, you can always like roll back to the initial state and like roll back to the initial state and like roll back to the initial state and like rerun it. Um, you can do that in rerun it. Um, you can do that in rerun it. Um, you can do that in parallel or you can do that in series. parallel or you can do that in series. parallel or you can do that in series. But, the reason that's important is But, the reason that's important is But, the reason that's important is because because because the the the main sort of method that we use for main sort of method that we use for main sort of method that we use for reinforcement learning today is GRPO, reinforcement learning today is GRPO, reinforcement learning today is GRPO, and that involves comparing many and that involves comparing many and that involves comparing many rollouts for the same prompt, and then rollouts for the same prompt, and then rollouts for the same prompt, and then comparing like relatively which one is comparing like relatively which one is comparing like relatively which one is better than the other. And the training better than the other. And the training better than the other. And the training engine will then up like make an edit to engine will then up like make an edit to engine will then up like make an edit to the model to upweight the trajectories the model to upweight the trajectories the model to upweight the trajectories that were more successful, and then that were more successful, and then that were more successful, and then downweight the ones that were less downweight the ones that were less downweight the ones that were less successful.

So, some challenges that we face in this So, some challenges that we face in this setup is that the environment is setup is that the environment is setup is that the environment is something that you want to basically, something that you want to basically, something that you want to basically, uh, use to replicate reality, so that uh, use to replicate reality, so that uh, use to replicate reality, so that after you're done training, like the after you're done training, like the after you're done training, like the improvements that you've seen actually improvements that you've seen actually improvements that you've seen actually translate to when you deploy these translate to when you deploy these translate to when you deploy these models into production. And models into production. And models into production. And the main sort of problem is the main sort of problem is the main sort of problem is like has kind of two names, which are like has kind of two names, which are like has kind of two names, which are both the same problem, environment both the same problem, environment both the same problem, environment fidelity and reward hacking. fidelity and reward hacking. fidelity and reward hacking. Essentially, the agent is exposed to an Essentially, the agent is exposed to an Essentially, the agent is exposed to an environment and sort of any, uh, environment and sort of any, uh, environment and sort of any, uh, any quirks of your environment will end any quirks of your environment will end any quirks of your environment will end up being something that your agent may up being something that your agent may up being something that your agent may like learn a model of. So, we have some like learn a model of. So, we have some like learn a model of. So, we have some examples that we've seen where in a examples that we've seen where in a examples that we've seen where in a trading run in the past, we had some trading run in the past, we had some trading run in the past, we had some like networking issues causing our like networking issues causing our like networking issues causing our environment to have tool calls that environment to have tool calls that environment to have tool calls that failed maybe around 10% of the time. failed maybe around 10% of the time. failed maybe around 10% of the time. If that is the case, then If that is the case, then If that is the case, then we actually saw that the model would we actually saw that the model would we actually saw that the model would then start outputting shorter and then start outputting shorter and then start outputting shorter and shorter responses. Now, this was really shorter responses. Now, this was really shorter responses. Now, this was really surprising to us because in our reward surprising to us because in our reward surprising to us because in our reward function, we actually didn't have any function, we actually didn't have any function, we actually didn't have any length penalty. So, length penalty. So, length penalty. So, like we we couldn't really tell why this like we we couldn't really tell why this like we we couldn't really tell why this was happening, but really what's going was happening, but really what's going was happening, but really what's going on here is if you think about maybe the on here is if you think about maybe the on here is if you think about maybe the model is like a human like walking along model is like a human like walking along model is like a human like walking along a sidewalk and like the tool call a sidewalk and like the tool call a sidewalk and like the tool call failures are like potholes in the failures are like potholes in the failures are like potholes in the sidewalk, sidewalk, sidewalk, like it makes a lot of sense that like it makes a lot of sense that like it makes a lot of sense that because there's so many potholes, the because there's so many potholes, the because there's so many potholes, the model doesn't want to run for that long model doesn't want to run for that long model doesn't want to run for that long uh because it might fall in a pothole uh because it might fall in a pothole uh because it might fall in a pothole and then get a zero reward for the and then get a zero reward for the and then get a zero reward for the rollout.

rollout. rollout. Um and then conversely, Um and then conversely, Um and then conversely, um it's also possible that your model um it's also possible that your model um it's also possible that your model just learns to like output more and more just learns to like output more and more just learns to like output more and more gibberish over time depending on like gibberish over time depending on like gibberish over time depending on like what your environment looks like. So, what your environment looks like. So, what your environment looks like. So, in a different case, we had a training in a different case, we had a training in a different case, we had a training run where we um have sandbox timeouts uh run where we um have sandbox timeouts uh run where we um have sandbox timeouts uh just so that they don't run forever, and just so that they don't run forever, and just so that they don't run forever, and we usually like filter out the rollouts we usually like filter out the rollouts we usually like filter out the rollouts that timed out from being trained on. that timed out from being trained on. that timed out from being trained on. One thing we saw was that if your tool One thing we saw was that if your tool One thing we saw was that if your tool calls take a long time, then if the calls take a long time, then if the calls take a long time, then if the model feels like the problem is really model feels like the problem is really model feels like the problem is really hard, it will actually just be hard, it will actually just be hard, it will actually just be incentivized to like abuse the tool incentivized to like abuse the tool incentivized to like abuse the tool calls and just like call a lot of them calls and just like call a lot of them calls and just like call a lot of them in quick succession and try to time out in quick succession and try to time out in quick succession and try to time out the sandbox so it avoids getting a the sandbox so it avoids getting a the sandbox so it avoids getting a reward of zero. It just gets the rollout reward of zero. It just gets the rollout reward of zero. It just gets the rollout dropped. dropped. dropped. So, as we scale to like more and more So, as we scale to like more and more So, as we scale to like more and more complicated tasks, the complicated tasks, the complicated tasks, the the task of like replicating these the task of like replicating these the task of like replicating these environments becomes increasingly environments becomes increasingly environments becomes increasingly difficult because difficult because difficult because it's very, very difficult to like it's very, very difficult to like it's very, very difficult to like perfectly simulate reality and sort of perfectly simulate reality and sort of perfectly simulate reality and sort of any mistake that you make, even if it's any mistake that you make, even if it's any mistake that you make, even if it's not intentional, will end up inducing not intentional, will end up inducing not intentional, will end up inducing these like subtle undesirable behaviors these like subtle undesirable behaviors these like subtle undesirable behaviors in your model.

in your model. in your model. So, that brings us to our next topic of So, that brings us to our next topic of So, that brings us to our next topic of bring your own harness, where we're bring your own harness, where we're bring your own harness, where we're basically asking like if the agent basically asking like if the agent basically asking like if the agent learns the exact environment learns the exact environment learns the exact environment distribution, distribution, distribution, why don't we just use that for our why don't we just use that for our why don't we just use that for our training? Like just directly the real training? Like just directly the real training? Like just directly the real environment, you will no longer need to environment, you will no longer need to environment, you will no longer need to uh replicate anything. You can just like uh replicate anything. You can just like uh replicate anything. You can just like use exactly how it's going to be used in use exactly how it's going to be used in use exactly how it's going to be used in production. production. production. This solves a lot of problems and This solves a lot of problems and This solves a lot of problems and sounds really good. sounds really good. sounds really good. The architecture looks something like The architecture looks something like The architecture looks something like this where this where this where we now have almost everything outside of we now have almost everything outside of we now have almost everything outside of our training stack. The only thing we our training stack. The only thing we our training stack. The only thing we have left is the model completion have left is the model completion have left is the model completion endpoint and some way to endpoint and some way to endpoint and some way to uh record the requests and responses uh record the requests and responses uh record the requests and responses that go in and out of the model. that go in and out of the model. that go in and out of the model. Everything else kind of lives outside of Everything else kind of lives outside of Everything else kind of lives outside of the training stack and can be run in the training stack and can be run in the training stack and can be run in whatever fashion uh that like whatever fashion uh that like whatever fashion uh that like an existing enterprise or like customer an existing enterprise or like customer an existing enterprise or like customer might be using. So, these would be like might be using. So, these would be like might be using. So, these would be like existing enterprise harnesses and essentially essentially the reason this is nice is because we the reason this is nice is because we the reason this is nice is because we can meet customers where they're at.

can meet customers where they're at. can meet customers where they're at. Like, if they're already using the model Like, if they're already using the model Like, if they're already using the model in a certain way, we can just take our in a certain way, we can just take our in a certain way, we can just take our like training methodology and just like like training methodology and just like like training methodology and just like plug it right in and then we can help plug it right in and then we can help plug it right in and then we can help them improve the model for like them improve the model for like them improve the model for like exactly the way that they're using it. exactly the way that they're using it. exactly the way that they're using it. So, all the orchestration loops and So, all the orchestration loops and So, all the orchestration loops and logic will now live outside of the logic will now live outside of the logic will now live outside of the training stack. training stack. training stack. Now, this sounds really good, but Now, this sounds really good, but Now, this sounds really good, but um This sounds really good, but the um This sounds really good, but the um This sounds really good, but the challenge here is challenge here is challenge here is in The challenge here The challenge here is in the like data where is in the like data where is in the like data where I as you're deploying this into uh I as you're deploying this into uh I as you're deploying this into uh production and you have less and less production and you have less and less production and you have less and less control over how the rollouts uh play control over how the rollouts uh play control over how the rollouts uh play out, you also have like less signal to out, you also have like less signal to out, you also have like less signal to learn from because learn from because learn from because the data is not in like a familiar the data is not in like a familiar the data is not in like a familiar format. format. format. And this topic is touched on in a And this topic is touched on in a And this topic is touched on in a related work by Nvidia. Uh this is like related work by Nvidia. Uh this is like related work by Nvidia. Uh this is like a paper from around a month ago where a paper from around a month ago where a paper from around a month ago where they introduce they introduce they introduce uh Polar, which is essentially uh Polar, which is essentially uh Polar, which is essentially a way to think about transitioning from a way to think about transitioning from a way to think about transitioning from a harness where you are kind of in a harness where you are kind of in a harness where you are kind of in charge of micro managing every aspect of charge of micro managing every aspect of charge of micro managing every aspect of the rollouts, kind of like what we were the rollouts, kind of like what we were the rollouts, kind of like what we were previously talking about. And previously talking about. And previously talking about. And transitioning to transitioning to transitioning to some method of just uh listening in on a some method of just uh listening in on a some method of just uh listening in on a black box harness, and you would no black box harness, and you would no black box harness, and you would no longer know exactly what the logic in longer know exactly what the logic in longer know exactly what the logic in here is.

So, some challenges is that So, some challenges is that some challenges we face in this setting some challenges we face in this setting some challenges we face in this setting are non-re-playability and offline or are non-re-playability and offline or are non-re-playability and offline or off-policy data. I think both of these off-policy data. I think both of these off-policy data. I think both of these are are are describing the same issue, which is just describing the same issue, which is just describing the same issue, which is just that because we've moved so much of the that because we've moved so much of the that because we've moved so much of the logic outside of our training stock, we logic outside of our training stock, we logic outside of our training stock, we just don't have any way of like just don't have any way of like just don't have any way of like enforcing invariants or like data enforcing invariants or like data enforcing invariants or like data structures that we like. We have to be structures that we like. We have to be structures that we like. We have to be more flexible about the way we do more flexible about the way we do more flexible about the way we do training, and because of that, it just training, and because of that, it just training, and because of that, it just becomes harder to train your model and becomes harder to train your model and becomes harder to train your model and it may create an update. So, an example it may create an update. So, an example it may create an update. So, an example would be for GRPO, which is like the would be for GRPO, which is like the would be for GRPO, which is like the traditional method, you would want to traditional method, you would want to traditional method, you would want to have many rollouts in parallel for your have many rollouts in parallel for your have many rollouts in parallel for your task, and that may not be possible task, and that may not be possible task, and that may not be possible anymore. If you think about um anymore. If you think about um anymore. If you think about um suppose like a customer chat customer suppose like a customer chat customer suppose like a customer chat customer support chat, and support chat, and support chat, and you have a record of how one of your you have a record of how one of your you have a record of how one of your chats went, there's not really a way chats went, there's not really a way chats went, there's not really a way that you could then go back and think, that you could then go back and think, that you could then go back and think, "Oh, if I like said or if I responded in "Oh, if I like said or if I responded in "Oh, if I like said or if I responded in this other way, like would the user have this other way, like would the user have this other way, like would the user have been happier?" been happier?" been happier?" Like there's no way to then get the Like there's no way to then get the Like there's no way to then get the user's response again.

user's response again. user's response again. But, we're optimistic because like we But, we're optimistic because like we But, we're optimistic because like we think that humans can do this kind of think that humans can do this kind of think that humans can do this kind of learning, and so it should be possible learning, and so it should be possible learning, and so it should be possible to like formulate some kind of method to like formulate some kind of method to like formulate some kind of method that would work for models as well. that would work for models as well. that would work for models as well. Like, if a human was in a customer Like, if a human was in a customer Like, if a human was in a customer support chat, they could understand support chat, they could understand support chat, they could understand somehow that like based on the somehow that like based on the somehow that like based on the customer's reaction, like what they said customer's reaction, like what they said customer's reaction, like what they said was wrong or what they said was good, was wrong or what they said was good, was wrong or what they said was good, and then be able to like internalize and then be able to like internalize and then be able to like internalize improvements uh for for the future. improvements uh for for the future. improvements uh for for the future. So, I want to talk about some of the So, I want to talk about some of the So, I want to talk about some of the frontier research directions we have um frontier research directions we have um frontier research directions we have um towards like solving this problem. towards like solving this problem. towards like solving this problem. There's kind of three main topics, which There's kind of three main topics, which There's kind of three main topics, which are self-distillation, automated data are self-distillation, automated data are self-distillation, automated data pipelines, and qualitative feedback pipelines, and qualitative feedback pipelines, and qualitative feedback ingestion. Self-distillation is a pretty ingestion. Self-distillation is a pretty ingestion. Self-distillation is a pretty new technique, which is still new technique, which is still new technique, which is still um, um, um, I would say relatively like narrowly I would say relatively like narrowly I would say relatively like narrowly scoped. So, we've seen successes in scoped. So, we've seen successes in scoped. So, we've seen successes in inducing like specific new behaviors inducing like specific new behaviors inducing like specific new behaviors with models, but it's with models, but it's with models, but it's it's still an open uh, research question it's still an open uh, research question it's still an open uh, research question of like how general can we push it? of like how general can we push it? of like how general can we push it? Automated data pipelines is an idea Automated data pipelines is an idea Automated data pipelines is an idea which maybe if you take like a big batch which maybe if you take like a big batch which maybe if you take like a big batch of traces, of traces, of traces, um, you would be able to like um, you would be able to like um, you would be able to like automatically like flag undesirable automatically like flag undesirable automatically like flag undesirable behaviors or failure modes, and then be behaviors or failure modes, and then be behaviors or failure modes, and then be able to like put together a like nice able to like put together a like nice able to like put together a like nice batch of training data, um, batch of training data, um, batch of training data, um, automatically, and then send that to the automatically, and then send that to the automatically, and then send that to the model and help it improve.

model and help it improve. model and help it improve. Currently, this is like pretty manual or Currently, this is like pretty manual or Currently, this is like pretty manual or like human in the loop, where like we go like human in the loop, where like we go like human in the loop, where like we go through traces ourselves, and we're like through traces ourselves, and we're like through traces ourselves, and we're like looking for looking for these failure looking for looking for these failure looking for looking for these failure modes manually, and then like describing modes manually, and then like describing modes manually, and then like describing how we can improve the model, and then how we can improve the model, and then how we can improve the model, and then um, looking curating those data sets um, looking curating those data sets um, looking curating those data sets ourselves. And then finally, I ourselves. And then finally, I ourselves. And then finally, I I think an interesting direction is I think an interesting direction is I think an interesting direction is qualitative feedback ingestion. So, as qualitative feedback ingestion. So, as qualitative feedback ingestion. So, as you move to these like production you move to these like production you move to these like production settings, sometimes you don't have settings, sometimes you don't have settings, sometimes you don't have access to like a clear-cut binary grade access to like a clear-cut binary grade access to like a clear-cut binary grade or like a numerical grade. Um, or like a numerical grade. Um, or like a numerical grade. Um, often times, what you receive back is often times, what you receive back is often times, what you receive back is like, "Hey, for this chat, the customer like, "Hey, for this chat, the customer like, "Hey, for this chat, the customer had this like piece of feedback." Um, had this like piece of feedback." Um, had this like piece of feedback." Um, if we can find a way to update our if we can find a way to update our if we can find a way to update our models based on that information, uh, models based on that information, uh, models based on that information, uh, that would also that would also that would also be prove extremely helpful. Um, and in be prove extremely helpful. Um, and in be prove extremely helpful. Um, and in fact, like self-distillation is one way fact, like self-distillation is one way fact, like self-distillation is one way in which we're exploring how we can do in which we're exploring how we can do in which we're exploring how we can do that, but it's like a pretty that, but it's like a pretty that, but it's like a pretty um, it's still a pretty open question. And then finally, I wanted to share a And then finally, I wanted to share a little bit about a vision for what the little bit about a vision for what the little bit about a vision for what the future of post-editing might look like future of post-editing might look like future of post-editing might look like if we sort of extrapolate out and take if we sort of extrapolate out and take if we sort of extrapolate out and take this sort of progression to its end.

this sort of progression to its end. this sort of progression to its end. I think eventually we might reach a I think eventually we might reach a I think eventually we might reach a setting where instead of just limiting setting where instead of just limiting setting where instead of just limiting ourselves to thinking about a specific ourselves to thinking about a specific ourselves to thinking about a specific task that we can improve the model on, task that we can improve the model on, task that we can improve the model on, we can actually just we can actually just we can actually just think of the model as this one think of the model as this one think of the model as this one deployment that can interact in many, deployment that can interact in many, deployment that can interact in many, many different settings. And like the many different settings. And like the many different settings. And like the task that you think about might just be task that you think about might just be task that you think about might just be the task of improving yourself the task of improving yourself the task of improving yourself on everything. on everything. on everything. Um and this model may be used for all Um and this model may be used for all Um and this model may be used for all sorts of different tasks, maybe across sorts of different tasks, maybe across sorts of different tasks, maybe across different users different users different users as well. as well. as well. And be able to sort of do some kind of And be able to sort of do some kind of And be able to sort of do some kind of reflection or introspection on like reflection or introspection on like reflection or introspection on like okay, for this sort of type of okay, for this sort of type of okay, for this sort of type of interaction, interaction, interaction, here's how I like self-evaluate and here's how I like self-evaluate and here's how I like self-evaluate and think that I'm doing. And then for this think that I'm doing. And then for this think that I'm doing. And then for this other type of interaction, other type of interaction, other type of interaction, um here's how I think I'm doing. And um here's how I think I'm doing. And um here's how I think I'm doing. And then being able to then being able to then being able to automatically take these interactions automatically take these interactions automatically take these interactions and compute weight updates from them and and compute weight updates from them and and compute weight updates from them and improve. And so, yeah, going back to the point And so, yeah, going back to the point that the agent learns every nook and that the agent learns every nook and that the agent learns every nook and cranny in your environment, um the exact cranny in your environment, um the exact cranny in your environment, um the exact environmental distribution, what if like environmental distribution, what if like environmental distribution, what if like the environment was just like every the environment was just like every the environment was just like every interaction that the agent ever has? And interaction that the agent ever has? And interaction that the agent ever has? And then in addition, we had some way that then in addition, we had some way that then in addition, we had some way that the model could evaluate itself.

the model could evaluate itself. the model could evaluate itself. Um basically one one like basically one one like question that we've Or sorry, one question that we've Or sorry, one question that we've Or sorry, one challenge that we've seen is like if challenge that we've seen is like if challenge that we've seen is like if you're only focusing on like what you're only focusing on like what you're only focusing on like what improving on one task at a time or like improving on one task at a time or like improving on one task at a time or like flagging one failure mode at a time, flagging one failure mode at a time, flagging one failure mode at a time, you're kind of playing a game of you're kind of playing a game of you're kind of playing a game of Whac-A-Mole where Whac-A-Mole where Whac-A-Mole where as soon as a new thing pops up, you need as soon as a new thing pops up, you need as soon as a new thing pops up, you need to scramble and like create new data or to scramble and like create new data or to scramble and like create new data or new environments and improve the model new environments and improve the model new environments and improve the model that way. that way. that way. With this kind of like With this kind of like With this kind of like self-improving self-improving self-improving system that understands interactions system that understands interactions system that understands interactions from like understands every interaction from like understands every interaction from like understands every interaction from the environment you would like kind from the environment you would like kind from the environment you would like kind of get around this problem and you of get around this problem and you of get around this problem and you wouldn't need to worry about it anymore. So I want to leave people on So I want to leave people on this quote from a paper on a year ago this quote from a paper on a year ago this quote from a paper on a year ago which I find extremely relevant now which I find extremely relevant now which I find extremely relevant now which is that AI is at the cusp of a new which is that AI is at the cusp of a new which is that AI is at the cusp of a new period in which experience will become period in which experience will become period in which experience will become the dominant medium of improvement and the dominant medium of improvement and the dominant medium of improvement and ultimately dwarf the scale of human data ultimately dwarf the scale of human data ultimately dwarf the scale of human data used in today's systems.

And yeah. And yeah. If you all have any questions I'm happy If you all have any questions I'm happy If you all have any questions I'm happy to take them after and uh to take them after and uh to take them after and uh Yeah, you can also email me at that Yeah, you can also email me at that Yeah, you can also email me at that address. address. address. Thank you. Thank you. Thank you. >> [applause]