Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke Labs

Hey everyone, um today I'll be talking Hey everyone, um today I'll be talking about data and uh environment curation about data and uh environment curation about data and uh environment curation for uh post- training LLMs. And I am for uh post- training LLMs. And I am for uh post- training LLMs. And I am Mahesh Satyimi. Um I'm co-founder and Mahesh Satyimi. Um I'm co-founder and Mahesh Satyimi. Um I'm co-founder and CEO of Bespoke Labs. And previously I CEO of Bespoke Labs. And previously I CEO of Bespoke Labs. And previously I was a researcher and uh engineer at uh was a researcher and uh engineer at uh was a researcher and uh engineer at uh Google deep mind. So very briefly I will Google deep mind. So very briefly I will Google deep mind. So very briefly I will tell you a little bit about uh bespoke tell you a little bit about uh bespoke tell you a little bit about uh bespoke and uh after that the talk will be and uh after that the talk will be and uh after that the talk will be mostly around uh opensource work we have mostly around uh opensource work we have mostly around uh opensource work we have done. So bespoke is an applied data done. So bespoke is an applied data done. So bespoke is an applied data research lab with a mission to help research lab with a mission to help research lab with a mission to help enterprises and frontier labs access enterprises and frontier labs access enterprises and frontier labs access high quality data and RL environments high quality data and RL environments high quality data and RL environments for their post training needs. So very for their post training needs. So very for their post training needs. So very briefly what we do and what we have done briefly what we do and what we have done briefly what we do and what we have done is that last year we put out something is that last year we put out something is that last year we put out something called curator which is a tool for called curator which is a tool for called curator which is a tool for curating uh synthetic data for post curating uh synthetic data for post curating uh synthetic data for post training with basically SFT and right training with basically SFT and right training with basically SFT and right after that actually deepse landed and we after that actually deepse landed and we after that actually deepse landed and we started an effort to curate reasoning started an effort to curate reasoning started an effort to curate reasoning data and that's how we started something data and that's how we started something data and that's how we started something called bespoke stratos which eventually called bespoke stratos which eventually called bespoke stratos which eventually formed something uh into the project formed something uh into the project formed something uh into the project called open thoughts which some of you called open thoughts which some of you called open thoughts which some of you hopefully know about and we have also hopefully know about and we have also hopefully know about and we have also been core contributors to terminal been core contributors to terminal been core contributors to terminal bench. Um you know these days we we do a bench. Um you know these days we we do a bench. Um you know these days we we do a lot of research and build and ship RL lot of research and build and ship RL lot of research and build and ship RL environments. So I was actually looking environments. So I was actually looking environments. So I was actually looking forward to the previous talk uh from forward to the previous talk uh from forward to the previous talk uh from Nick who is also you know uh doing Nick who is also you know uh doing Nick who is also you know uh doing something similar something similar something similar uh and and the other thing we do is we uh and and the other thing we do is we uh and and the other thing we do is we do a lot of post training and help do a lot of post training and help do a lot of post training and help enterprises uh to get their own custom

enterprises uh to get their own custom enterprises uh to get their own custom models. Right? That's the name of that's models. Right? That's the name of that's models. Right? That's the name of that's how we ended up with the bespoke uh t how we ended up with the bespoke uh t how we ended up with the bespoke uh t title for the company. title for the company. title for the company. Uh the the other thing I want to kind of Uh the the other thing I want to kind of Uh the the other thing I want to kind of mention is there's there is this you mention is there's there is this you mention is there's there is this you know you know in in our industry there know you know in in our industry there know you know in in our industry there are a lot of people who create data uh are a lot of people who create data uh are a lot of people who create data uh create RL environments and then there create RL environments and then there create RL environments and then there are the uh researchers who consume this. are the uh researchers who consume this. are the uh researchers who consume this. But I feel like there is this slight But I feel like there is this slight But I feel like there is this slight mismatch and it's kind of beneficial for mismatch and it's kind of beneficial for mismatch and it's kind of beneficial for someone to kind of go do both at the someone to kind of go do both at the someone to kind of go do both at the same time. And in fact uh as you're same time. And in fact uh as you're same time. And in fact uh as you're curating data you want to put yourself curating data you want to put yourself curating data you want to put yourself in the shoes of the researcher to see in the shoes of the researcher to see in the shoes of the researcher to see what what does it take to you know uh what what does it take to you know uh what what does it take to you know uh actually move the metrics on the models. actually move the metrics on the models. actually move the metrics on the models. So that's one of the motivations of how So that's one of the motivations of how So that's one of the motivations of how we kind of think about the other thing I we kind of think about the other thing I we kind of think about the other thing I want to kind of talk about is you know want to kind of talk about is you know want to kind of talk about is you know how you know uh AI has evolved right. So how you know uh AI has evolved right. So how you know uh AI has evolved right. So early on we used to think about and early on we used to think about and early on we used to think about and evaluate models on what they know. Um evaluate models on what they know. Um evaluate models on what they know. Um for example this is a this was a very for example this is a this was a very for example this is a this was a very popular benchmark on uh testing LLMs on popular benchmark on uh testing LLMs on popular benchmark on uh testing LLMs on various kinds of STEM humanities and all various kinds of STEM humanities and all various kinds of STEM humanities and all that knowledge and these days we have that knowledge and these days we have that knowledge and these days we have all these benchmarks that test uh how all these benchmarks that test uh how all these benchmarks that test uh how how agents are able to do things we have how agents are able to do things we have how agents are able to do things we have moved on from knowing to doing right so moved on from knowing to doing right so moved on from knowing to doing right so that's the idea of agents obviously and that's the idea of agents obviously and that's the idea of agents obviously and the one of the key principles or one of the one of the key principles or one of the one of the key principles or one of the key things about agents is that they the key things about agents is that they the key things about agents is that they are autonomous And there are as I was are autonomous And there are as I was are autonomous And there are as I was saying there are many benchmarks saying there are many benchmarks saying there are many benchmarks including uh swb bench terminal bench

including uh swb bench terminal bench including uh swb bench terminal bench and so on but ultimately for many people and so on but ultimately for many people and so on but ultimately for many people what they care about is are these agents what they care about is are these agents what they care about is are these agents autonomous for long durations of time uh autonomous for long durations of time uh autonomous for long durations of time uh Nick had uh sorry uh Ross had a great Nick had uh sorry uh Ross had a great Nick had uh sorry uh Ross had a great talk on long horizon right so that's the talk on long horizon right so that's the talk on long horizon right so that's the goal is eventually we make these agents goal is eventually we make these agents goal is eventually we make these agents autonomous for maybe few hours or you autonomous for maybe few hours or you autonomous for maybe few hours or you know few days or few weeks and what is know few days or few weeks and what is know few days or few weeks and what is it that's blocking the uh autonomy of it that's blocking the uh autonomy of it that's blocking the uh autonomy of agents. It's basically reliability, agents. It's basically reliability, agents. It's basically reliability, right? So, at some point something falls right? So, at some point something falls right? So, at some point something falls apart like either they called the wrong apart like either they called the wrong apart like either they called the wrong tool or they made a mistake and you know tool or they made a mistake and you know tool or they made a mistake and you know what what not, right? And what what's what what not, right? And what what's what what not, right? And what what's one lever to improve reliability there? one lever to improve reliability there? one lever to improve reliability there? There are of course many um obviously There are of course many um obviously There are of course many um obviously you can prompt your way to improving the you can prompt your way to improving the you can prompt your way to improving the agents u reliability or you can uh agents u reliability or you can uh agents u reliability or you can uh update the harness you know the tools update the harness you know the tools update the harness you know the tools and whatnot but postraining is a very and whatnot but postraining is a very and whatnot but postraining is a very powerful tool to improve reliability or powerful tool to improve reliability or powerful tool to improve reliability or or maybe even pre-train good models or maybe even pre-train good models or maybe even pre-train good models right so if you think of frontier labs right so if you think of frontier labs right so if you think of frontier labs this is one of their primary mechanisms this is one of their primary mechanisms this is one of their primary mechanisms of improving agents over to to uh get of improving agents over to to uh get of improving agents over to to uh get better [clears throat] better [clears throat] better [clears throat] u capability ities in u capability ities in u capability ities in uh various domains or you know or for uh various domains or you know or for uh various domains or you know or for better you know benchmark numbers or better you know benchmark numbers or better you know benchmark numbers or better u um autonomy for long longer and better u um autonomy for long longer and better u um autonomy for long longer and longer durations and for post training longer durations and for post training longer durations and for post training one of the popular techniques as you one of the popular techniques as you one of the popular techniques as you know is reinforcement learning and

know is reinforcement learning and know is reinforcement learning and that's kind of um something you know a that's kind of um something you know a that's kind of um something you know a lot of you are excited about is the you lot of you are excited about is the you lot of you are excited about is the you know notion of RL environments but know notion of RL environments but know notion of RL environments but ultimately for post training be it SFT ultimately for post training be it SFT ultimately for post training be it SFT or or uh reinforcement learning data is or or uh reinforcement learning data is or or uh reinforcement learning data is the bottleneck right so when when I talk the bottleneck right so when when I talk the bottleneck right so when when I talk about data RLNs are also something I'm about data RLNs are also something I'm about data RLNs are also something I'm calling it as data it's just the data is calling it as data it's just the data is calling it as data it's just the data is now in a very different shape now in a very different shape now in a very different shape u again here you know compute is kind of u again here you know compute is kind of u again here you know compute is kind of well definfined models you know uh good well definfined models you know uh good well definfined models you know uh good sort of models exist and the uh sort of models exist and the uh sort of models exist and the uh infrastructure to post train for example infrastructure to post train for example infrastructure to post train for example u the There are various providers like u the There are various providers like u the There are various providers like fireworks, tinker or uh slime world and fireworks, tinker or uh slime world and fireworks, tinker or uh slime world and whatnot. So all all of those are whatnot. So all all of those are whatnot. So all all of those are somewhat well defined. Most of the somewhat well defined. Most of the somewhat well defined. Most of the places where people struggle especially places where people struggle especially places where people struggle especially enterprises is that they don't have enterprises is that they don't have enterprises is that they don't have access to good quality data and RLNs and access to good quality data and RLNs and access to good quality data and RLNs and this obviously also applies to frontier this obviously also applies to frontier this obviously also applies to frontier labs where they have all this infra labs where they have all this infra labs where they have all this infra setup and they are you know needing good setup and they are you know needing good setup and they are you know needing good quality RLMs right uh beyond so that quality RLMs right uh beyond so that quality RLMs right uh beyond so that that's one of the this is kind of how we that's one of the this is kind of how we that's one of the this is kind of how we are thinking about why to invest time in are thinking about why to invest time in are thinking about why to invest time in you know doing data research and RLN you know doing data research and RLN you know doing data research and RLN research and as a side note one of the research and as a side note one of the research and as a side note one of the um other there are many other benefits um other there are many other benefits um other there are many other benefits of post- training for example you can of post- training for example you can of post- training for example you can reduce latency or improve cost reduce latency or improve cost reduce latency or improve cost throughput and whatnot and I'll give one throughput and whatnot and I'll give one throughput and whatnot and I'll give one concrete example of a post- training concrete example of a post- training concrete example of a post- training work we did uh with one of the

work we did uh with one of the work we did uh with one of the enterprises enterprises enterprises so in this talk I will mostly uh cover so in this talk I will mostly uh cover so in this talk I will mostly uh cover some of the work we have done in the some of the work we have done in the some of the work we have done in the open-source uh community so we did some open-source uh community so we did some open-source uh community so we did some work on curating reasoning data for work on curating reasoning data for work on curating reasoning data for reasoning models and for uh you curating reasoning models and for uh you curating reasoning models and for uh you curating trajectories and uh environments for trajectories and uh environments for trajectories and uh environments for agents and recently we had an engagement agents and recently we had an engagement agents and recently we had an engagement with post training which uh I I'll very with post training which uh I I'll very with post training which uh I I'll very briefly talk about and some tools on briefly talk about and some tools on briefly talk about and some tools on data curation data curation data curation so open thoughts um is a so open thoughts um is a so open thoughts um is a reasoning data set as well as a paper reasoning data set as well as a paper reasoning data set as well as a paper right so we we started this effort last right so we we started this effort last right so we we started this effort last year as I was saying this uh we we after year as I was saying this uh we we after year as I was saying this uh we we after DC came out we realized that there is uh DC came out we realized that there is uh DC came out we realized that there is uh lack lack of very high quality reasoning lack lack of very high quality reasoning lack lack of very high quality reasoning data in the uh community. Obviously the data in the uh community. Obviously the data in the uh community. Obviously the labs have access to good data but labs have access to good data but labs have access to good data but outside we didn't have access to data outside we didn't have access to data outside we didn't have access to data right so we we at bespoke started this right so we we at bespoke started this right so we we at bespoke started this effort called bespoke stratos and then effort called bespoke stratos and then effort called bespoke stratos and then we realized that this is actually quite we realized that this is actually quite we realized that this is actually quite useful so we joined um together with useful so we joined um together with useful so we joined um together with various folks in uh Stanford UC Berkeley various folks in uh Stanford UC Berkeley various folks in uh Stanford UC Berkeley Udub and so on to create this consortium Udub and so on to create this consortium Udub and so on to create this consortium called open thoughts and we did lot of called open thoughts and we did lot of called open thoughts and we did lot of work on basically identifying the work on basically identifying the work on basically identifying the curation recipe and we also published curation recipe and we also published curation recipe and we also published this as a paper in night of this year this as a paper in night of this year this as a paper in night of this year and this is the main figure of the and this is the main figure of the and this is the main figure of the paper. So what it shows is like we we paper. So what it shows is like we we paper. So what it shows is like we we figured out a curation recipe and it figured out a curation recipe and it figured out a curation recipe and it shows the scaling law right. So again shows the scaling law right. So again shows the scaling law right. So again this is last year when Amy uh and and uh

this is last year when Amy uh and and uh this is last year when Amy uh and and uh live codebench and these these were some live codebench and these these were some live codebench and these these were some of the popular benchmarks. What we of the popular benchmarks. What we of the popular benchmarks. What we showed is that with this recipe if you showed is that with this recipe if you showed is that with this recipe if you keep you know scaling up the data set keep you know scaling up the data set keep you know scaling up the data set size the the you know the the it's a size the the you know the the it's a size the the you know the the it's a scalable recipe right the the metrics scalable recipe right the the metrics scalable recipe right the the metrics also improve uh it's actually very also improve uh it's actually very also improve uh it's actually very widely used as well for example this is widely used as well for example this is widely used as well for example this is um Microsoft cso tweeting about the work um Microsoft cso tweeting about the work um Microsoft cso tweeting about the work and this alle Alex is my uh co-founder and this alle Alex is my uh co-founder and this alle Alex is my uh co-founder he's a chief scientist and also a he's a chief scientist and also a he's a chief scientist and also a professor at UC Berkeley and this is professor at UC Berkeley and this is professor at UC Berkeley and this is John Schulman talking about open John Schulman talking about open John Schulman talking about open thoughts that he as he and his uh thoughts that he as he and his uh thoughts that he as he and his uh colleagues have been using it internally colleagues have been using it internally colleagues have been using it internally at uh thinking machines right and some at uh thinking machines right and some at uh thinking machines right and some of their blog posts also reference this of their blog posts also reference this of their blog posts also reference this so I'll uh talk about how we did the so I'll uh talk about how we did the so I'll uh talk about how we did the curation for open thoughts um this is curation for open thoughts um this is curation for open thoughts um this is the pipeline that we used so you start the pipeline that we used so you start the pipeline that we used so you start with curate you start with a bunch of with curate you start with a bunch of with curate you start with a bunch of source questions right so there are source questions right so there are source questions right so there are various data sets out there that have various data sets out there that have various data sets out there that have the uh prompt response and we choose the uh prompt response and we choose the uh prompt response and we choose with the prompt so we start with the with the prompt so we start with the with the prompt so we start with the prompts. These are various uh sources we prompts. These are various uh sources we prompts. These are various uh sources we have. And then uh if you look at the have. And then uh if you look at the have. And then uh if you look at the paper, so if you look at this graph for paper, so if you look at this graph for paper, so if you look at this graph for any given data point, say if there are any given data point, say if there are any given data point, say if there are 10,000 samples that you want, the 10,000 samples that you want, the 10,000 samples that you want, the question is then how do you choose uh question is then how do you choose uh question is then how do you choose uh the questions from all these different the questions from all these different the questions from all these different data set so that you have 10,000 uh for data set so that you have 10,000 uh for data set so that you have 10,000 uh for the data point. So the the then there is the data point. So the the then there is the data point. So the the then there is the aspect around how do you mix these

the aspect around how do you mix these the aspect around how do you mix these questions. So uh you can use various questions. So uh you can use various questions. So uh you can use various methods. So the paper talks about for methods. So the paper talks about for methods. So the paper talks about for example using LLMs to check for whether example using LLMs to check for whether example using LLMs to check for whether this is a good good question hardness of this is a good good question hardness of this is a good good question hardness of a question and so on. And then you want a question and so on. And then you want a question and so on. And then you want to filter questions u and generate the to filter questions u and generate the to filter questions u and generate the answers. Again this is all like driven answers. Again this is all like driven answers. Again this is all like driven by LLMs right? So this is the curation by LLMs right? So this is the curation by LLMs right? So this is the curation recipe we did for uh creating this recipe we did for uh creating this recipe we did for uh creating this reasoning data set and the answer reasoning data set and the answer reasoning data set and the answer generation is using teacher models. So generation is using teacher models. So generation is using teacher models. So you can take other reasoning data uh you can take other reasoning data uh you can take other reasoning data uh reasoning models such as deepseek or reasoning models such as deepseek or reasoning models such as deepseek or quenbased models or even gemini and quenbased models or even gemini and quenbased models or even gemini and whatnot and then you can also filter the whatnot and then you can also filter the whatnot and then you can also filter the answers once you have the answers for uh answers once you have the answers for uh answers once you have the answers for uh these questions. Um and and then you can these questions. Um and and then you can these questions. Um and and then you can also you know given a question generate also you know given a question generate also you know given a question generate multiple answers or a single answer. So multiple answers or a single answer. So multiple answers or a single answer. So these are various knobs in the curation these are various knobs in the curation these are various knobs in the curation recipe and the systematic way of doing recipe and the systematic way of doing recipe and the systematic way of doing this is like you run ablations and this is like you run ablations and this is like you run ablations and figure out which uh you know in each of figure out which uh you know in each of figure out which uh you know in each of these stages what works and you kind of these stages what works and you kind of these stages what works and you kind of proceed to the next. So after doing all proceed to the next. So after doing all proceed to the next. So after doing all of this you get the final recipe right.

of this you get the final recipe right. of this you get the final recipe right. So this uh you can read this paper it So this uh you can read this paper it So this uh you can read this paper it has lots and lots of uh you know has lots and lots of uh you know has lots and lots of uh you know information about how we did the information about how we did the information about how we did the curation but here are some of the curation but here are some of the curation but here are some of the learnings that you know some of them are learnings that you know some of them are learnings that you know some of them are quite uh counterintuitive and some of quite uh counterintuitive and some of quite uh counterintuitive and some of this was also covered in last year's uh this was also covered in last year's uh this was also covered in last year's uh AI uh AI engineer conference. AI uh AI engineer conference. AI uh AI engineer conference. For example sampling um multiple answers For example sampling um multiple answers For example sampling um multiple answers per question works pretty well. This is per question works pretty well. This is per question works pretty well. This is something that uh we it's it's kind of something that uh we it's it's kind of something that uh we it's it's kind of counterintuitive. counterintuitive. counterintuitive. So as an example, something else we So as an example, something else we So as an example, something else we could have done is we could have had could have done is we could have had could have done is we could have had more much many more questions and then more much many more questions and then more much many more questions and then just answered them exactly once versus just answered them exactly once versus just answered them exactly once versus taking one question and answering them taking one question and answering them taking one question and answering them 16 times. 16 times. 16 times. The I think the the reasoning is The I think the the reasoning is The I think the the reasoning is probably that it gives like a variety of probably that it gives like a variety of probably that it gives like a variety of how reasoning is done. So the the during how reasoning is done. So the the during how reasoning is done. So the the during finetuning we also use the the reasoning finetuning we also use the the reasoning finetuning we also use the the reasoning traces, right? So I think the diversity traces, right? So I think the diversity traces, right? So I think the diversity helps there. And the other thing we saw helps there. And the other thing we saw helps there. And the other thing we saw is like the stronger teachers are not is like the stronger teachers are not is like the stronger teachers are not always the best uh uh stronger models always the best uh uh stronger models always the best uh uh stronger models are not always the better teachers. And are not always the better teachers. And are not always the better teachers. And there were a few other counterintuitive there were a few other counterintuitive there were a few other counterintuitive aspects around like you know uh aspects around like you know uh aspects around like you know uh synthetic question generation or synthetic question generation or synthetic question generation or question answering working whereas question answering working whereas question answering working whereas answer filtering and other aspects not answer filtering and other aspects not answer filtering and other aspects not working very well.

working very well. working very well. And after the open thoughts work which And after the open thoughts work which And after the open thoughts work which was around uh data curation for was around uh data curation for was around uh data curation for reasoning models such as you know uh reasoning models such as you know uh reasoning models such as you know uh deepse kind of models, we moved on to deepse kind of models, we moved on to deepse kind of models, we moved on to open thoughts agents which is um very open thoughts agents which is um very open thoughts agents which is um very similar but how do you curate these the similar but how do you curate these the similar but how do you curate these the the data and RL environments for uh the data and RL environments for uh the data and RL environments for uh training agents now right not models. We training agents now right not models. We training agents now right not models. We we have a very similar figure here. we have a very similar figure here. we have a very similar figure here. Again we want to establish scaling loss. Again we want to establish scaling loss. Again we want to establish scaling loss. Um so as you increase the data set size Um so as you increase the data set size Um so as you increase the data set size we want to make sure that the curation we want to make sure that the curation we want to make sure that the curation recipe actually works. Uh recipe actually works. Uh recipe actually works. Uh and again I'm I'm not going to go into and again I'm I'm not going to go into and again I'm I'm not going to go into details here but very similarly there details here but very similarly there details here but very similarly there are various ways of choosing different are various ways of choosing different are various ways of choosing different sources for example stack exchange and sources for example stack exchange and sources for example stack exchange and and whatnot. How do you mix the test? and whatnot. How do you mix the test? and whatnot. How do you mix the test? How do you filter? Generating the How do you filter? Generating the How do you filter? Generating the rollouts uh choosing the teacher and so rollouts uh choosing the teacher and so rollouts uh choosing the teacher and so on. And again these are some of the on. And again these are some of the on. And again these are some of the lessons learnings u lessons learnings u lessons learnings u as an example even here we saw that as an example even here we saw that as an example even here we saw that stronger models are not necessarily the stronger models are not necessarily the stronger models are not necessarily the uh best teachers right so we found out uh best teachers right so we found out uh best teachers right so we found out some some some of the I think uh um quen some some some of the I think uh um quen some some some of the I think uh um quen models were better than for example um models were better than for example um models were better than for example um um um um um um um claude models I think and sampling claude models I think and sampling claude models I think and sampling multiple answers again helped in this multiple answers again helped in this multiple answers again helped in this case synthetic rewriting and task case synthetic rewriting and task case synthetic rewriting and task augmentation um is something we thought augmentation um is something we thought augmentation um is something we thought will work but it didn't very work work will work but it didn't very work work will work but it didn't very work work very well and the other thing is like in very well and the other thing is like in very well and the other thing is like in in this whole process of building this

in this whole process of building this in this whole process of building this open thoughts agent SFT still open thoughts agent SFT still open thoughts agent SFT still contributed a lot to the gains um RL was contributed a lot to the gains um RL was contributed a lot to the gains um RL was kind of you know it's very comput kind of you know it's very comput kind of you know it's very comput inensive and for for the last few few inensive and for for the last few few inensive and for for the last few few percentages it really helped uh but but percentages it really helped uh but but percentages it really helped uh but but you know in many of the situations for you know in many of the situations for you know in many of the situations for example in enterprises SFT actually example in enterprises SFT actually example in enterprises SFT actually works works pretty well, right? works works pretty well, right? works works pretty well, right? And here is one concrete example I And here is one concrete example I And here is one concrete example I wanted to share on um uh actually wanted to share on um uh actually wanted to share on um uh actually deploying something to production, deploying something to production, deploying something to production, right? By post training. So we have seen right? By post training. So we have seen right? By post training. So we have seen a lot of people talk about post- a lot of people talk about post- a lot of people talk about post- training but in enterprise settings we training but in enterprise settings we training but in enterprise settings we haven't seen lot of successes at least haven't seen lot of successes at least haven't seen lot of successes at least haven't seen uh that here is a very haven't seen uh that here is a very haven't seen uh that here is a very concrete example of uh with intude there concrete example of uh with intude there concrete example of uh with intude there is a this app called credit karma which is a this app called credit karma which is a this app called credit karma which if you install there is a page place if you install there is a page place if you install there is a page place where you can uh the the the app gives where you can uh the the the app gives where you can uh the the the app gives you a reasoning as to why a credit card you a reasoning as to why a credit card you a reasoning as to why a credit card has been recommended and this you can has been recommended and this you can has been recommended and this you can prompt a model to do this. But one of prompt a model to do this. But one of prompt a model to do this. But one of the reasons one of the places where it the reasons one of the places where it the reasons one of the places where it fails is that the you know it it it's fails is that the you know it it it's fails is that the you know it it it's not always compliant. So you have to not always compliant. So you have to not always compliant. So you have to have a long list of rules to make sure have a long list of rules to make sure have a long list of rules to make sure the responses are compliant and that the responses are compliant and that the responses are compliant and that actually blows up the latency. So actually blows up the latency. So actually blows up the latency. So answer here is like you want to curate answer here is like you want to curate answer here is like you want to curate data and post train right seems kind of data and post train right seems kind of data and post train right seems kind of straightforward but one of the things straightforward but one of the things straightforward but one of the things that we ran into is um the data set can that we ran into is um the data set can that we ran into is um the data set can be quite impa imbalance and lot lots and be quite impa imbalance and lot lots and be quite impa imbalance and lot lots and lots of places for example you will have lots of places for example you will have lots of places for example you will have 0% APR and the model after fine-tuning

0% APR and the model after fine-tuning 0% APR and the model after fine-tuning can kind of hallucinate the these uh can kind of hallucinate the these uh can kind of hallucinate the these uh numbers so this again kind of ties back numbers so this again kind of ties back numbers so this again kind of ties back to what Ross talked about some time back to what Ross talked about some time back to what Ross talked about some time back with respect to attacks with respect to attacks with respect to attacks And we kind uh we created this specific And we kind uh we created this specific And we kind uh we created this specific uh curation recipe where instead of just uh curation recipe where instead of just uh curation recipe where instead of just having these uh question the the prompt having these uh question the the prompt having these uh question the the prompt response pairs in plain language, we response pairs in plain language, we response pairs in plain language, we added these uh tags which helped the added these uh tags which helped the added these uh tags which helped the model to focus on you know uh the the model to focus on you know uh the the model to focus on you know uh the the kind of form rather than the specific kind of form rather than the specific kind of form rather than the specific numbers itself and that gave a big boost numbers itself and that gave a big boost numbers itself and that gave a big boost and uh we we saw that um the the overall and uh we we saw that um the the overall and uh we we saw that um the the overall the compliance metrics improved the the compliance metrics improved the the compliance metrics improved the latency improved, the throughput latency improved, the throughput latency improved, the throughput improved and eventually you know they improved and eventually you know they improved and eventually you know they they are able to own the model right as they are able to own the model right as they are able to own the model right as frontier models improve they don't need frontier models improve they don't need frontier models improve they don't need to kind of go and u um uh update it and to kind of go and u um uh update it and to kind of go and u um uh update it and also as we see now the uh frontier also as we see now the uh frontier also as we see now the uh frontier [clears throat] models are also getting [clears throat] models are also getting [clears throat] models are also getting more and more expensive and you know more and more expensive and you know more and more expensive and you know this kind of g gives them a very good this kind of g gives them a very good this kind of g gives them a very good way for owning the model and also um way for owning the model and also um way for owning the model and also um lowering the costs.

lowering the costs. lowering the costs. I think with that I want to briefly I think with that I want to briefly I think with that I want to briefly touch upon um uh you know curator the touch upon um uh you know curator the touch upon um uh you know curator the tooling that we had built last year um tooling that we had built last year um tooling that we had built last year um which is for curating reasoning data. So which is for curating reasoning data. So which is for curating reasoning data. So uh what it does is you can basically uh uh what it does is you can basically uh uh what it does is you can basically uh you know um specify the you you can you know um specify the you you can you know um specify the you you can either go with say a hugging face data either go with say a hugging face data either go with say a hugging face data set where you have various prompts or uh set where you have various prompts or uh set where you have various prompts or uh in many situations you may have in many situations you may have in many situations you may have collected logs and you want to get the collected logs and you want to get the collected logs and you want to get the responses and fine-tune a model. So this responses and fine-tune a model. So this responses and fine-tune a model. So this curator kind of makes it pretty easy to curator kind of makes it pretty easy to curator kind of makes it pretty easy to do that and it comes with the do that and it comes with the do that and it comes with the integration with you know uh tinker and integration with you know uh tinker and integration with you know uh tinker and fireworks and this this is again the fireworks and this this is again the fireworks and this this is again the tool that we used um originally for tool that we used um originally for tool that we used um originally for curating open thoughts curating open thoughts curating open thoughts and here is a very very detailed diagram and here is a very very detailed diagram and here is a very very detailed diagram of what we are building today but um of what we are building today but um of what we are building today but um this again connects back to um what Ross this again connects back to um what Ross this again connects back to um what Ross was talking about where he was talking was talking about where he was talking was talking about where he was talking about algorithms about algorithms about algorithms uh environments and compute root right uh environments and compute root right uh environments and compute root right so it it feels like you know we are kind so it it feels like you know we are kind so it it feels like you know we are kind of converging on something very similar of converging on something very similar of converging on something very similar so if you think about uh the the stack so if you think about uh the the stack so if you think about uh the the stack that is needed to say not just curate that is needed to say not just curate that is needed to say not just curate these RL environments but to post train these RL environments but to post train these RL environments but to post train models one of the things you need is models one of the things you need is models one of the things you need is obviously handle on like how do you obviously handle on like how do you obviously handle on like how do you build these RL environments how do you build these RL environments how do you build these RL environments how do you measure the quality how do you track the measure the quality how do you track the measure the quality how do you track the different versions and so on so that's different versions and so on so that's different versions and so on so that's one of the layers and below that you one of the layers and below that you one of the layers and below that you want various infrastructure to uh um

want various infrastructure to uh um want various infrastructure to uh um sand use sandboxes, right? to spin up sand use sandboxes, right? to spin up sand use sandboxes, right? to spin up the rollouts to to spin up the sandboxes the rollouts to to spin up the sandboxes the rollouts to to spin up the sandboxes to generate rollouts and especially if to generate rollouts and especially if to generate rollouts and especially if you have long horizon rollouts then you have long horizon rollouts then you have long horizon rollouts then maybe at some points you need to do a maybe at some points you need to do a maybe at some points you need to do a checkpointing and then you need to be checkpointing and then you need to be checkpointing and then you need to be able to snapshot or roll back to able to snapshot or roll back to able to snapshot or roll back to something else right so that's the other something else right so that's the other something else right so that's the other uh the lower level u you know compute uh the lower level u you know compute uh the lower level u you know compute and orchestration and at the top I have and orchestration and at the top I have and orchestration and at the top I have been giving examples on post training so been giving examples on post training so been giving examples on post training so there is all this uh layer around like there is all this uh layer around like there is all this uh layer around like how do you do SFT how do you do RL and how do you do SFT how do you do RL and how do you do SFT how do you do RL and so on but there is also this method so on but there is also this method so on but there is also this method called Japa which is around uh which is called Japa which is around uh which is called Japa which is around uh which is on prompt optimization. I don't know if on prompt optimization. I don't know if on prompt optimization. I don't know if you if you guys have heard of it but you you if you guys have heard of it but you you if you guys have heard of it but you can use LLMs itself to uh to to kind of can use LLMs itself to uh to to kind of can use LLMs itself to uh to to kind of optimize the prompts based on optimize the prompts based on optimize the prompts based on reflection. Um so that also works pretty reflection. Um so that also works pretty reflection. Um so that also works pretty well for updating the system prompts and well for updating the system prompts and well for updating the system prompts and also the harnesses. So this is kind of I also the harnesses. So this is kind of I also the harnesses. So this is kind of I feel like you know the the new feel like you know the the new feel like you know the the new architecture or the new reference uh architecture or the new reference uh architecture or the new reference uh stack for how how at least we are stack for how how at least we are stack for how how at least we are building and how many others are building and how many others are building and how many others are building um the the stack on how to building um the the stack on how to building um the the stack on how to build the RLMs and then also post train build the RLMs and then also post train build the RLMs and then also post train agents. I think with that uh I will uh agents. I think with that uh I will uh agents. I think with that uh I will uh end the talk and u you know happy to end the talk and u you know happy to end the talk and u you know happy to take questions offline.