The State of Model Routing — NVIDIA, Cognition, OpenRouter

>> have been really exciting. We've tried >> have been really exciting. We've tried to get a bunch of the industry leaders to get a bunch of the industry leaders to get a bunch of the industry leaders together to talk about some of the together to talk about some of the together to talk about some of the problems that are that we're facing as problems that are that we're facing as problems that are that we're facing as we try to run more on local. we try to run more on local. we try to run more on local. If you guys were here for the first If you guys were here for the first If you guys were here for the first panel, panel, panel, one of the things that we talked about one of the things that we talked about one of the things that we talked about was model routing. We firmly believe was model routing. We firmly believe was model routing. We firmly believe that we're in a that we're in a that we're in a multi-model world. I think you heard multi-model world. I think you heard multi-model world. I think you heard this from many of the panelists. this from many of the panelists. this from many of the panelists. Anyone who is deploying AI in production Anyone who is deploying AI in production Anyone who is deploying AI in production and who is doing so locally is seeing and who is doing so locally is seeing and who is doing so locally is seeing that multi-model world. That's why we that multi-model world. That's why we that multi-model world. That's why we released these NeMo Triton models at released these NeMo Triton models at released these NeMo Triton models at NVIDIA. Everything is released from the NVIDIA. Everything is released from the NVIDIA. Everything is released from the data sets to the weights with recipes so data sets to the weights with recipes so data sets to the weights with recipes so that you can customize them. We do that that you can customize them. We do that that you can customize them. We do that because we know that people customizing because we know that people customizing because we know that people customizing models is going to be huge. models is going to be huge. models is going to be huge. And so, this panel is really exciting And so, this panel is really exciting And so, this panel is really exciting cuz we're going to talk specifically cuz we're going to talk specifically cuz we're going to talk specifically about model routing. So, as you are about model routing. So, as you are about model routing. So, as you are picking which model to use, how does picking which model to use, how does picking which model to use, how does that how essentially how does that that how essentially how does that that how essentially how does that tooling itself look? Um, do you guys tooling itself look? Um, do you guys tooling itself look? Um, do you guys want to introduce yourselves? want to introduce yourselves? want to introduce yourselves? >> Yeah, sure. Um, I'm Walden. I'm the >> Yeah, sure. Um, I'm Walden. I'm the >> Yeah, sure. Um, I'm Walden. I'm the co-founder of Cognition. We build Devin, co-founder of Cognition. We build Devin, co-founder of Cognition. We build Devin, AI software engineer. Uh, in addition to AI software engineer. Uh, in addition to AI software engineer. Uh, in addition to the product, we spend a lot of time the product, we spend a lot of time the product, we spend a lot of time partnering with our customers to figure partnering with our customers to figure partnering with our customers to figure out how they should deploy these models out how they should deploy these models out how they should deploy these models uh and these agents. And one of the uh and these agents. And one of the uh and these agents. And one of the things they're constantly asking us things they're constantly asking us things they're constantly asking us nowadays is basically how do I know the nowadays is basically how do I know the nowadays is basically how do I know the ROI of our models and how do I know ROI of our models and how do I know ROI of our models and how do I know which tasks I can actually let our which tasks I can actually let our which tasks I can actually let our engineers spend the most expensive engineers spend the most expensive engineers spend the most expensive models on versus, you know, letting them models on versus, you know, letting them models on versus, you know, letting them use a more cost-efficient model. And so, use a more cost-efficient model. And so, use a more cost-efficient model. And so, that's why we're we're also thinking a that's why we're we're also thinking a that's why we're we're also thinking a lot more about multi-model model routing lot more about multi-model model routing lot more about multi-model model routing nowadays.

nowadays. nowadays. >> Totally. >> Totally. >> Totally. >> Yeah. I'm Carter. You guys heard from me >> Yeah. I'm Carter. You guys heard from me >> Yeah. I'm Carter. You guys heard from me a little bit earlier, but if you weren't a little bit earlier, but if you weren't a little bit earlier, but if you weren't here, I'm a developer tech engineer at here, I'm a developer tech engineer at here, I'm a developer tech engineer at NVIDIA. And uh ultimately, I spend a lot NVIDIA. And uh ultimately, I spend a lot NVIDIA. And uh ultimately, I spend a lot of time thinking about how to get of time thinking about how to get of time thinking about how to get intelligence into as many developers' intelligence into as many developers' intelligence into as many developers' hands as possible. And uh something that hands as possible. And uh something that hands as possible. And uh something that is continually becoming a not an issue, is continually becoming a not an issue, is continually becoming a not an issue, but something that is top of mind for a but something that is top of mind for a but something that is top of mind for a lot of developers is uh as you use more lot of developers is uh as you use more lot of developers is uh as you use more intelligence and the frontier models get intelligence and the frontier models get intelligence and the frontier models get more expensive, uh it becomes somewhat more expensive, uh it becomes somewhat more expensive, uh it becomes somewhat cost prohibitive to to use the best uh cost prohibitive to to use the best uh cost prohibitive to to use the best uh tools, what feels like the best tools, tools, what feels like the best tools, tools, what feels like the best tools, as much as you would like to use them. as much as you would like to use them. as much as you would like to use them. And so, this has become a recent uh you And so, this has become a recent uh you And so, this has become a recent uh you know, know, know, focus is, how can we, you know, still focus is, how can we, you know, still focus is, how can we, you know, still get the same desired outputs, but get the same desired outputs, but get the same desired outputs, but actually uh both as an individual actually uh both as an individual actually uh both as an individual developer, but also imagine startups and developer, but also imagine startups and developer, but also imagine startups and small companies, how can you leverage small companies, how can you leverage small companies, how can you leverage this incredible tool without totally this incredible tool without totally this incredible tool without totally breaking the bank? breaking the bank? breaking the bank? >> Uh I'm Donne. I have worked on model >> Uh I'm Donne. I have worked on model >> Uh I'm Donne. I have worked on model evaluations evaluations evaluations uh both in terms of its accuracies and uh both in terms of its accuracies and uh both in terms of its accuracies and efficiency and cost understanding of the efficiency and cost understanding of the efficiency and cost understanding of the model. Uh and then I try and understand model. Uh and then I try and understand model. Uh and then I try and understand those implement those learnings and uh those implement those learnings and uh those implement those learnings and uh help build a router. So, it's basically help build a router. So, it's basically help build a router. So, it's basically my job is to understand the behavior of my job is to understand the behavior of my job is to understand the behavior of of the model on an intimate level and of the model on an intimate level and of the model on an intimate level and then use those learnings to both improve then use those learnings to both improve then use those learnings to both improve the model and try and design a system of the model and try and design a system of the model and try and design a system of model that can work together with each model that can work together with each model that can work together with each other.

other. other. >> Totally. Um yeah, I love a lot of the >> Totally. Um yeah, I love a lot of the >> Totally. Um yeah, I love a lot of the research that you're doing at Nvidia as research that you're doing at Nvidia as research that you're doing at Nvidia as we kind of see the space through. I we kind of see the space through. I we kind of see the space through. I think what's really interesting is model think what's really interesting is model think what's really interesting is model routing itself is pretty new still. And routing itself is pretty new still. And routing itself is pretty new still. And so, what you'll notice is there isn't a so, what you'll notice is there isn't a so, what you'll notice is there isn't a very clear solution here. That was very clear solution here. That was very clear solution here. That was something that came up on the first something that came up on the first something that came up on the first panel is that there is a lot of space panel is that there is a lot of space panel is that there is a lot of space for startups and for companies in the for startups and for companies in the for startups and for companies in the ecosystem to fill in a solution here cuz ecosystem to fill in a solution here cuz ecosystem to fill in a solution here cuz we're still figuring out how to best do we're still figuring out how to best do we're still figuring out how to best do these patterns. And I think, Walden, I these patterns. And I think, Walden, I these patterns. And I think, Walden, I want to kind of ask you. So, um want to kind of ask you. So, um want to kind of ask you. So, um Cognition just released Fusion, your Cognition just released Fusion, your Cognition just released Fusion, your guys's model router. And when you guys guys's model router. And when you guys guys's model router. And when you guys released it, you in your blog said that released it, you in your blog said that released it, you in your blog said that you're actually getting better you're actually getting better you're actually getting better performance than Fable, um than these performance than Fable, um than these performance than Fable, um than these frontier models. And I feel like that frontier models. And I feel like that frontier models. And I feel like that was a very surprising statement to hear was a very surprising statement to hear was a very surprising statement to hear because we were thinking that you're because we were thinking that you're because we were thinking that you're getting as good or close enough usually getting as good or close enough usually getting as good or close enough usually when we're running on edge, when we're when we're running on edge, when we're when we're running on edge, when we're running local in these compute strained, running local in these compute strained, running local in these compute strained, smaller footprint models. But you guys smaller footprint models. But you guys smaller footprint models. But you guys are getting better. Can you explain how? are getting better. Can you explain how? are getting better. Can you explain how? >> Yeah, absolutely. Um so, >> Yeah, absolutely. Um so, >> Yeah, absolutely. Um so, I also want to be be clear about I also want to be be clear about I also want to be be clear about something here is like, you know, we're something here is like, you know, we're something here is like, you know, we're not saying that that we gap above Fable not saying that that we gap above Fable not saying that that we gap above Fable level performance in in the same way level performance in in the same way level performance in in the same way that maybe Fable level performance gaps that maybe Fable level performance gaps that maybe Fable level performance gaps above other models. I think actually above other models. I think actually above other models. I think actually there's this really unintuitive there's this really unintuitive there's this really unintuitive dynamic where smarter models actually dynamic where smarter models actually dynamic where smarter models actually get better and better at delegating get better and better at delegating get better and better at delegating work. And so one of the work. And so one of the work. And so one of the philosophies we had with building a philosophies we had with building a philosophies we had with building a model router is we don't want to route model router is we don't want to route model router is we don't want to route people to a dumber model and then people to a dumber model and then people to a dumber model and then suddenly you're stuck with a model that suddenly you're stuck with a model that suddenly you're stuck with a model that doesn't know how to do your task. Next doesn't know how to do your task. Next doesn't know how to do your task. Next thing you know you're you're switching thing you know you're you're switching thing you know you're you're switching yourself back to smarter model anyways yourself back to smarter model anyways yourself back to smarter model anyways and now taking that expensive cost. And and now taking that expensive cost. And and now taking that expensive cost. And in general we think a lot of the in general we think a lot of the in general we think a lot of the existing model routing systems out there existing model routing systems out there existing model routing systems out there are probably the same ones people have

are probably the same ones people have are probably the same ones people have been using like a year ago. And so we been using like a year ago. And so we been using like a year ago. And so we really wanted to put out a new framework really wanted to put out a new framework really wanted to put out a new framework that actually lets people that actually lets people that actually lets people still feel like and and still have a still feel like and and still have a still feel like and and still have a frontier model in their system while frontier model in their system while frontier model in their system while getting all these like cost benefits. So getting all these like cost benefits. So getting all these like cost benefits. So yeah we we're reducing the cost of Fable yeah we we're reducing the cost of Fable yeah we we're reducing the cost of Fable level intelligence by 40%. The way we do level intelligence by 40%. The way we do level intelligence by 40%. The way we do that is we allow Fable to still do like that is we allow Fable to still do like that is we allow Fable to still do like the planning and the the hard decision the planning and the the hard decision the planning and the the hard decision making but delegate a lot of the work to making but delegate a lot of the work to making but delegate a lot of the work to an implementation model. And the an implementation model. And the an implementation model. And the implementation model can be you know one implementation model can be you know one implementation model can be you know one of these open source models be it a of these open source models be it a of these open source models be it a cheaper mini model. Um the unintuitive cheaper mini model. Um the unintuitive cheaper mini model. Um the unintuitive thing is even though it's cheaper thing is even though it's cheaper thing is even though it's cheaper because you're delegating the work to because you're delegating the work to because you're delegating the work to another model you can let that model go another model you can let that model go another model you can let that model go at the task with much more depth and at the task with much more depth and at the task with much more depth and intensity than you might otherwise. Like intensity than you might otherwise. Like intensity than you might otherwise. Like you can spin off like you know three sub you can spin off like you know three sub you can spin off like you know three sub agents to go and like explore the code agents to go and like explore the code agents to go and like explore the code base and maybe that's actually more base and maybe that's actually more base and maybe that's actually more comprehensive than if you had just let comprehensive than if you had just let comprehensive than if you had just let Fable explore the code base itself. So Fable explore the code base itself. So Fable explore the code base itself. So you're actually getting this like nice you're actually getting this like nice you're actually getting this like nice trade-off where it's both more cost trade-off where it's both more cost trade-off where it's both more cost efficient and it's also more efficient and it's also more efficient and it's also more comprehensive overall. comprehensive overall. comprehensive overall. >> Interesting. I see. So you're saying by >> Interesting. I see. So you're saying by >> Interesting. I see. So you're saying by using a bunch of smaller models you're using a bunch of smaller models you're using a bunch of smaller models you're essentially like essentially like essentially like for one example scouring the code base for one example scouring the code base for one example scouring the code base you can you you can explore it you can you you can explore it you can you you can explore it potentially better than if you were to potentially better than if you were to potentially better than if you were to just have one model just have one model just have one model I don't know figure out with what it's I don't know figure out with what it's I don't know figure out with what it's limited with its limited context with limited with its limited context with limited with its limited context with whatever path it's on.

whatever path it's on. whatever path it's on. >> Yeah, totally. >> Yeah, totally. >> Yeah, totally. >> But also if you think about the budget >> But also if you think about the budget >> But also if you think about the budget of if you were to say the frontier model of if you were to say the frontier model of if you were to say the frontier model costs this amount per token and the costs this amount per token and the costs this amount per token and the smaller model is this amount per per smaller model is this amount per per smaller model is this amount per per token and it's significantly cheaper token and it's significantly cheaper token and it's significantly cheaper then you can use a lot more tokens from then you can use a lot more tokens from then you can use a lot more tokens from the smaller model with the smaller model with the smaller model with >> still within the budget that it would >> still within the budget that it would >> still within the budget that it would have been from the Frontier model. have been from the Frontier model. have been from the Frontier model. >> I would also like to encourage everyone >> I would also like to encourage everyone >> I would also like to encourage everyone to think there are jagged capabilities to think there are jagged capabilities to think there are jagged capabilities in most models, right? So, coding is not in most models, right? So, coding is not in most models, right? So, coding is not one domain. Within, let's say data one domain. Within, let's say data one domain. Within, let's say data visualization, visualization, visualization, there'll be scikit-learn, there'll be there'll be scikit-learn, there'll be there'll be scikit-learn, there'll be matplotlib, there'll be something else. matplotlib, there'll be something else. matplotlib, there'll be something else. It largely comes down comes down to the It largely comes down comes down to the It largely comes down comes down to the training corpora that training corpora that training corpora that went into each of the models, so right? went into each of the models, so right? went into each of the models, so right? So, one model, while you're trying to do So, one model, while you're trying to do So, one model, while you're trying to do X type of work, let's say data X type of work, let's say data X type of work, let's say data visualization, visualization, visualization, and the other type is Y, that means and the other type is Y, that means and the other type is Y, that means let's say model building. Let's say let's say model building. Let's say let's say model building. Let's say you're trying to have a data science you're trying to have a data science you're trying to have a data science work stream, where you're trying to work stream, where you're trying to work stream, where you're trying to optimize for some kind of prediction, optimize for some kind of prediction, optimize for some kind of prediction, and then visualizing your results. and then visualizing your results. and then visualizing your results. Within that task, Within that task, Within that task, different models will have different different models will have different different models will have different strengths. So, not it's not necessary strengths. So, not it's not necessary strengths. So, not it's not necessary that model A that model A that model A if scores higher on a coding benchmark, if scores higher on a coding benchmark, if scores higher on a coding benchmark, is just plain better at every task that is just plain better at every task that is just plain better at every task that is. So, is. So, is. So, routing is a routing is a routing is a task of intimately intimately task of intimately intimately task of intimately intimately understanding of behavior of and understanding of behavior of and understanding of behavior of and strengths and weaknesses of different strengths and weaknesses of different strengths and weaknesses of different models, and then applying them thusly, models, and then applying them thusly, models, and then applying them thusly, right? I would I would encourage right? I would I would encourage right? I would I would encourage everyone to think, "Hey, models are everyone to think, "Hey, models are everyone to think, "Hey, models are strong at different things, rather than strong at different things, rather than strong at different things, rather than like there's one model to rule them like there's one model to rule them like there's one model to rule them all."

all." all." >> I see. And by the way, real quick, thank >> I see. And by the way, real quick, thank >> I see. And by the way, real quick, thank you Alex for joining. Yeah. you Alex for joining. Yeah. you Alex for joining. Yeah. >> Sorry I'm late. >> Sorry I'm late. >> Sorry I'm late. >> No. Oh, is that so? >> No. Oh, is that so? >> No. Oh, is that so? >> I might need yours in. >> I might need yours in. >> I might need yours in. Sorry I'm late. Sorry I'm late. Sorry I'm late. >> [laughter] >> [laughter] >> [laughter] >> I'm Alex from OpenRouter. >> I'm Alex from OpenRouter. >> I'm Alex from OpenRouter. Um thanks for having me now. Um thanks for having me now. Um thanks for having me now. >> Yeah, of course. Thank you so much. You >> Yeah, of course. Thank you so much. You >> Yeah, of course. Thank you so much. You came right from the airport, so this is came right from the airport, so this is came right from the airport, so this is perfect. I think um tonight that's perfect. I think um tonight that's perfect. I think um tonight that's that's super interesting. So, um the way that's super interesting. So, um the way that's super interesting. So, um the way that you're thinking through through that you're thinking through through that you're thinking through through model routing, it's not even just uh model routing, it's not even just uh model routing, it's not even just uh delegating to necessarily a smaller delegating to necessarily a smaller delegating to necessarily a smaller model, but like and maybe this is kind model, but like and maybe this is kind model, but like and maybe this is kind of what you're saying is can you can you of what you're saying is can you can you of what you're saying is can you can you put essentially a swarm of agents to put essentially a swarm of agents to put essentially a swarm of agents to accomplish the same task, and suddenly accomplish the same task, and suddenly accomplish the same task, and suddenly routing the task between them is is much routing the task between them is is much routing the task between them is is much is a is a problem to solve in and of is a is a problem to solve in and of is a is a problem to solve in and of itself. itself. itself. >> Yeah. So, if you look at like let's say >> Yeah. So, if you look at like let's say >> Yeah. So, if you look at like let's say let's take an easy example. Let's take a let's take an easy example. Let's take a let's take an easy example. Let's take a science or like scientific discovery as science or like scientific discovery as science or like scientific discovery as an example, right? Usually these are an example, right? Usually these are an example, right? Usually these are one-shot problems. It's incredibly hard. one-shot problems. It's incredibly hard. one-shot problems. It's incredibly hard. You have models think through this You have models think through this You have models think through this process, right? So, in that you have process, right? So, in that you have process, right? So, in that you have tons of sub-domains. Like tons and tons tons of sub-domains. Like tons and tons tons of sub-domains. Like tons and tons and tons. So, and tons. So, and tons. So, in that aspect, if you think about in that aspect, if you think about in that aspect, if you think about post-training like the post-training post-training like the post-training post-training like the post-training process of a model, they'd be tuned with process of a model, they'd be tuned with process of a model, they'd be tuned with different teachers. They'd be tuned on different teachers. They'd be tuned on different teachers. They'd be tuned on different sub-tasks.

different sub-tasks. different sub-tasks. So, those those um So, those those um So, those those um those overlapping strengths will be those overlapping strengths will be those overlapping strengths will be readily apparent when you're trying to readily apparent when you're trying to readily apparent when you're trying to understand failures of each models on understand failures of each models on understand failures of each models on different different sub-tasks. Once you different different sub-tasks. Once you different different sub-tasks. Once you understand that, you can orchestrate understand that, you can orchestrate understand that, you can orchestrate your system to leverage that arbitrage your system to leverage that arbitrage your system to leverage that arbitrage essentially, and that essentially essentially, and that essentially essentially, and that essentially becomes free. So, becomes free. So, becomes free. So, I think this is on LM router bench. I think this is on LM router bench. I think this is on LM router bench. There was there are tons of benchmarks There was there are tons of benchmarks There was there are tons of benchmarks out there. But, if you use these out there. But, if you use these out there. But, if you use these techniques, you can get like up to 10% techniques, you can get like up to 10% techniques, you can get like up to 10% higher accuracy even, right? It depends higher accuracy even, right? It depends higher accuracy even, right? It depends on the model pool. Depends on the task on the model pool. Depends on the task on the model pool. Depends on the task at hand. But, I would encourage to think at hand. But, I would encourage to think at hand. But, I would encourage to think about the complementary nature of about the complementary nature of about the complementary nature of models. models. models. >> I see. Do you see so >> I see. Do you see so >> I see. Do you see so in kind of the way that you were in kind of the way that you were in kind of the way that you were describing the way that the task is describing the way that the task is describing the way that the task is broken up, do you see that the some of broken up, do you see that the some of broken up, do you see that the some of the smaller models because the token the smaller models because the token the smaller models because the token cost is cheaper, are they using more cost is cheaper, are they using more cost is cheaper, are they using more tokens? Like is it Are Are you tokens? Like is it Are Are you tokens? Like is it Are Are you specifically routing so that they do or specifically routing so that they do or specifically routing so that they do or that they so that they are chattier? that they so that they are chattier? that they so that they are chattier? >> Uh oh yeah, they absolutely do use more >> Uh oh yeah, they absolutely do use more >> Uh oh yeah, they absolutely do use more tokens. tokens. tokens. I I actually I want to I I actually I want to I I actually I want to kind of like riff on something that kind of like riff on something that kind of like riff on something that Tuhin was saying, which is like you Tuhin was saying, which is like you Tuhin was saying, which is like you know, a lot of times when you look at know, a lot of times when you look at know, a lot of times when you look at these different benchmarks, you'll see these different benchmarks, you'll see these different benchmarks, you'll see that the small models will perform that the small models will perform that the small models will perform better than like even the frontier better than like even the frontier better than like even the frontier models in certain cases.

models in certain cases. models in certain cases. I think a lot of people they look at I think a lot of people they look at I think a lot of people they look at this and they immediately jump to, "Oh, this and they immediately jump to, "Oh, this and they immediately jump to, "Oh, how can we just route like, you know, how can we just route like, you know, how can we just route like, you know, the task where the small models do the task where the small models do the task where the small models do better just straight to the smaller better just straight to the smaller better just straight to the smaller models?" I I think that one of the models?" I I think that one of the models?" I I think that one of the things we really want to emphasize with things we really want to emphasize with things we really want to emphasize with our recent blog post and recent Devin our recent blog post and recent Devin our recent blog post and recent Devin Fusion was that this kind of like naive Fusion was that this kind of like naive Fusion was that this kind of like naive like initial routing to based on the like initial routing to based on the like initial routing to based on the task type is extremely fragile, task type is extremely fragile, task type is extremely fragile, especially the more agentic the task you especially the more agentic the task you especially the more agentic the task you you work on is. So, for example, like a you work on is. So, for example, like a you work on is. So, for example, like a real developer, you might ask your agent real developer, you might ask your agent real developer, you might ask your agent first, "Oh, how does this code base first, "Oh, how does this code base first, "Oh, how does this code base work?" And then you you go deeper and work?" And then you you go deeper and work?" And then you you go deeper and you're like, "Okay, actually, can you you're like, "Okay, actually, can you you're like, "Okay, actually, can you implement some features for me?" And implement some features for me?" And implement some features for me?" And then you go deeper and you're like, "Oh, then you go deeper and you're like, "Oh, then you go deeper and you're like, "Oh, can you like now go to like a live test can you like now go to like a live test can you like now go to like a live test of this feature and debug deep cases?" of this feature and debug deep cases?" of this feature and debug deep cases?" The complexity changes and the type of The complexity changes and the type of The complexity changes and the type of task changes over time, and you don't task changes over time, and you don't task changes over time, and you don't want to be left with some subpar model want to be left with some subpar model want to be left with some subpar model for the the task that you're now on. I for the the task that you're now on. I for the the task that you're now on. I think this is why people like frontier think this is why people like frontier think this is why people like frontier models so much is they're they're just models so much is they're they're just models so much is they're they're just like generally intelligent and they're like generally intelligent and they're like generally intelligent and they're capable of shifting between various capable of shifting between various capable of shifting between various different domains, even if you can eke different domains, even if you can eke different domains, even if you can eke out better performance in very specific out better performance in very specific out better performance in very specific tasks. Um, and and what So, the tasks. Um, and and what So, the tasks. Um, and and what So, the challenge is, how do you get a small challenge is, how do you get a small challenge is, how do you get a small model to know that it's out of its depth model to know that it's out of its depth model to know that it's out of its depth and you need to now like go switch to and you need to now like go switch to and you need to now like go switch to another model or go like, you know, go another model or go like, you know, go another model or go like, you know, go to a smart smarter model. And and our to a smart smarter model. And and our to a smart smarter model. And and our solution to this is you kind of just solution to this is you kind of just solution to this is you kind of just always have this like main frontier always have this like main frontier always have this like main frontier agent that's watching, even if it's not agent that's watching, even if it's not agent that's watching, even if it's not the one doing the work. It should at the one doing the work. It should at the one doing the work. It should at least be keeping tabs to figure out, least be keeping tabs to figure out, least be keeping tabs to figure out, "Okay, we like the the agent I delegated "Okay, we like the the agent I delegated "Okay, we like the the agent I delegated to now is like out of its depth. I need to now is like out of its depth. I need to now is like out of its depth. I need to kind of like move it to to something to kind of like move it to to something to kind of like move it to to something else." Um, and and overall, just the else." Um, and and overall, just the else." Um, and and overall, just the guarantee of always having frontier guarantee of always having frontier guarantee of always having frontier intelligence present, I think reduces intelligence present, I think reduces intelligence present, I think reduces the the fragility of of these systems the the fragility of of these systems the the fragility of of these systems quite a lot.

quite a lot. quite a lot. >> How does the sharing of context between >> How does the sharing of context between >> How does the sharing of context between one of those smaller agents who has one of those smaller agents who has one of those smaller agents who has basically completed up to some level of basically completed up to some level of basically completed up to some level of a task and decides, "Actually, I don't a task and decides, "Actually, I don't a task and decides, "Actually, I don't think I'm the right person for this. I think I'm the right person for this. I think I'm the right person for this. I need to hand it back to the the need to hand it back to the the need to hand it back to the the foundational model." Um, of course, you foundational model." Um, of course, you foundational model." Um, of course, you don't want to have the entire trace of don't want to have the entire trace of don't want to have the entire trace of that that that >> Yeah. >> Yeah. >> Yeah. >> smaller agent be passed back to the >> smaller agent be passed back to the >> smaller agent be passed back to the larger model. But so, how do you get larger model. But so, how do you get larger model. But so, how do you get that level of specificity while that level of specificity while that level of specificity while basically providing the information it basically providing the information it basically providing the information it needs, but not more? needs, but not more? needs, but not more? >> Yeah, absolutely. So, I I I think like >> Yeah, absolutely. So, I I I think like >> Yeah, absolutely. So, I I I think like the the context here is it's actually the the context here is it's actually the the context here is it's actually very easy to actually create a system very easy to actually create a system very easy to actually create a system that's more expensive that's more expensive that's more expensive as soon as you're running like, you as soon as you're running like, you as soon as you're running like, you know, multiple models together cuz oh, know, multiple models together cuz oh, know, multiple models together cuz oh, no, like, you know, this one file no, like, you know, this one file no, like, you know, this one file reading. Now, every every one of these reading. Now, every every one of these reading. Now, every every one of these models is now reading this one file models is now reading this one file models is now reading this one file reading. So, now you're you're you're reading. So, now you're you're you're reading. So, now you're you're you're being charged like three times as much. being charged like three times as much. being charged like three times as much. Um, Um, Um, the the the the trick that we we spent a the the the the trick that we we spent a the the the the trick that we we spent a lot of our time on is um, lot of our time on is um, lot of our time on is um, most of the context by default will only most of the context by default will only most of the context by default will only be going to like one model. So, like be going to like one model. So, like be going to like one model. So, like most of the context let's say will be most of the context let's say will be most of the context let's say will be going to the small model. But, the thing going to the small model. But, the thing going to the small model. But, the thing you need to then tune very well is okay, you need to then tune very well is okay, you need to then tune very well is okay, like maybe you still show like what like maybe you still show like what like maybe you still show like what files it's reading, maybe you show like files it's reading, maybe you show like files it's reading, maybe you show like the high-level thinking of what it's the high-level thinking of what it's the high-level thinking of what it's doing back to the main model. Maybe you doing back to the main model. Maybe you doing back to the main model. Maybe you have the small model um, you tune its have the small model um, you tune its have the small model um, you tune its ability to present the context back to ability to present the context back to ability to present the context back to the main model. Um, and actually a lot the main model. Um, and actually a lot the main model. Um, and actually a lot of these problems already have been well of these problems already have been well of these problems already have been well studied in many domains already like studied in many domains already like studied in many domains already like context compaction is something you context compaction is something you context compaction is something you already have to solve if you want to do already have to solve if you want to do already have to solve if you want to do like really long-running agents. And so, like really long-running agents. And so, like really long-running agents. And so, this problem of taking long context this problem of taking long context this problem of taking long context compacting it in a way that is now compacting it in a way that is now compacting it in a way that is now understandable is the one that you can understandable is the one that you can understandable is the one that you can also apply to this domain and just kind also apply to this domain and just kind also apply to this domain and just kind of give the compacted context back to of give the compacted context back to of give the compacted context back to back to the main agent.

back to the main agent. back to the main agent. >> Context compaction is something that you >> Context compaction is something that you >> Context compaction is something that you know, I'm familiar with but I hadn't know, I'm familiar with but I hadn't know, I'm familiar with but I hadn't really thought about as you're doing really thought about as you're doing really thought about as you're doing model routing and as you're trying to model routing and as you're trying to model routing and as you're trying to share context across now potentially share context across now potentially share context across now potentially many models, you're you're expanding the many models, you're you're expanding the many models, you're you're expanding the amount of what could be seen as wasteful amount of what could be seen as wasteful amount of what could be seen as wasteful tokens or redundant tokens just just tokens or redundant tokens just just tokens or redundant tokens just just because you have to process that across because you have to process that across because you have to process that across the many models. the many models. the many models. >> Yeah. >> Yeah. >> Yeah. Yeah. Um, I I I think there's the way I Yeah. Um, I I I think there's the way I Yeah. Um, I I I think there's the way I describe it is I think we are early in describe it is I think we are early in describe it is I think we are early in in the in the model routing domain. Um, in the in the model routing domain. Um, in the in the model routing domain. Um, I I I hope that a year from now that I I I hope that a year from now that I I I hope that a year from now that even the techniques we kind of use for even the techniques we kind of use for even the techniques we kind of use for Devin Fusion you people look back on Devin Fusion you people look back on Devin Fusion you people look back on that and are like, "Oh, these are some that and are like, "Oh, these are some that and are like, "Oh, these are some like really legacy ideas and and now we like really legacy ideas and and now we like really legacy ideas and and now we have like much better methods at routing have like much better methods at routing have like much better methods at routing between models." Um, and when people between models." Um, and when people between models." Um, and when people actually start co-designing their models actually start co-designing their models actually start co-designing their models with this in mind we're going to be be with this in mind we're going to be be with this in mind we're going to be be in a much better world. in a much better world. in a much better world. >> Yeah, I I echo what you said that um, >> Yeah, I I echo what you said that um, >> Yeah, I I echo what you said that um, I think routing I think routing I think routing will evolve as the task evolves when will evolve as the task evolves when will evolve as the task evolves when when you start task, right? So, um, it's when you start task, right? So, um, it's when you start task, right? So, um, it's more useful to see things in terms of more useful to see things in terms of more useful to see things in terms of subtasks and sessions uh, than subtasks and sessions uh, than subtasks and sessions uh, than individual problems that you're trying individual problems that you're trying individual problems that you're trying to solve because more than likely you're to solve because more than likely you're to solve because more than likely you're when you're working through a problem when you're working through a problem when you're working through a problem you're asking a lot of questions, you're you're asking a lot of questions, you're you're asking a lot of questions, you're you're exploring different things. Um, you're exploring different things. Um, you're exploring different things. Um, and and and it is imperative that you try like it is imperative that you try like it is imperative that you try like people who design routers is imperative people who design routers is imperative people who design routers is imperative that they try and understand these that they try and understand these that they try and understand these phases phases phases of different complexities and then try of different complexities and then try of different complexities and then try and apply some logic for essentially and apply some logic for essentially and apply some logic for essentially side kicking side kicking side kicking tasks or

tasks or tasks or leveraging expertise from other models leveraging expertise from other models leveraging expertise from other models that's that's pretty on point. that's that's pretty on point. that's that's pretty on point. >> Yeah, I'd love to hear from the the >> Yeah, I'd love to hear from the the >> Yeah, I'd love to hear from the the router guy. router guy. router guy. >> [laughter] >> [laughter] >> [laughter] >> Yeah, um >> Yeah, um >> Yeah, um I think those are these are like I think those are these are like I think those are these are like important points and one of the biggest important points and one of the biggest important points and one of the biggest debates I think we have internally is debates I think we have internally is debates I think we have internally is whether that outer model that's doing whether that outer model that's doing whether that outer model that's doing the orchestration should be the big the orchestration should be the big the orchestration should be the big model or the small model. model or the small model. model or the small model. You get like very different results You get like very different results You get like very different results depending on your choice depending on your choice depending on your choice and in fact and it's not even clear what and in fact and it's not even clear what and in fact and it's not even clear what the pricing impact would be because if the pricing impact would be because if the pricing impact would be because if your outer model that's doing the your outer model that's doing the your outer model that's doing the orchestration is the big model it can orchestration is the big model it can orchestration is the big model it can leverage its caching to like make more leverage its caching to like make more leverage its caching to like make more of its decisions and it's caching is of its decisions and it's caching is of its decisions and it's caching is going to be like a dramatic price going to be like a dramatic price going to be like a dramatic price savings compared to the small models savings compared to the small models savings compared to the small models caching a lot of the time caching a lot of the time caching a lot of the time especially for like perform for you know especially for like perform for you know especially for like perform for you know issues that are on the bright line like issues that are on the bright line like issues that are on the bright line like zooming out a little bit I think the zooming out a little bit I think the zooming out a little bit I think the what you want from all the models out what you want from all the models out what you want from all the models out there when you do model fusion is to there when you do model fusion is to there when you do model fusion is to benefit from all the data that is being benefit from all the data that is being benefit from all the data that is being trained on across all the labs and and trained on across all the labs and and trained on across all the labs and and not just the data from one lab right or not just the data from one lab right or not just the data from one lab right or one source and a model is just like a one source and a model is just like a one source and a model is just like a combination of like the data and its combination of like the data and its combination of like the data and its understanding of the data both its understanding of the data both its understanding of the data both its compute and the quality of its RL so compute and the quality of its RL so compute and the quality of its RL so long term long term long term I think you want you want models where I think you want you want models where I think you want you want models where they know that oh this is like in they know that oh this is like in they know that oh this is like in distribution like this is in my data you distribution like this is in my data you distribution like this is in my data you can use small models pretty easily

can use small models pretty easily can use small models pretty easily and get a cost savings but if it's out and get a cost savings but if it's out and get a cost savings but if it's out of distribution small models may of distribution small models may of distribution small models may actually increase your cost because of actually increase your cost because of actually increase your cost because of how often they'll like call tools and how often they'll like call tools and how often they'll like call tools and how crazy their loops will be. Like if how crazy their loops will be. Like if how crazy their loops will be. Like if you run terminal bench on Opus and you run terminal bench on Opus and you run terminal bench on Opus and Haiku, like Opus will do about three Haiku, like Opus will do about three Haiku, like Opus will do about three times better times better times better at 1/10 the cost of Haiku, even though at 1/10 the cost of Haiku, even though at 1/10 the cost of Haiku, even though Haiku's significantly cheaper per token. Haiku's significantly cheaper per token. Haiku's significantly cheaper per token. So it really becomes a huge problem if So it really becomes a huge problem if So it really becomes a huge problem if you use a too small of a model, you use a too small of a model, you use a too small of a model, particularly on tasks that are out of particularly on tasks that are out of particularly on tasks that are out of domain for the train data. When you're domain for the train data. When you're domain for the train data. When you're doing something like classifying text, doing something like classifying text, doing something like classifying text, like hey, is this like a person's name like hey, is this like a person's name like hey, is this like a person's name or is this an organization's name? or is this an organization's name? or is this an organization's name? That's super in domain. So you would you That's super in domain. So you would you That's super in domain. So you would you don't want that kind of task to go to a don't want that kind of task to go to a don't want that kind of task to go to a large model. You want it to go to a a large model. You want it to go to a a large model. You want it to go to a a small model. Everyone has that in their small model. Everyone has that in their small model. Everyone has that in their domain. Um so deep like being able to domain. Um so deep like being able to domain. Um so deep like being able to understand in domain, out of domain is a understand in domain, out of domain is a understand in domain, out of domain is a lot of work that we're doing for for um lot of work that we're doing for for um lot of work that we're doing for for um open router fusion. open router fusion. open router fusion. And uh and then also figuring out like And uh and then also figuring out like And uh and then also figuring out like what like how to orchestrate the outer what like how to orchestrate the outer what like how to orchestrate the outer and inner models um for different types and inner models um for different types and inner models um for different types of task. And like of task. And like of task. And like um um um yeah, it's it's an early industry. It's yeah, it's it's an early industry. It's yeah, it's it's an early industry. It's like an early field of research. Uh most like an early field of research. Uh most like an early field of research. Uh most research on model fusion has not been research on model fusion has not been research on model fusion has not been very detailed, not been very like, you very detailed, not been very like, you very detailed, not been very like, you know, optimistic sometimes. Um it's only know, optimistic sometimes. Um it's only know, optimistic sometimes. Um it's only like just, you know, recently getting uh like just, you know, recently getting uh like just, you know, recently getting uh more optimistic. And I think um I'm more optimistic. And I think um I'm more optimistic. And I think um I'm personally very optimistic about it. And personally very optimistic about it. And personally very optimistic about it. And uh you know, like we we're very like uh you know, like we we're very like uh you know, like we we're very like ecosystem-driven, ecosystem-driven, ecosystem-driven, collaborative company. And a lot of our

collaborative company. And a lot of our collaborative company. And a lot of our like we work with a lot of partners to like we work with a lot of partners to like we work with a lot of partners to try like help improve their try like help improve their try like help improve their orchestration pipelines with good orchestration pipelines with good orchestration pipelines with good primitives, like the, you know, primitives, like the, you know, primitives, like the, you know, sub-agent and like the advisor tool, um sub-agent and like the advisor tool, um sub-agent and like the advisor tool, um which is kind of similar to what you which is kind of similar to what you which is kind of similar to what you were talking about. were talking about. were talking about. >> I'm curious. So uh >> I'm curious. So uh >> I'm curious. So uh help me understand when uh if a it makes help me understand when uh if a it makes help me understand when uh if a it makes total sense that um total sense that um total sense that um a small model, if it's in domain, would a small model, if it's in domain, would a small model, if it's in domain, would be cheaper. But if it's not, then it's be cheaper. But if it's not, then it's be cheaper. But if it's not, then it's going to thrash around uh as it tries to going to thrash around uh as it tries to going to thrash around uh as it tries to get an answer. get an answer. get an answer. When you're describing whether the When you're describing whether the When you're describing whether the uh uh uh con- like the main agent should be the con- like the main agent should be the con- like the main agent should be the the the local model or the cloud model? the the local model or the cloud model? the the local model or the cloud model? Does that is that a decision that's then Does that is that a decision that's then Does that is that a decision that's then dependent on whether the task is dependent on whether the task is dependent on whether the task is something that it that's in domain or something that it that's in domain or something that it that's in domain or not? not? not? Does my question kind of make sense? Does my question kind of make sense? Does my question kind of make sense? >> I don't know. Um, I basically I don't >> I don't know. Um, I basically I don't >> I don't know. Um, I basically I don't it's kind of early to it's kind of early to it's kind of early to >> um >> um >> um to say. I I think the to say. I I think the to say. I I think the I wait the results that we published, I wait the results that we published, I wait the results that we published, which are which are which are a couple weeks ago, which were focused a couple weeks ago, which were focused a couple weeks ago, which were focused on deep research, not coding. on deep research, not coding. on deep research, not coding. >> Mhm.

>> Mhm. >> Mhm. >> Um >> Um >> Um we had the smart model be the wrapper we had the smart model be the wrapper we had the smart model be the wrapper model, be the outer model, and we got model, be the outer model, and we got model, be the outer model, and we got the best results from doing that. Um, the best results from doing that. Um, the best results from doing that. Um, but it but it but it for deep research it works the best. Um, for deep research it works the best. Um, for deep research it works the best. Um, for other tasks, it's like kind of for other tasks, it's like kind of for other tasks, it's like kind of unclear. Like we there it's fusion is unclear. Like we there it's fusion is unclear. Like we there it's fusion is not super well optimized for coding. And not super well optimized for coding. And not super well optimized for coding. And uh it might be that like a smaller model uh it might be that like a smaller model uh it might be that like a smaller model ends up being like a higher um ends up being like a higher um ends up being like a higher um efficiency per you know, fewer dollars efficiency per you know, fewer dollars efficiency per you know, fewer dollars um per like completed successfully um per like completed successfully um per like completed successfully completed task, but it's kind of early completed task, but it's kind of early completed task, but it's kind of early to say. to say. to say. >> Yeah. >> Yeah. >> Yeah. >> [clears throat] >> [clears throat] >> [clears throat] >> One thing you said earlier is like, oh, >> One thing you said earlier is like, oh, >> One thing you said earlier is like, oh, you get the caching benefit from like you get the caching benefit from like you get the caching benefit from like the main line agent. Um, you actually the main line agent. Um, you actually the main line agent. Um, you actually can get the caching benefit from the can get the caching benefit from the can get the caching benefit from the side agent. And um, this is actually one side agent. And um, this is actually one side agent. And um, this is actually one of the key things we talked about with of the key things we talked about with of the key things we talked about with our Devin fusion launch is that um, you our Devin fusion launch is that um, you our Devin fusion launch is that um, you kind of are leaving a lot on the table kind of are leaving a lot on the table kind of are leaving a lot on the table if you do a main agent and sub agents if you do a main agent and sub agents if you do a main agent and sub agents type system. Um, so we don't use sub type system. Um, so we don't use sub type system. Um, so we don't use sub agents. We use what we call a sidekick, agents. We use what we call a sidekick, agents. We use what we call a sidekick, which is um, one sub agent that which is um, one sub agent that which is um, one sub agent that continually has a running context. So continually has a running context. So continually has a running context. So the main agent doesn't need to the main agent doesn't need to the main agent doesn't need to re-provide uh, context from earlier. Um, re-provide uh, context from earlier. Um, re-provide uh, context from earlier. Um, it's all still in the KB cache, right?

it's all still in the KB cache, right? it's all still in the KB cache, right? Like it's 10 times cheaper on on all Like it's 10 times cheaper on on all Like it's 10 times cheaper on on all those cache tokens. Um, and then if you those cache tokens. Um, and then if you those cache tokens. Um, and then if you want to like switch the smart model to want to like switch the smart model to want to like switch the smart model to be like the one on the side or the one be like the one on the side or the one be like the one on the side or the one in charge, um it's actually totally in charge, um it's actually totally in charge, um it's actually totally fine, and you can kind of like do do the fine, and you can kind of like do do the fine, and you can kind of like do do the swapping back and forth. swapping back and forth. swapping back and forth. We're also spending a lot of time right We're also spending a lot of time right We're also spending a lot of time right now thinking about now thinking about now thinking about how do you train models to actually work how do you train models to actually work how do you train models to actually work collaboratively with other models? I collaboratively with other models? I collaboratively with other models? I actually I think there's a lot of actually I think there's a lot of actually I think there's a lot of literature out there on how you RL one literature out there on how you RL one literature out there on how you RL one model to do a task end-to-end. How can model to do a task end-to-end. How can model to do a task end-to-end. How can you RL model to also be good at you RL model to also be good at you RL model to also be good at collaboration? collaboration? collaboration? And when we think about it, we actually And when we think about it, we actually And when we think about it, we actually try both of these setups where let's RL try both of these setups where let's RL try both of these setups where let's RL the model being the orchestrator and the the model being the orchestrator and the the model being the orchestrator and the one deciding what gets delegated to one deciding what gets delegated to one deciding what gets delegated to other models, see how well that other models, see how well that other models, see how well that performs. Um and we also performs. Um and we also performs. Um and we also orchestrate it in a way where the model orchestrate it in a way where the model orchestrate it in a way where the model we're training is actually the executor, we're training is actually the executor, we're training is actually the executor, the sidekick, and see how well is it at the sidekick, and see how well is it at the sidekick, and see how well is it at executing other models' instructions. Um executing other models' instructions. Um executing other models' instructions. Um and we expect that to to be a probably a and we expect that to to be a probably a and we expect that to to be a probably a big lift in this next step of like big lift in this next step of like big lift in this next step of like multi-model orchestration is don't just multi-model orchestration is don't just multi-model orchestration is don't just like take models as they are and like take models as they are and like take models as they are and orchestrate them, but like can you orchestrate them, but like can you orchestrate them, but like can you actually co-design your models with the actually co-design your models with the actually co-design your models with the orchestration system? orchestration system? orchestration system? >> Yeah, that makes sense. I mean, with >> Yeah, that makes sense. I mean, with >> Yeah, that makes sense. I mean, with Nemo Tron and you know, with all the Nemo Tron and you know, with all the Nemo Tron and you know, with all the foundational models where um we're foundational models where um we're foundational models where um we're essentially post-training them for the essentially post-training them for the essentially post-training them for the harnesses that they're getting used. If harnesses that they're getting used. If harnesses that they're getting used. If the harness is going to include a lot of the harness is going to include a lot of the harness is going to include a lot of routing, then that makes sense that routing, then that makes sense that routing, then that makes sense that makes its way in to the post-training.

makes its way in to the post-training. makes its way in to the post-training. >> Yeah. Um are you guys thinking a lot >> Yeah. Um are you guys thinking a lot >> Yeah. Um are you guys thinking a lot about the kind of like the model about the kind of like the model about the kind of like the model training at at Nvidia for this kind of training at at Nvidia for this kind of training at at Nvidia for this kind of purposes? purposes? purposes? >> Yeah, so we have a technology called >> The mic's go? >> The mic's go? >> Right. >> Right. >> Right. >> Hello. >> Hello. >> Hello. >> Testing. Oh, this one works. >> Testing. Oh, this one works. >> Testing. Oh, this one works. >> [laughter] >> [laughter] >> [laughter] >> Okay. Uh so we have a technology called >> Okay. Uh so we have a technology called >> Okay. Uh so we have a technology called Flex Run. So you have uh we we have a Flex Run. So you have uh we we have a Flex Run. So you have uh we we have a setup where uh there's a the uh there's setup where uh there's a the uh there's setup where uh there's a the uh there's a main model, then we distill it into a main model, then we distill it into a main model, then we distill it into smaller uh footprints. And then based on smaller uh footprints. And then based on smaller uh footprints. And then based on the based on the task at hand, you can the based on the task at hand, you can the based on the task at hand, you can switch which model does the decoding. switch which model does the decoding. switch which model does the decoding. Right? So there's a lot of fancy stuff Right? So there's a lot of fancy stuff Right? So there's a lot of fancy stuff you can do uh within a model artifact, you can do uh within a model artifact, you can do uh within a model artifact, too. too. too. Uh Uh Uh to essentially to essentially to essentially only activate only activate only activate a class of model or a section of a class of model or a section of a class of model or a section of weights, depending on the task at hand weights, depending on the task at hand weights, depending on the task at hand or the complexity at hand. or the complexity at hand. or the complexity at hand. Most In most cases, you can essentially Most In most cases, you can essentially Most In most cases, you can essentially understand the novelty of a question to understand the novelty of a question to understand the novelty of a question to a model if you have access to the recipe a model if you have access to the recipe a model if you have access to the recipe with which it was with which it was with which it was trained. So, this works very well for trained. So, this works very well for trained. So, this works very well for open models, right? Like or any model open models, right? Like or any model open models, right? Like or any model you have access to its data for, right?

you have access to its data for, right? you have access to its data for, right? Because you can literally decide if it's Because you can literally decide if it's Because you can literally decide if it's in like see if it's in distribution or in like see if it's in distribution or in like see if it's in distribution or not. Uh not. Uh not. Uh again, if you have studies from when it again, if you have studies from when it again, if you have studies from when it was trained, you can also see how much was trained, you can also see how much was trained, you can also see how much essentially how much was your essentially how much was your essentially how much was your distillation gap across teachers and the distillation gap across teachers and the distillation gap across teachers and the artifact that you trained, right? artifact that you trained, right? artifact that you trained, right? Because sure, you have domain data from Because sure, you have domain data from Because sure, you have domain data from all all the different domains you're all all the different domains you're all all the different domains you're tuning, but it's not guaranteed that it tuning, but it's not guaranteed that it tuning, but it's not guaranteed that it uh absorbed all the that data evenly uh absorbed all the that data evenly uh absorbed all the that data evenly across the models, right? So, um it across the models, right? So, um it across the models, right? So, um it becomes it becomes very interesting uh becomes it becomes very interesting uh becomes it becomes very interesting uh to start thinking about these flexible to start thinking about these flexible to start thinking about these flexible weights and flexible model sizes weights and flexible model sizes weights and flexible model sizes essentially. essentially. essentially. There's also I wanted to add with the There's also I wanted to add with the There's also I wanted to add with the context piece, right? So, context piece, right? So, context piece, right? So, how do you think about how do you think about how do you think about ASTs and and context compression ASTs and and context compression ASTs and and context compression representations? representations? representations? Uh compaction in its very nature is Uh compaction in its very nature is Uh compaction in its very nature is lossy, right? So, just like headroom is lossy, right? So, just like headroom is lossy, right? So, just like headroom is there, RTK is there, right? Uh these there, RTK is there, right? Uh these there, RTK is there, right? Uh these these spaces code bases are usually these spaces code bases are usually these spaces code bases are usually designed to have representations that we designed to have representations that we designed to have representations that we carry forward through life and you carry forward through life and you carry forward through life and you essentially give models the capability essentially give models the capability essentially give models the capability to further expand on them. It's it's to further expand on them. It's it's to further expand on them. It's it's more more like loss less-ish more more like loss less-ish more more like loss less-ish compression, which can retain states of compression, which can retain states of compression, which can retain states of a models or states of agents. What do a models or states of agents. What do a models or states of agents. What do you think about that?

you think about that? you think about that? >> Yeah, I think this gets to like kind of >> Yeah, I think this gets to like kind of >> Yeah, I think this gets to like kind of like a fundamental philosophy of how like a fundamental philosophy of how like a fundamental philosophy of how agents and context should work. One agents and context should work. One agents and context should work. One exercise I like to do is like you know, exercise I like to do is like you know, exercise I like to do is like you know, as a human, like how many numbers can as a human, like how many numbers can as a human, like how many numbers can you like if I just start spitting out you like if I just start spitting out you like if I just start spitting out numbers now, right? Like how many can numbers now, right? Like how many can numbers now, right? Like how many can you remember before, you know, like you you remember before, you know, like you you remember before, you know, like you start losing track of them. I think it's start losing track of them. I think it's start losing track of them. I think it's actually very few, right? So, in some actually very few, right? So, in some actually very few, right? So, in some ways you could argue that your context ways you could argue that your context ways you could argue that your context window is actually shorter than these window is actually shorter than these window is actually shorter than these language models. And yet, you can language models. And yet, you can language models. And yet, you can actually be very effective at that, actually be very effective at that, actually be very effective at that, right? Your context is is very lossy. right? Your context is is very lossy. right? Your context is is very lossy. Um, I think one of the nice things that Um, I think one of the nice things that Um, I think one of the nice things that people are starting to realize with people are starting to realize with people are starting to realize with agents is like you have a lot of agents is like you have a lot of agents is like you have a lot of non-lossy systems that you can fall back non-lossy systems that you can fall back non-lossy systems that you can fall back to. So, you have a file system. Like uh to. So, you have a file system. Like uh to. So, you have a file system. Like uh if in your memory all your memory is if in your memory all your memory is if in your memory all your memory is that you read some file earlier, you that you read some file earlier, you that you read some file earlier, you don't need to remember the whole file. don't need to remember the whole file. don't need to remember the whole file. You maybe remember the important parts, You maybe remember the important parts, You maybe remember the important parts, but you can still have the full version but you can still have the full version but you can still have the full version of the file on your system. And that's of the file on your system. And that's of the file on your system. And that's kind of my goal when I'm thinking about kind of my goal when I'm thinking about kind of my goal when I'm thinking about how do we build a good context how do we build a good context how do we build a good context engineered harness is the harness should engineered harness is the harness should engineered harness is the harness should have everything in needs to find what it have everything in needs to find what it have everything in needs to find what it needs to have even if it doesn't have needs to have even if it doesn't have needs to have even if it doesn't have everything immediately available. everything immediately available. everything immediately available. >> In that case, uh do you think that the >> In that case, uh do you think that the >> In that case, uh do you think that the context sharing problem will become context sharing problem will become context sharing problem will become cheaper and less of a problem in future?

cheaper and less of a problem in future? cheaper and less of a problem in future? >> Yeah, it's definitely possible as well. >> Yeah, it's definitely possible as well. >> Yeah, it's definitely possible as well. I've seen cases where um you know, the I've seen cases where um you know, the I've seen cases where um you know, the psychic agent does a bunch of work. It psychic agent does a bunch of work. It psychic agent does a bunch of work. It tells the main model, "Oh yeah, like tells the main model, "Oh yeah, like tells the main model, "Oh yeah, like here's here's all the things I found." here's here's all the things I found." here's here's all the things I found." Instead of dumping the full thing, it Instead of dumping the full thing, it Instead of dumping the full thing, it just references them by file. And then just references them by file. And then just references them by file. And then the main model is actually generally you the main model is actually generally you the main model is actually generally you find these larger, smarter models, find these larger, smarter models, find these larger, smarter models, they're actually more token efficient they're actually more token efficient they're actually more token efficient with how they use tools and how they with how they use tools and how they with how they use tools and how they read. And so, they actually read the read. And so, they actually read the read. And so, they actually read the files in a way where they only see the files in a way where they only see the files in a way where they only see the important parts, right? Or they decide important parts, right? Or they decide important parts, right? Or they decide that, "Oh, actually I only need to look that, "Oh, actually I only need to look that, "Oh, actually I only need to look at a subset of this." Or, "Oh, I can run at a subset of this." Or, "Oh, I can run at a subset of this." Or, "Oh, I can run a single command and just know if a single command and just know if a single command and just know if everything is done properly." Um, it's everything is done properly." Um, it's everything is done properly." Um, it's actually quite amazing um actually quite amazing um actually quite amazing um the fact that you know, these the fact that you know, these the fact that you know, these multi-model systems they actually seem multi-model systems they actually seem multi-model systems they actually seem to scale and get better with to scale and get better with to scale and get better with intelligence, which is um not something intelligence, which is um not something intelligence, which is um not something we should just take for granted, right? we should just take for granted, right? we should just take for granted, right? It's not obvious that actually more It's not obvious that actually more It's not obvious that actually more expensive models are actually creating expensive models are actually creating expensive models are actually creating an overall cheaper system. an overall cheaper system. an overall cheaper system. >> Yeah, like >> Yeah, like >> Yeah, like the scaling laws uh if you have a larger the scaling laws uh if you have a larger the scaling laws uh if you have a larger model, it's going to be more efficient model, it's going to be more efficient model, it's going to be more efficient with its tokens. Uh smaller models less with its tokens. Uh smaller models less with its tokens. Uh smaller models less efficient with its tokens. efficient with its tokens. efficient with its tokens. I guess I had a question for you, Alex.

I guess I had a question for you, Alex. I guess I had a question for you, Alex. Uh do you guys uh like Uh do you guys uh like Uh do you guys uh like weigh more importance on uh the actual weigh more importance on uh the actual weigh more importance on uh the actual infrastructure side of routing? So, for infrastructure side of routing? So, for infrastructure side of routing? So, for instance, KV cache aware routing or uh instance, KV cache aware routing or uh instance, KV cache aware routing or uh like this where most of the businesses like this where most of the businesses like this where most of the businesses right now Uh or the are you seeing right now Uh or the are you seeing right now Uh or the are you seeing strong pull off like people actually strong pull off like people actually strong pull off like people actually deploying routers in production? >> So open router >> So open router is a like a marketplace for language is a like a marketplace for language is a like a marketplace for language models. We we exist at models. We we exist at models. We we exist at like we can't see into the KV caches of like we can't see into the KV caches of like we can't see into the KV caches of models unless we're running them models unless we're running them models unless we're running them ourselves, which is pretty rare. ourselves, which is pretty rare. ourselves, which is pretty rare. Um we Um we Um we we we do spend a lot of time optimizing we we do spend a lot of time optimizing we we do spend a lot of time optimizing for cash hits and we we like pass for cash hits and we we like pass for cash hits and we we like pass through cash hits directly to users. through cash hits directly to users. through cash hits directly to users. Um but like in terms of KV cash Um but like in terms of KV cash Um but like in terms of KV cash optimizations we we we can't do any of optimizations we we we can't do any of optimizations we we we can't do any of like specific work there. What we do for like specific work there. What we do for like specific work there. What we do for um model routing is we we try to like um model routing is we we try to like um model routing is we we try to like find the best model or best combination find the best model or best combination find the best model or best combination of models for the prompt and then when of models for the prompt and then when of models for the prompt and then when we see a cash hit, we will will like use we see a cash hit, we will will like use we see a cash hit, we will will like use up the duration of the cash and send the up the duration of the cash and send the up the duration of the cash and send the downstream customer like the the full downstream customer like the the full downstream customer like the the full savings of the cash hit.

savings of the cash hit. savings of the cash hit. Um Um Um there's more work that we can do here there's more work that we can do here there's more work that we can do here where we could say, okay, this looks where we could say, okay, this looks where we could say, okay, this looks like something where there's significant like something where there's significant like something where there's significant benefit to switching the model right benefit to switching the model right benefit to switching the model right now, but you haven't used up the full now, but you haven't used up the full now, but you haven't used up the full cash. You still have like 2 minutes cash. You still have like 2 minutes cash. You still have like 2 minutes left. Um and it we think it's probably left. Um and it we think it's probably left. Um and it we think it's probably worth switching the model and losing the worth switching the model and losing the worth switching the model and losing the rest of your cash um and letting people rest of your cash um and letting people rest of your cash um and letting people kind of like tweak their tolerance for kind of like tweak their tolerance for kind of like tweak their tolerance for that behavior. Um we've we've been doing that behavior. Um we've we've been doing that behavior. Um we've we've been doing a little bit of that, but we haven't a little bit of that, but we haven't a little bit of that, but we haven't like exposed it to customers yet. like exposed it to customers yet. like exposed it to customers yet. >> [clears throat] >> [clears throat] >> [clears throat] >> Awesome. >> Awesome. >> Awesome. >> What what is next for you guys in terms >> What what is next for you guys in terms >> What what is next for you guys in terms of your your your model routing? Cuz as of your your your model routing? Cuz as of your your your model routing? Cuz as you mentioned, you know, you mentioned, you know, you mentioned, you know, it is kind of like a different direction it is kind of like a different direction it is kind of like a different direction from like the the marketplace business from like the the marketplace business from like the the marketplace business that exists today. So I'd love to hear. that exists today. So I'd love to hear. that exists today. So I'd love to hear. >> So I mean we've been doing like we we've >> So I mean we've been doing like we we've >> So I mean we've been doing like we we've had a an auto router for like 2 years had a an auto router for like 2 years had a an auto router for like 2 years almost. Um but when we launched it, almost. Um but when we launched it, almost. Um but when we launched it, there was like no adoption of it. It was there was like no adoption of it. It was there was like no adoption of it. It was people really wanted to use specific people really wanted to use specific people really wanted to use specific models. models. models. Um and the the auto router just had like Um and the the auto router just had like Um and the the auto router just had like no real usage. It was it we mostly saw no real usage. It was it we mostly saw no real usage. It was it we mostly saw it as like a discovery play point. Like, it as like a discovery play point. Like, it as like a discovery play point. Like, hey, this is how you discover which hey, this is how you discover which hey, this is how you discover which model might be good for your prompt. And model might be good for your prompt. And model might be good for your prompt. And then then then um at around like um at around like um at around like January this year with open claw, it January this year with open claw, it January this year with open claw, it exploded. And the reason it exploded is exploded. And the reason it exploded is exploded. And the reason it exploded is because there's this fundamental because there's this fundamental because there's this fundamental um idiosyncrasy in open claw where it um idiosyncrasy in open claw where it um idiosyncrasy in open claw where it sends heartbeats every like 10 minutes sends heartbeats every like 10 minutes sends heartbeats every like 10 minutes to your model of choice just to see like to your model of choice just to see like to your model of choice just to see like if the you know, if the the the client

if the you know, if the the the client if the you know, if the the the client was still active. was still active. was still active. >> [laughter] >> [laughter] >> [laughter] >> And that means that if you set Opus to >> And that means that if you set Opus to >> And that means that if you set Opus to be your default model, it would be like be your default model, it would be like be your default model, it would be like using a lot of tokens on this like using a lot of tokens on this like using a lot of tokens on this like heartbeat process. And so heartbeat process. And so heartbeat process. And so um this was the very beginning of like a um this was the very beginning of like a um this was the very beginning of like a very popular app with two completely very popular app with two completely very popular app with two completely different intelligence needs, completely different intelligence needs, completely different intelligence needs, completely different. And the models the open different. And the models the open different. And the models the open source models have improved to a point source models have improved to a point source models have improved to a point where like it makes sense to segment the where like it makes sense to segment the where like it makes sense to segment the market and at at least those two areas. market and at at least those two areas. market and at at least those two areas. And so that's that was how it got And so that's that was how it got And so that's that was how it got started. And then we saw a lot more started. And then we saw a lot more started. And then we saw a lot more segmentation blossom afterwards. And so segmentation blossom afterwards. And so segmentation blossom afterwards. And so um and now a whole bunch of agents and um and now a whole bunch of agents and um and now a whole bunch of agents and apps on open router use the the apps on open router use the the apps on open router use the the different routers that we have. We have different routers that we have. We have different routers that we have. We have a couple of them. We have Pareto code a couple of them. We have Pareto code a couple of them. We have Pareto code which gives you the Pareto Pareto which gives you the Pareto Pareto which gives you the Pareto Pareto optimal model for coding tasks given a optimal model for coding tasks given a optimal model for coding tasks given a certain like threshold that you can certain like threshold that you can certain like threshold that you can tune. tune. tune. Um we have fusion which orchestrates Um we have fusion which orchestrates Um we have fusion which orchestrates multiple models and gives you a fused multiple models and gives you a fused multiple models and gives you a fused result. And um and then we'll have other result. And um and then we'll have other result. And um and then we'll have other experiments in the future. What we want experiments in the future. What we want experiments in the future. What we want to do is basically like create good to do is basically like create good to do is basically like create good primitives primitives primitives that developers can use to like get that developers can use to like get that developers can use to like get really advanced with how they use model really advanced with how they use model really advanced with how they use model orchestration.

orchestration. orchestration. Um Um Um kind of like sidecar sort of like that. kind of like sidecar sort of like that. kind of like sidecar sort of like that. Um but also give people like a really Um but also give people like a really Um but also give people like a really easy thing that they can just like set a easy thing that they can just like set a easy thing that they can just like set a slug to that works with all harnesses slug to that works with all harnesses slug to that works with all harnesses and just like gets the job done. and just like gets the job done. and just like gets the job done. >> I feel like it's super interesting how >> I feel like it's super interesting how >> I feel like it's super interesting how much of a perfect storm there is for much of a perfect storm there is for much of a perfect storm there is for model routing right now because on one model routing right now because on one model routing right now because on one hand, you know, ignore agents, ignore hand, you know, ignore agents, ignore hand, you know, ignore agents, ignore open claw for a second, like just to open claw for a second, like just to open claw for a second, like just to squeeze better performance, it seems squeeze better performance, it seems squeeze better performance, it seems like we should be smarter about how we like we should be smarter about how we like we should be smarter about how we tackle problems. That's obvious, right? tackle problems. That's obvious, right? tackle problems. That's obvious, right? If you make a plan, if you make a If you make a plan, if you make a If you make a plan, if you make a strategy, that's a that's a better way strategy, that's a that's a better way strategy, that's a that's a better way to to go about your day. So, I'm not to to go about your day. So, I'm not to to go about your day. So, I'm not surprised that you're going to see, you surprised that you're going to see, you surprised that you're going to see, you know, better code get written or more know, better code get written or more know, better code get written or more performing code get written and less performing code get written and less performing code get written and less buggy code get written if you break the buggy code get written if you break the buggy code get written if you break the problem down. And so, routing problem down. And so, routing problem down. And so, routing specifically for that uh use case makes specifically for that uh use case makes specifically for that uh use case makes a ton of sense. But then hearing this, a ton of sense. But then hearing this, a ton of sense. But then hearing this, yeah, like the profile of workloads yeah, like the profile of workloads yeah, like the profile of workloads changed with agents, right? They're very changed with agents, right? They're very changed with agents, right? They're very they it went from like I ask questions, they it went from like I ask questions, they it went from like I ask questions, I get a response, then it went to I get a response, then it went to I get a response, then it went to reasoning where I ask questions, it reasoning where I ask questions, it reasoning where I ask questions, it reasons, and then it comes back, and reasons, and then it comes back, and reasons, and then it comes back, and then it went to uh yeah, this heartbeat, then it went to uh yeah, this heartbeat, then it went to uh yeah, this heartbeat, right? Like if um if my agents are right? Like if um if my agents are right? Like if um if my agents are running optimally, uh there's a token running optimally, uh there's a token running optimally, uh there's a token being being being generated every second. Um and suddenly generated every second. Um and suddenly generated every second. Um and suddenly that is its own need for model routing, that is its own need for model routing, that is its own need for model routing, and it feels like um hearing the and it feels like um hearing the and it feels like um hearing the different solutions uh to kind of tackle different solutions uh to kind of tackle different solutions uh to kind of tackle each of those is very interesting. Even each of those is very interesting. Even each of those is very interesting. Even >> Well, I was just going to say I think >> Well, I was just going to say I think >> Well, I was just going to say I think that yeah, the the use cases for model that yeah, the the use cases for model that yeah, the the use cases for model routing are are there are many of them.

routing are are there are many of them. routing are are there are many of them. And so, one could be getting a better And so, one could be getting a better And so, one could be getting a better answer, one could be, you know, saving answer, one could be, you know, saving answer, one could be, you know, saving money and trying to get the same answer. money and trying to get the same answer. money and trying to get the same answer. One that we haven't even talked about One that we haven't even talked about One that we haven't even talked about yet, which is probably the most relevant yet, which is probably the most relevant yet, which is probably the most relevant maybe even to this crowd, is uh when do maybe even to this crowd, is uh when do maybe even to this crowd, is uh when do you want to actually run a model locally you want to actually run a model locally you want to actually run a model locally versus when do you actually need versus when do you actually need versus when do you actually need something like a frontier model to to something like a frontier model to to something like a frontier model to to perform that task. And that might be perform that task. And that might be perform that task. And that might be something to the effect of like uh for something to the effect of like uh for something to the effect of like uh for privacy protecting information. You privacy protecting information. You privacy protecting information. You know, when I'm running some local, it know, when I'm running some local, it know, when I'm running some local, it can you detect that my prompt has can you detect that my prompt has can you detect that my prompt has sensitive information? And if so, do sensitive information? And if so, do sensitive information? And if so, do that on on device, um but then maybe that on on device, um but then maybe that on on device, um but then maybe even anonymize some of that information even anonymize some of that information even anonymize some of that information to go do the more advanced workloads up to go do the more advanced workloads up to go do the more advanced workloads up on top of that information in the cloud. on top of that information in the cloud. on top of that information in the cloud. Um another example would be again for Um another example would be again for Um another example would be again for the cost savings, but it's like, "Hey, I the cost savings, but it's like, "Hey, I the cost savings, but it's like, "Hey, I bought this DGX Spark and I'm not I know bought this DGX Spark and I'm not I know bought this DGX Spark and I'm not I know I'm not at 100% utilization. How can I I'm not at 100% utilization. How can I I'm not at 100% utilization. How can I make sure that as part of my workloads, make sure that as part of my workloads, make sure that as part of my workloads, whether it's the heartbeat and open claw whether it's the heartbeat and open claw whether it's the heartbeat and open claw or what have you, that I'm leveraging or what have you, that I'm leveraging or what have you, that I'm leveraging that compute um to the fullest of its that compute um to the fullest of its that compute um to the fullest of its ability because I'm only paying for the ability because I'm only paying for the ability because I'm only paying for the electrons that are coming in for my electrons that are coming in for my electrons that are coming in for my power bill, um but I'm paying full price power bill, um but I'm paying full price power bill, um but I'm paying full price for the tokens in the cloud?" And I for the tokens in the cloud?" And I for the tokens in the cloud?" And I think that's a whole other area of model think that's a whole other area of model think that's a whole other area of model routing that I know that we're doing routing that I know that we're doing routing that I know that we're doing some work with at NVIDIA um that I think some work with at NVIDIA um that I think some work with at NVIDIA um that I think will be will be really cool as it starts will be will be really cool as it starts will be will be really cool as it starts to uh the hybrid of local and cloud to uh the hybrid of local and cloud to uh the hybrid of local and cloud starts to to really emerge as its own starts to to really emerge as its own starts to to really emerge as its own sector.

sector. sector. >> Yeah, I'd be curious to know what you >> Yeah, I'd be curious to know what you >> Yeah, I'd be curious to know what you guys take uh take is on. Like if you guys take uh take is on. Like if you guys take uh take is on. Like if you self-host a model, the cost dynamics self-host a model, the cost dynamics self-host a model, the cost dynamics change, right? You have a considerably change, right? You have a considerably change, right? You have a considerably higher cost at a higher context length higher cost at a higher context length higher cost at a higher context length because your throughput slows down as because your throughput slows down as because your throughput slows down as the context gets deeper. So, rather than the context gets deeper. So, rather than the context gets deeper. So, rather than switching to a cheaper model, even if switching to a cheaper model, even if switching to a cheaper model, even if you have self-hosted models in data you have self-hosted models in data you have self-hosted models in data center, you can use compaction uh to center, you can use compaction uh to center, you can use compaction uh to bring your throughput back up. Uh have bring your throughput back up. Uh have bring your throughput back up. Uh have you guys uh thought about this part of you guys uh thought about this part of you guys uh thought about this part of it? Like compaction versus it? Like compaction versus it? Like compaction versus uh just routing because one is you have uh just routing because one is you have uh just routing because one is you have fewer tokens to work with. One is we fewer tokens to work with. One is we fewer tokens to work with. One is we have cheaper tokens. have cheaper tokens. have cheaper tokens. >> Yeah, I I think in practice um >> Yeah, I I think in practice um >> Yeah, I I think in practice um compacting alone doesn't solve the the compacting alone doesn't solve the the compacting alone doesn't solve the the cost or or throughput problems because a cost or or throughput problems because a cost or or throughput problems because a lot of times it's just like the lot of times it's just like the lot of times it's just like the uh the differential in like model uh the differential in like model uh the differential in like model intelligence and cost is just so big intelligence and cost is just so big intelligence and cost is just so big where also by the way, when you compact, where also by the way, when you compact, where also by the way, when you compact, you're taking a cache miss. So, you're you're taking a cache miss. So, you're you're taking a cache miss. So, you're actually then now like paying 10 times actually then now like paying 10 times actually then now like paying 10 times as much for the for those input tokens as much for the for those input tokens as much for the for those input tokens if you didn't compact. Um the main if you didn't compact. Um the main if you didn't compact. Um the main reason we compact is actually reason we compact is actually reason we compact is actually intelligence. Um intelligence. Um intelligence. Um all all these like multipliers they all all these like multipliers they all all these like multipliers they advertise some like insane context advertise some like insane context advertise some like insane context window, like a million tokens. Uh I window, like a million tokens. Uh I window, like a million tokens. Uh I would like never recommend using like would like never recommend using like would like never recommend using like these models past like 200K tokens, these models past like 200K tokens, these models past like 200K tokens, under 100K if you can. Um the the the under 100K if you can. Um the the the under 100K if you can. Um the the the intelligence uh just kind of like falls intelligence uh just kind of like falls intelligence uh just kind of like falls off a cliff at some point. Um sorry, off a cliff at some point. Um sorry, off a cliff at some point. Um sorry, Anthropic if you're if you're watching.

Anthropic if you're if you're watching. Anthropic if you're if you're watching. But I I think that uh you know, But I I think that uh you know, But I I think that uh you know, compaction is like a very useful tool if compaction is like a very useful tool if compaction is like a very useful tool if you are going to have to take a cache you are going to have to take a cache you are going to have to take a cache miss anyway um one way or another, like miss anyway um one way or another, like miss anyway um one way or another, like when you're routing to another model and when you're routing to another model and when you're routing to another model and you want to just like minimize the the you want to just like minimize the the you want to just like minimize the the window there. window there. window there. >> Do you find that, um, in the sidecar, >> Do you find that, um, in the sidecar, >> Do you find that, um, in the sidecar, >> Yeah. >> Yeah. >> Yeah. >> um, when small models are generating >> um, when small models are generating >> um, when small models are generating lots of tokens, lots of tokens, lots of tokens, um, um, um, is that like an is that like one of the is that like an is that like one of the is that like an is that like one of the best reasons to switch it to a larger best reasons to switch it to a larger best reasons to switch it to a larger model? Like, basically, when when small model? Like, basically, when when small model? Like, basically, when when small models generate lots of tokens, I wonder models generate lots of tokens, I wonder models generate lots of tokens, I wonder if that's like a if that's like a if that's like a box of root the root cause of of box of root the root cause of of box of root the root cause of of intelligence problems down the road. You intelligence problems down the road. You intelligence problems down the road. You want your big model to generate the big want your big model to generate the big want your big model to generate the big token chunks, the small models to token chunks, the small models to token chunks, the small models to generate smaller token chunks, right? generate smaller token chunks, right? generate smaller token chunks, right? >> that question, you mentioned like a >> that question, you mentioned like a >> that question, you mentioned like a small model essentially needing to flag small model essentially needing to flag small model essentially needing to flag that it needs help from the larger that it needs help from the larger that it needs help from the larger model. What is that mechanism? Cuz that model. What is that mechanism? Cuz that model. What is that mechanism? Cuz that seems like the what's the indicator and seems like the what's the indicator and seems like the what's the indicator and then what's the mechanism for it to do then what's the mechanism for it to do then what's the mechanism for it to do so? so? so? >> Yeah, totally. Uh, so there are a lot of >> Yeah, totally. Uh, so there are a lot of >> Yeah, totally. Uh, so there are a lot of mechanisms we we talk about in our blog mechanisms we we talk about in our blog mechanisms we we talk about in our blog post about how uh we just detect that we post about how uh we just detect that we post about how uh we just detect that we need to like change the model up. Um, need to like change the model up. Um, need to like change the model up. Um, to to uh I guess to answer your question to to uh I guess to answer your question to to uh I guess to answer your question first, how does the the small model first, how does the the small model first, how does the the small model detect? Um, actually, the the thing that detect? Um, actually, the the thing that detect? Um, actually, the the thing that we spend a lot of time on is, um, how do we spend a lot of time on is, um, how do we spend a lot of time on is, um, how do we make sure the small model is like we make sure the small model is like we make sure the small model is like good at detecting it. Unfortunately, good at detecting it. Unfortunately, good at detecting it. Unfortunately, there's a lot of cases where you do need there's a lot of cases where you do need there's a lot of cases where you do need the big model to detect it. Um, one the big model to detect it. Um, one the big model to detect it. Um, one thing that we could don't go into the thing that we could don't go into the thing that we could don't go into the blog post is, um, you have some kind of blog post is, um, you have some kind of blog post is, um, you have some kind of cadence on which you're refreshing the cadence on which you're refreshing the cadence on which you're refreshing the cache anyways, cuz by default there's cache anyways, cuz by default there's cache anyways, cuz by default there's some like 5-minute lifetime on these some like 5-minute lifetime on these some like 5-minute lifetime on these caches. If you're going to go refresh a caches. If you're going to go refresh a caches. If you're going to go refresh a cache anyways, you basically can get a

cache anyways, you basically can get a cache anyways, you basically can get a free like big like frontier model call free like big like frontier model call free like big like frontier model call uh if you kind of like ask the right uh if you kind of like ask the right uh if you kind of like ask the right question. So, it's at that point where question. So, it's at that point where question. So, it's at that point where you might say, "Hey, just take a look at you might say, "Hey, just take a look at you might say, "Hey, just take a look at what the small model is doing. Does it what the small model is doing. Does it what the small model is doing. Does it feel like it's kind of like going into feel like it's kind of like going into feel like it's kind of like going into some rabbit hole and and need some help some rabbit hole and and need some help some rabbit hole and and need some help now?" Um, now?" Um, now?" Um, >> What's the need for the 5-minute >> What's the need for the 5-minute >> What's the need for the 5-minute refresh? refresh? refresh? >> Uh, it's just like a practical like you >> Uh, it's just like a practical like you >> Uh, it's just like a practical like you you you have to pay some kind of like you you have to pay some kind of like you you have to pay some kind of like cost to like keep these like KV cache cost to like keep these like KV cache cost to like keep these like KV cache caches warm. Um, and so most caches just caches warm. Um, and so most caches just caches warm. Um, and so most caches just get evicted on some kind of cadence. get evicted on some kind of cadence. get evicted on some kind of cadence. >> works is like at inference time you only >> works is like at inference time you only >> works is like at inference time you only have so many so many cache you can keep have so many so many cache you can keep have so many so many cache you can keep kept loaded in the GPU. So, once if a kept loaded in the GPU. So, once if a kept loaded in the GPU. So, once if a cache is not being used again and again, cache is not being used again and again, cache is not being used again and again, it's offloaded. it's offloaded. it's offloaded. >> I see. >> I see. >> I see. >> So, it's lost essentially. So, that's >> So, it's lost essentially. So, that's >> So, it's lost essentially. So, that's why the inference provider asks you for why the inference provider asks you for why the inference provider asks you for money. Uh but if you self-hosted, you money. Uh but if you self-hosted, you money. Uh but if you self-hosted, you can get around this problem. You can you can get around this problem. You can you can get around this problem. You can you can make it as long as you want based on can make it as long as you want based on can make it as long as you want based on your big business logic. your big business logic. your big business logic. >> But do do you see a world where we'll >> But do do you see a world where we'll >> But do do you see a world where we'll have like much more dynamic cache have like much more dynamic cache have like much more dynamic cache durations rather than just the 5-minute, durations rather than just the 5-minute, durations rather than just the 5-minute, 1-hour? 1-hour? 1-hour? >> It depends on who's do who's deploying >> It depends on who's do who's deploying >> It depends on who's do who's deploying the model where, right? So, if you have the model where, right? So, if you have the model where, right? So, if you have a GPU which has a GPU which has a GPU which has like a lot of memory uh which like the like a lot of memory uh which like the like a lot of memory uh which like the ratio of let's say SCMs to memory is ratio of let's say SCMs to memory is ratio of let's say SCMs to memory is memory more heavily skewed or if you're memory more heavily skewed or if you're memory more heavily skewed or if you're working with unified memory uh working with unified memory uh working with unified memory uh and you have systems like where I Rubin, and you have systems like where I Rubin, and you have systems like where I Rubin, uh you you have a lot of tricks to play uh you you have a lot of tricks to play uh you you have a lot of tricks to play here, right? Uh the 5-minute uh window here, right? Uh the 5-minute uh window here, right? Uh the 5-minute uh window is what a lot of providers right now is what a lot of providers right now is what a lot of providers right now put, but that's uh

put, but that's uh put, but that's uh that's that's more an operational that's that's more an operational that's that's more an operational operational operational determination operational operational determination operational operational determination rather than a like a science-based or rather than a like a science-based or rather than a like a science-based or like a core physics law determination. like a core physics law determination. like a core physics law determination. So, you can technically see over time So, you can technically see over time So, you can technically see over time maybe some uh some maybe some uh some maybe some uh some APIs are priced differently, uh but uh APIs are priced differently, uh but uh APIs are priced differently, uh but uh if you do self-deploy again, you can you if you do self-deploy again, you can you if you do self-deploy again, you can you can get past a lot of this. The the cost can get past a lot of this. The the cost can get past a lot of this. The the cost economics really change when you move economics really change when you move economics really change when you move from self-hosted models to uh from self-hosted models to uh from self-hosted models to uh API providers uh because you have a lot API providers uh because you have a lot API providers uh because you have a lot more control and uh you don't have to more control and uh you don't have to more control and uh you don't have to guess the shape of your workload. So, guess the shape of your workload. So, guess the shape of your workload. So, let's say if your workload is 32K uh let's say if your workload is 32K uh let's say if your workload is 32K uh like on average 32K cache, 1K input, 1K like on average 32K cache, 1K input, 1K like on average 32K cache, 1K input, 1K output, uh output, uh output, uh and someone else's like let's say 64K uh and someone else's like let's say 64K uh and someone else's like let's say 64K uh 1K 1K, uh if you use some provider, they 1K 1K, uh if you use some provider, they 1K 1K, uh if you use some provider, they are amortizing everyone's use uh use are amortizing everyone's use uh use are amortizing everyone's use uh use case and then giving you a price, right?

case and then giving you a price, right? case and then giving you a price, right? And they have optimized, quote and And they have optimized, quote and And they have optimized, quote and quote, for general use. If you quote, for general use. If you quote, for general use. If you self-host, you can optimize specifically self-host, you can optimize specifically self-host, you can optimize specifically for your use, and you'll likely pay much for your use, and you'll likely pay much for your use, and you'll likely pay much less. less. less. >> Yeah. This is like kind of like the >> Yeah. This is like kind of like the >> Yeah. This is like kind of like the level of like, you know, level of like, you know, level of like, you know, hardware-software frontier that we kind hardware-software frontier that we kind hardware-software frontier that we kind of like think about. When When we of like think about. When When we of like think about. When When we started Cognition, we were working on started Cognition, we were working on started Cognition, we were working on the first agents, I think one reason why the first agents, I think one reason why the first agents, I think one reason why no one else worked on agents is they no one else worked on agents is they no one else worked on agents is they were just extremely expensive. This was were just extremely expensive. This was were just extremely expensive. This was before cash tokens was a thing that API before cash tokens was a thing that API before cash tokens was a thing that API providers paid for. Like if you were providers paid for. Like if you were providers paid for. Like if you were sending 100,000 tokens and the same sending 100,000 tokens and the same sending 100,000 tokens and the same 100,000 tokens, you were paying full 100,000 tokens, you were paying full 100,000 tokens, you were paying full price for those tokens back in 2024 when price for those tokens back in 2024 when price for those tokens back in 2024 when we started. we started. we started. One of the key things that let us build One of the key things that let us build One of the key things that let us build Devin and build these first agents was Devin and build these first agents was Devin and build these first agents was we actually bought direct compute we actually bought direct compute we actually bought direct compute capacity from these providers, and capacity from these providers, and capacity from these providers, and instead of paying on a per-token basis, instead of paying on a per-token basis, instead of paying on a per-token basis, we just paid for the underlying compute, we just paid for the underlying compute, we just paid for the underlying compute, knowing that the economics of the knowing that the economics of the knowing that the economics of the compute was that we were actually paying compute was that we were actually paying compute was that we were actually paying far less for for the cash tokens that far less for for the cash tokens that far less for for the cash tokens that we'd send over. we'd send over. we'd send over. Um Um Um And And then nowadays, you know, that uh And And then nowadays, you know, that uh And And then nowadays, you know, that uh you know, there's there's like similar, you know, there's there's like similar, you know, there's there's like similar, you know, things I would like about you know, things I would like about you know, things I would like about like, you know, having a version of the like, you know, having a version of the like, you know, having a version of the cache that maybe like you can just back cache that maybe like you can just back cache that maybe like you can just back out to like storage in S3 or something out to like storage in S3 or something out to like storage in S3 or something and just like hold for much longer.

and just like hold for much longer. and just like hold for much longer. >> Yeah. Now, this is not extremely >> Yeah. Now, this is not extremely >> Yeah. Now, this is not extremely relevant to a DGX organization like relevant to a DGX organization like relevant to a DGX organization like setup, but if anyone's looking to do setup, but if anyone's looking to do setup, but if anyone's looking to do what you guys have wanted to try out what you guys have wanted to try out what you guys have wanted to try out Dynamo. We have a lot of pre-fixed cache Dynamo. We have a lot of pre-fixed cache Dynamo. We have a lot of pre-fixed cache optimizations in there. optimizations in there. optimizations in there. >> Yeah. >> Yeah. >> Yeah. Um Um Um And then And then And then going back to your question, Alex, I going back to your question, Alex, I going back to your question, Alex, I think you said like, "Oh, like you think you said like, "Oh, like you think you said like, "Oh, like you When a small model is going off and When a small model is going off and When a small model is going off and generating a ton of tokens, is that like generating a ton of tokens, is that like generating a ton of tokens, is that like a interesting time to back off?" Um to a interesting time to back off?" Um to a interesting time to back off?" Um to be honest, we haven't explored that be honest, we haven't explored that be honest, we haven't explored that right yet. So, that that might actually right yet. So, that that might actually right yet. So, that that might actually be a very interesting thing to to take a be a very interesting thing to to take a be a very interesting thing to to take a look at. Um It It is weird. I think like look at. Um It It is weird. I think like look at. Um It It is weird. I think like some small models do tend to kind of be some small models do tend to kind of be some small models do tend to kind of be less like token efficient than others, less like token efficient than others, less like token efficient than others, but um they also seem to be trained on but um they also seem to be trained on but um they also seem to be trained on like their own traces, so maybe in a way like their own traces, so maybe in a way like their own traces, so maybe in a way it ends up like balancing out. A lot of it ends up like balancing out. A lot of it ends up like balancing out. A lot of these things I I feel like we have to be these things I I feel like we have to be these things I I feel like we have to be like very empirical about to actually like very empirical about to actually like very empirical about to actually know. know. know. >> So, uh just to add on >> So, uh just to add on >> So, uh just to add on that um um that um um that um um you have a lot of like these days there you have a lot of like these days there you have a lot of like these days there are a lot of hallucination probes. So, are a lot of hallucination probes. So, are a lot of hallucination probes. So, probes that work on either the internal probes that work on either the internal probes that work on either the internal state like internal state of the models state like internal state of the models state like internal state of the models directly. So, you can have some form of directly. So, you can have some form of directly. So, you can have some form of either magnitude analysis done or linear either magnitude analysis done or linear either magnitude analysis done or linear probes or just uh the end types of probes or just uh the end types of probes or just uh the end types of probes that you can see and you can probes that you can see and you can probes that you can see and you can essentially rate like how how much you essentially rate like how how much you essentially rate like how how much you think is is tending towards think is is tending towards think is is tending towards hallucination.

hallucination. hallucination. Uh so, that kind of gives you a proxy Uh so, that kind of gives you a proxy Uh so, that kind of gives you a proxy for how lost it is. for how lost it is. for how lost it is. Uh like how lost a model is in its Uh like how lost a model is in its Uh like how lost a model is in its thinking. Uh so, you can use like thinking. Uh so, you can use like thinking. Uh so, you can use like different kinds of probes to understand different kinds of probes to understand different kinds of probes to understand like the perplexity within a model. like the perplexity within a model. like the perplexity within a model. >> Well, that's interesting. So, yeah, >> Well, that's interesting. So, yeah, >> Well, that's interesting. So, yeah, instead of using the quantity of tokens instead of using the quantity of tokens instead of using the quantity of tokens that are as indicative of a of a model that are as indicative of a of a model that are as indicative of a of a model being lost, it's the it's hallucinating being lost, it's the it's hallucinating being lost, it's the it's hallucinating more and you're more and you're more and you're >> Yeah, so uh so, essentially what is >> Yeah, so uh so, essentially what is >> Yeah, so uh so, essentially what is cache, right? It's It's the prefill It's cache, right? It's It's the prefill It's cache, right? It's It's the prefill It's the prefill stage, right? So, you What the prefill stage, right? So, you What the prefill stage, right? So, you What is a prefill stage? It's just a vector is a prefill stage? It's just a vector is a prefill stage? It's just a vector at the end of the day. So, you can at the end of the day. So, you can at the end of the day. So, you can do tune all kinds of classifiers to do tune all kinds of classifiers to do tune all kinds of classifiers to understand uh different aspects of those understand uh different aspects of those understand uh different aspects of those collections of vectors. So, with those collections of vectors. So, with those collections of vectors. So, with those kind of probes, you can kind of probes, you can kind of probes, you can guesstimate a lot of uh states guesstimate a lot of uh states guesstimate a lot of uh states of a model. of a model. of a model. >> I see. One One question I have is um you >> I see. One One question I have is um you >> I see. One One question I have is um you know, different models behave uh know, different models behave uh know, different models behave uh differently and um differently and um differently and um the kind of means that these prompts the kind of means that these prompts the kind of means that these prompts aren't portable. So, as you're doing aren't portable. So, as you're doing aren't portable. So, as you're doing model routing, how do you handle model routing, how do you handle model routing, how do you handle essentially if you're if you're going to essentially if you're if you're going to essentially if you're if you're going to a different model architecture, a different model architecture, a different model architecture, um what do you have to do to the prompts um what do you have to do to the prompts um what do you have to do to the prompts and how much is that a factor into and how much is that a factor into and how much is that a factor into either of you guys' model routing either of you guys' model routing either of you guys' model routing solution?

solution? solution? >> Like what like how how is the prompt >> Like what like how how is the prompt >> Like what like how how is the prompt itself a factor in the in the routing? itself a factor in the in the routing? itself a factor in the in the routing? >> Yeah. >> Yeah. >> Yeah. >> Yeah, um >> Yeah, um >> Yeah, um well, I think well, I think well, I think with building agents, there are all with building agents, there are all with building agents, there are all kinds of paper cuts and edge cases that kinds of paper cuts and edge cases that kinds of paper cuts and edge cases that are domain-specific are domain-specific are domain-specific and like the value of an agent company and like the value of an agent company and like the value of an agent company like the value of Devin is all these like the value of Devin is all these like the value of Devin is all these like doom loops that you've discovered like doom loops that you've discovered like doom loops that you've discovered that are across all industries and the that are across all industries and the that are across all industries and the best ways to recover from them best ways to recover from them best ways to recover from them and like man here I mean it manifests and like man here I mean it manifests and like man here I mean it manifests big time in what the prompts are going big time in what the prompts are going big time in what the prompts are going to be both for like you know how the the to be both for like you know how the the to be both for like you know how the the advisor model gets called advisor model gets called advisor model gets called um um um uh you know the smart friend uh you know the smart friend uh you know the smart friend >> [laughter] >> [laughter] >> [laughter] >> the how the like subtask agents get >> the how the like subtask agents get >> the how the like subtask agents get called and and the best thing is that called and and the best thing is that called and and the best thing is that like anyone can like like anyone can like like anyone can like like any engineer or any like agent can like any engineer or any like agent can like any engineer or any like agent can inspect the traces and like adjust the inspect the traces and like adjust the inspect the traces and like adjust the prompt and then see the like live prompt and then see the like live prompt and then see the like live accuracy accuracy accuracy long time. So I mean long time. So I mean long time. So I mean basically I just think that that's part basically I just think that that's part basically I just think that that's part the prompt is part of the the startup the prompt is part of the the startup the prompt is part of the the startup building process and is also really easy building process and is also really easy building process and is also really easy to observe and to observe and to observe and like and have and have like multiple like and have and have like multiple like and have and have like multiple people and agents collaborating on them.

people and agents collaborating on them. people and agents collaborating on them. >> Yeah. One thing I'd love to do with our >> Yeah. One thing I'd love to do with our >> Yeah. One thing I'd love to do with our fusion product and we don't have this fusion product and we don't have this fusion product and we don't have this yet and so this is kind of maybe a yet and so this is kind of maybe a yet and so this is kind of maybe a preview of some some things we work on preview of some some things we work on preview of some some things we work on is you know you can tune it against a is you know you can tune it against a is you know you can tune it against a data set but the the real thing you want data set but the the real thing you want data set but the the real thing you want when you're building a real agent when you're building a real agent when you're building a real agent someone uses is just like tune it someone uses is just like tune it someone uses is just like tune it against what actual people use it for against what actual people use it for against what actual people use it for and and what actual models they get and and what actual models they get and and what actual models they get routed to. And so there's a lot of routed to. And so there's a lot of routed to. And so there's a lot of signals for this signals for this signals for this like if if someone sends a prompt and like if if someone sends a prompt and like if if someone sends a prompt and then Devin is working and then you see then Devin is working and then you see then Devin is working and then you see that the user decides themselves like that the user decides themselves like that the user decides themselves like upgrade to a different model or they upgrade to a different model or they upgrade to a different model or they decide to downgrade or the system decide to downgrade or the system decide to downgrade or the system detects that we originally sent to the detects that we originally sent to the detects that we originally sent to the wrong one we now got to replace. Like wrong one we now got to replace. Like wrong one we now got to replace. Like that's actually a really useful stream that's actually a really useful stream that's actually a really useful stream of signals and we're actually getting of signals and we're actually getting of signals and we're actually getting into this world of like auto research into this world of like auto research into this world of like auto research where like maybe we can just have like a where like maybe we can just have like a where like maybe we can just have like a constant stream of prompts what it constant stream of prompts what it constant stream of prompts what it should have been what it was instead and should have been what it was instead and should have been what it was instead and build a system internally that's just build a system internally that's just build a system internally that's just capturing all of this and then capturing all of this and then capturing all of this and then reiterating on our routing system until reiterating on our routing system until reiterating on our routing system until it eventually kind of like fits the real it eventually kind of like fits the real it eventually kind of like fits the real production data. Um that's kind of like production data. Um that's kind of like production data. Um that's kind of like now that we it's it's it's public and now that we it's it's it's public and now that we it's it's it's public and and people are using it, this is now and people are using it, this is now and people are using it, this is now something that we're we're thinking something that we're we're thinking something that we're we're thinking about. about. about. >> Have you guys looked into prompt tuning >> Have you guys looked into prompt tuning >> Have you guys looked into prompt tuning and do you find it useful like say Japa?

and do you find it useful like say Japa? and do you find it useful like say Japa? >> Yeah, so there are like these prompt >> Yeah, so there are like these prompt >> Yeah, so there are like these prompt tuning frameworks from like a few years tuning frameworks from like a few years tuning frameworks from like a few years ago that ago that ago that tried to do some kind of like gradient tried to do some kind of like gradient tried to do some kind of like gradient descent type thing. descent type thing. descent type thing. I'm like I'm actually personally less I'm like I'm actually personally less I'm like I'm actually personally less bullish on these kind of like low level bullish on these kind of like low level bullish on these kind of like low level mechanical prompt tuning harnesses mechanical prompt tuning harnesses mechanical prompt tuning harnesses versus just telling like a smart model versus just telling like a smart model versus just telling like a smart model like here is the decision that was made like here is the decision that was made like here is the decision that was made and the context and the context and the context figure out why it went wrong. Sometimes figure out why it went wrong. Sometimes figure out why it went wrong. Sometimes you can do something as dumb as asking you can do something as dumb as asking you can do something as dumb as asking the model why did you do this instead of the model why did you do this instead of the model why did you do this instead of this and cite the prompts this and cite the prompts this and cite the prompts and then just have your agent your dev and then just have your agent your dev and then just have your agent your dev and just go and update the prompts, and just go and update the prompts, and just go and update the prompts, rerun the the test as a regression, make rerun the the test as a regression, make rerun the the test as a regression, make sure make sure it changes. Like it's a sure make sure it changes. Like it's a sure make sure it changes. Like it's a lot heavier weight of a system, but I lot heavier weight of a system, but I lot heavier weight of a system, but I kind of trust the intelligence of a kind of trust the intelligence of a kind of trust the intelligence of a system like that a lot more. system like that a lot more. system like that a lot more. >> So we're we're running out of time, so >> So we're we're running out of time, so >> So we're we're running out of time, so we're going to wrap up real quick, but I we're going to wrap up real quick, but I we're going to wrap up real quick, but I think what's really interesting is just think what's really interesting is just think what's really interesting is just from talking to you guys, we can kind of from talking to you guys, we can kind of from talking to you guys, we can kind of see how see how see how new this space is, right? How much of new this space is, right? How much of new this space is, right? How much of this is actually just research. this is actually just research. this is actually just research. We're starting to see new products come We're starting to see new products come We're starting to see new products come in. I'm really excited about your guys' in. I'm really excited about your guys' in. I'm really excited about your guys' solutions as you guys enter the space. solutions as you guys enter the space. solutions as you guys enter the space. The the ways and the needs that you need The the ways and the needs that you need The the ways and the needs that you need routing for, you know, even on a DGX routing for, you know, even on a DGX routing for, you know, even on a DGX Spark when you're doing local inference, Spark when you're doing local inference, Spark when you're doing local inference, you have more compute and if the memory you have more compute and if the memory you have more compute and if the memory is filled, one is filled, one is filled, one or if it's if the memory utilization is or if it's if the memory utilization is or if it's if the memory utilization is high, one thing you need to do is high, one thing you need to do is high, one thing you need to do is increase the compute utilization. And so increase the compute utilization. And so increase the compute utilization. And so one way you can do that is by spawning one way you can do that is by spawning one way you can do that is by spawning multiple agents that are working multiple agents that are working multiple agents that are working collaboratively. So that collaborative collaboratively. So that collaborative collaboratively. So that collaborative piece is something that not only is piece is something that not only is piece is something that not only is optimal for all of these cloud workloads optimal for all of these cloud workloads optimal for all of these cloud workloads that you guys are doing, but but that you guys are doing, but but that you guys are doing, but but specifically that is how you extract specifically that is how you extract specifically that is how you extract more performance out of this edge more performance out of this edge more performance out of this edge hardware. And I think um hardware. And I think um hardware. And I think um you know, a question here is

you know, a question here is you know, a question here is and maybe to to end on is a router going and maybe to to end on is a router going and maybe to to end on is a router going to be something that we see as a product to be something that we see as a product to be something that we see as a product or is that going to be seen as part of or is that going to be seen as part of or is that going to be seen as part of the plumbing here? Are models going to the plumbing here? Are models going to the plumbing here? Are models going to get good at routing to other models get good at routing to other models get good at routing to other models because they know they need to be because they know they need to be because they know they need to be collaborative or or harnesses going to collaborative or or harnesses going to collaborative or or harnesses going to know that they are working across know that they are working across know that they are working across multiple models. multiple models. multiple models. >> I think we already see this. Like you >> I think we already see this. Like you >> I think we already see this. Like you know, at Cognition we're training our know, at Cognition we're training our know, at Cognition we're training our models to be able to be good models to be able to be good models to be able to be good collaborators. I think it's very clear collaborators. I think it's very clear collaborators. I think it's very clear that new frontier models like the Fable that new frontier models like the Fable that new frontier models like the Fable models and GPT-5.5 and 5.6 models are models and GPT-5.5 and 5.6 models are models and GPT-5.5 and 5.6 models are like themselves like naturally like themselves like naturally like themselves like naturally collaborative and and better at collaborative and and better at collaborative and and better at delegation. So I think we're ready there delegation. So I think we're ready there delegation. So I think we're ready there at that point. at that point. at that point. >> Interesting. >> Interesting. >> Interesting. >> Yeah, I I think that the systems are are >> Yeah, I I think that the systems are are >> Yeah, I I think that the systems are are kind of becoming not muddied in some kind of becoming not muddied in some kind of becoming not muddied in some sense, but I think that ultimately we're sense, but I think that ultimately we're sense, but I think that ultimately we're understanding that as we step up the the understanding that as we step up the the understanding that as we step up the the abstraction ladder and build more more abstraction ladder and build more more abstraction ladder and build more more things to create this smarter things to create this smarter things to create this smarter blob, blob, blob, which obviously we should hopefully and which obviously we should hopefully and which obviously we should hopefully and we do understand how we're building it we do understand how we're building it we do understand how we're building it and why we're building it, and why we're building it, and why we're building it, that it's going to it's going to become that it's going to it's going to become that it's going to it's going to become a system that you look at kind of both a system that you look at kind of both a system that you look at kind of both the the different components of the the the different components of the the the different components of the system, but system, but system, but it's not just going to be just models.

it's not just going to be just models. it's not just going to be just models. There's not going to There's not going There's not going to There's not going There's not going to There's not going to be a thing as like a really great to be a thing as like a really great to be a thing as like a really great harness that is in absence of a really harness that is in absence of a really harness that is in absence of a really great model and vice versa. great model and vice versa. great model and vice versa. >> Yeah. >> Yeah. >> Yeah. Makes sense. Makes sense. Makes sense. >> I think >> I think >> I think applications, especially as built on applications, especially as built on applications, especially as built on non-deterministic systems like models, non-deterministic systems like models, non-deterministic systems like models, operate in a very low-trust environment. operate in a very low-trust environment. operate in a very low-trust environment. So yes, most of the improvements will So yes, most of the improvements will So yes, most of the improvements will likely be distributed across both models likely be distributed across both models likely be distributed across both models and the harnesses, but I think overall and the harnesses, but I think overall and the harnesses, but I think overall it's it's it's mostly it's There There it's it's it's mostly it's There There it's it's it's mostly it's There There will There will have to be some form of will There will have to be some form of will There will have to be some form of controller trying to have some form of controller trying to have some form of controller trying to have some form of arbitration because even from the model arbitration because even from the model arbitration because even from the model perspective you aren't in a perfectly perspective you aren't in a perfectly perspective you aren't in a perfectly visible world. You don't know the visible world. You don't know the visible world. You don't know the behavior of every model, so it's behavior of every model, so it's behavior of every model, so it's it's going to be at the orchestration it's going to be at the orchestration it's going to be at the orchestration level where you have these kind of level where you have these kind of level where you have these kind of things. And this has traditionally been things. And this has traditionally been things. And this has traditionally been shown by other industries like when web shown by other industries like when web shown by other industries like when web when web launched, you know, you had when web launched, you know, you had when web launched, you know, you had traffic-based routing. So it's traffic-based routing. So it's traffic-based routing. So it's different. But all the sort of routing different. But all the sort of routing different. But all the sort of routing controls have been centralized over controls have been centralized over controls have been centralized over time.

time. time. >> Makes sense. >> Makes sense. >> Makes sense. >> I I it's most likely going to be good >> I I it's most likely going to be good >> I I it's most likely going to be good news in the future. news in the future. news in the future. Um Um Um and and I and and I and and I I think like caching is a big reason for I think like caching is a big reason for I think like caching is a big reason for that. Even if you I think like a to take that. Even if you I think like a to take that. Even if you I think like a to take the flip side of of this argument, um the flip side of of this argument, um the flip side of of this argument, um the you know, it might be that in the the you know, it might be that in the the you know, it might be that in the future we have like one big model that's future we have like one big model that's future we have like one big model that's like, "I know I am the like most like, "I know I am the like most like, "I know I am the like most efficient at everything and I'm like way efficient at everything and I'm like way efficient at everything and I'm like way more efficient than Haiku. I'll solve more efficient than Haiku. I'll solve more efficient than Haiku. I'll solve every task better than Haiku can at like every task better than Haiku can at like every task better than Haiku can at like a lower price. a lower price. a lower price. Um why should I ever delegate to Haiku?" Um why should I ever delegate to Haiku?" Um why should I ever delegate to Haiku?" There's something like that actually There's something like that actually There's something like that actually could could be a model that we have in could could be a model that we have in could could be a model that we have in the future. Um the future. Um the future. Um but you're always going to have these but you're always going to have these but you're always going to have these like, you know, for example, caching. It like, you know, for example, caching. It like, you know, for example, caching. It could be that like you tell the model could be that like you tell the model could be that like you tell the model that this other model like does have the that this other model like does have the that this other model like does have the right context in cash and uh you know, right context in cash and uh you know, right context in cash and uh you know, the the orchestrator model just always the the orchestrator model just always the the orchestrator model just always has more context and the models have to has more context and the models have to has more context and the models have to be aligned. be aligned. be aligned. So, I I think like it's So, I I think like it's So, I I think like it's I I don't really see a world where like I I don't really see a world where like I I don't really see a world where like we wouldn't be able to get models to we wouldn't be able to get models to we wouldn't be able to get models to collaborate really well and and I think collaborate really well and and I think collaborate really well and and I think they're going to get better over time.

they're going to get better over time. they're going to get better over time. Um Um Um in part because they're you know, they in part because they're you know, they in part because they're you know, they just have limited memory. just have limited memory. just have limited memory. So, I I think that's kind of one one So, I I think that's kind of one one So, I I think that's kind of one one deciding factor and another deciding factor and another deciding factor and another is that um there will be like uh there is that um there will be like uh there is that um there will be like uh there will continue to be will continue to be will continue to be like if you just look at like the the like if you just look at like the the like if you just look at like the the the rankings on OpenRouter, if you look the rankings on OpenRouter, if you look the rankings on OpenRouter, if you look at our our public data and you look at at our our public data and you look at at our our public data and you look at like the top model being used by dollar like the top model being used by dollar like the top model being used by dollar spent on classification tasks, well, spent on classification tasks, well, spent on classification tasks, well, guess what it is. It's Opus. guess what it is. It's Opus. guess what it is. It's Opus. >> [laughter] >> [laughter] >> [laughter] >> I think there are there are there are >> I think there are there are there are >> I think there are there are there are big opportunities for like using small big opportunities for like using small big opportunities for like using small models for in distribution easy tasks models for in distribution easy tasks models for in distribution easy tasks and the and like as time goes on, that's and the and like as time goes on, that's and the and like as time goes on, that's going to be a larger and larger going to be a larger and larger going to be a larger and larger percentage of tasks relative to like the percentage of tasks relative to like the percentage of tasks relative to like the most valuable tasks that um very smart most valuable tasks that um very smart most valuable tasks that um very smart models spend most of their time on. models spend most of their time on. models spend most of their time on. >> Totally. Well, I want to thank you guys >> Totally. Well, I want to thank you guys >> Totally. Well, I want to thank you guys so much. Can we all give everyone a so much. Can we all give everyone a so much. Can we all give everyone a round of applause? Thank you. round of applause? Thank you. round of applause? Thank you. >> [applause]