Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean

Hello everyone. Hello everyone. So preferences over benchmarks. The talk So preferences over benchmarks. The talk So preferences over benchmarks. The talk today is about model routing and today is about model routing and today is about model routing and specifically why the way most people specifically why the way most people specifically why the way most people think about picking a model which think about picking a model which think about picking a model which usually is chasing you know to the top usually is chasing you know to the top usually is chasing you know to the top of a benchmark is actually the wrong of a benchmark is actually the wrong of a benchmark is actually the wrong instinct. instinct. instinct. I'm Archa VP of engineering for I'm Archa VP of engineering for I'm Archa VP of engineering for inference engine and AI infrastructure inference engine and AI infrastructure inference engine and AI infrastructure at Digital Ocean and I'll be joined by at Digital Ocean and I'll be joined by at Digital Ocean and I'll be joined by Tyler who built parts of the router and Tyler who built parts of the router and Tyler who built parts of the router and will actually do a live demo for us will actually do a live demo for us will actually do a live demo for us today. We both work on the managed agent today. We both work on the managed agent today. We both work on the managed agent orchestration and inference engine orchestration and inference engine orchestration and inference engine products at digital ocean. So you may know digital ocean as So you may know digital ocean as droplets, databases and app platform. droplets, databases and app platform. droplets, databases and app platform. All of that is true. We are also the AI All of that is true. We are also the AI All of that is true. We are also the AI native cloud. This is five integrated native cloud. This is five integrated native cloud. This is five integrated layers starting from infrastructure all layers starting from infrastructure all layers starting from infrastructure all the way up to the managed agents with the way up to the managed agents with the way up to the managed agents with the inference engine right in the the inference engine right in the the inference engine right in the middle. And that's why we are here middle. And that's why we are here middle. And that's why we are here talking about inference router. Routing talking about inference router. Routing talking about inference router. Routing lives in the inference engine. And if lives in the inference engine. And if lives in the inference engine. And if you want to know more about our stack you want to know more about our stack you want to know more about our stack and the full story, please come find us and the full story, please come find us and the full story, please come find us at the booth.

So everybody is reaching out for the So everybody is reaching out for the model routing and let's look at three model routing and let's look at three model routing and let's look at three reasons why the three reasons that are reasons why the three reasons that are reasons why the three reasons that are breaking the one model habit for most breaking the one model habit for most breaking the one model habit for most users. The first one I want to talk users. The first one I want to talk users. The first one I want to talk about is cost. Spend is exploding and about is cost. Spend is exploding and about is cost. Spend is exploding and even companies like Walmart, Uber, even companies like Walmart, Uber, even companies like Walmart, Uber, Microsoft, they're actively capping Microsoft, they're actively capping Microsoft, they're actively capping usage to control the inference bills. usage to control the inference bills. usage to control the inference bills. The second one is fit. One model for The second one is fit. One model for The second one is fit. One model for every task is likely an overkill. We're every task is likely an overkill. We're every task is likely an overkill. We're essentially paying frontier rates for a essentially paying frontier rates for a essentially paying frontier rates for a work that a much smaller model will be work that a much smaller model will be work that a much smaller model will be able to handle really well. And the able to handle really well. And the able to handle really well. And the third one, which for me is the most third one, which for me is the most third one, which for me is the most important one, is the risk. The risk important one, is the risk. The risk important one, is the risk. The risk associated with one single model. models associated with one single model. models associated with one single model. models can go down and if you bet your entire can go down and if you bet your entire can go down and if you bet your entire product and production on one model, you product and production on one model, you product and production on one model, you have no failover when something degrades and model orchestration is actually the and model orchestration is actually the new phas. new phas. new phas. As you all know, cloud cost optimization As you all know, cloud cost optimization As you all know, cloud cost optimization took us about 15 years for it to took us about 15 years for it to took us about 15 years for it to actually become a real good discipline actually become a real good discipline actually become a real good discipline and for companies to get it right. This and for companies to get it right. This and for companies to get it right. This one actually is arriving in months and one actually is arriving in months and one actually is arriving in months and not years. And here's the premise that I not years. And here's the premise that I not years. And here's the premise that I think everybody gets wrong about this.

think everybody gets wrong about this. think everybody gets wrong about this. We all think of like what is the best We all think of like what is the best We all think of like what is the best model for a job. Here's the thing. There model for a job. Here's the thing. There model for a job. Here's the thing. There is no single best model. The right one is no single best model. The right one is no single best model. The right one depends on the actual request. For depends on the actual request. For depends on the actual request. For example, if you're doing classification example, if you're doing classification example, if you're doing classification and labeling, a small open model may and labeling, a small open model may and labeling, a small open model may very well work really well for you and very well work really well for you and very well work really well for you and will give you really good cost will give you really good cost will give you really good cost optimizations. optimizations. optimizations. However, if you're running code However, if you're running code However, if you're running code completion in line, you will likely need completion in line, you will likely need completion in line, you will likely need really fast routing and that is where a really fast routing and that is where a really fast routing and that is where a faster larger routing model comes into faster larger routing model comes into faster larger routing model comes into picture. Think about code generation and picture. Think about code generation and picture. Think about code generation and bug fixing. You're likely good with an bug fixing. You're likely good with an bug fixing. You're likely good with an mid openw weight model. Uh and again, mid openw weight model. Uh and again, mid openw weight model. Uh and again, it'll bring you like really good cost it'll bring you like really good cost it'll bring you like really good cost optimizations over using a frontier for optimizations over using a frontier for optimizations over using a frontier for something that is likely an overkill in something that is likely an overkill in something that is likely an overkill in this situation. But then you're looking this situation. But then you're looking this situation. But then you're looking at like really accuracy critical tasks at like really accuracy critical tasks at like really accuracy critical tasks like code review and security, you're like code review and security, you're like code review and security, you're likely going to lean towards a frontier likely going to lean towards a frontier likely going to lean towards a frontier model. So essentially what makes a model right So essentially what makes a model right for a request? It's a mix that no public for a request? It's a mix that no public for a request? It's a mix that no public leaderboard can actually encode for you leaderboard can actually encode for you leaderboard can actually encode for you because it's the task itself. What are because it's the task itself. What are because it's the task itself. What are you actually trying to achieve? What is you actually trying to achieve? What is you actually trying to achieve? What is your model trying to achieve? The system your model trying to achieve? The system your model trying to achieve? The system prompts and tools around it. that is the prompts and tools around it. that is the prompts and tools around it. that is the methodology by which you're getting methodology by which you're getting methodology by which you're getting something done using a model. The cost something done using a model. The cost something done using a model. The cost you're willing to spend. This is a very you're willing to spend. This is a very you're willing to spend. This is a very very important aspect very important aspect very important aspect and latency the use case needs. Not all and latency the use case needs. Not all and latency the use case needs. Not all use cases need the same amount of use cases need the same amount of use cases need the same amount of latency. So depending on what you're latency. So depending on what you're latency. So depending on what you're trying to do, this can vary widely. And trying to do, this can vary widely. And trying to do, this can vary widely. And finally, the end user preference. All of

finally, the end user preference. All of finally, the end user preference. All of this is driven by what the end user this is driven by what the end user this is driven by what the end user really wants out of your application. really wants out of your application. really wants out of your application. So, I'm going to welcome Tyler on to So, I'm going to welcome Tyler on to So, I'm going to welcome Tyler on to stage so that he can actually show you stage so that he can actually show you stage so that he can actually show you the inference light router live in the inference light router live in the inference light router live in action and show you how it can really action and show you how it can really action and show you how it can really help with all of these key aspects that help with all of these key aspects that help with all of these key aspects that I'm calling out here. >> Testing. All right. Thank you, Archa. >> Testing. All right. Thank you, Archa. Okay. So many builders have tried auto Okay. So many builders have tried auto Okay. So many builders have tried auto routing before, but the problem was that routing before, but the problem was that routing before, but the problem was that it feels like a black box. The router it feels like a black box. The router it feels like a black box. The router makes a choice and if that choice makes a choice and if that choice makes a choice and if that choice results in poor performance, you really results in poor performance, you really results in poor performance, you really have no way of improving it. We built have no way of improving it. We built have no way of improving it. We built ours differently at the architecture ours differently at the architecture ours differently at the architecture level, which is what you can see on the level, which is what you can see on the level, which is what you can see on the screen. A request runs through our open screen. A request runs through our open screen. A request runs through our open proxy plan and our purpose-built routing proxy plan and our purpose-built routing proxy plan and our purpose-built routing model. Both open source. There is no model. Both open source. There is no model. Both open source. There is no vendor lockin, which is a key digital vendor lockin, which is a key digital vendor lockin, which is a key digital ocean value. ocean value. ocean value. You describe what matters for your You describe what matters for your You describe what matters for your workload, costs, latency, quality, workload, costs, latency, quality, workload, costs, latency, quality, preferred models preferred models preferred models or hard rules. Then the router uses that or hard rules. Then the router uses that or hard rules. Then the router uses that context to pick the right model per context to pick the right model per context to pick the right model per request. Because the routing model is request. Because the routing model is request. Because the routing model is specialized for this job, it's super specialized for this job, it's super specialized for this job, it's super fast, under 200 milliseconds, and it fast, under 200 milliseconds, and it fast, under 200 milliseconds, and it costs customers nothing extra. In our costs customers nothing extra. In our costs customers nothing extra. In our evaluations, it actually has beating evaluations, it actually has beating evaluations, it actually has beating frontier models like the GBT 5 series frontier models like the GBT 5 series frontier models like the GBT 5 series models at routing task itself with a models at routing task itself with a models at routing task itself with a fraction of the latency.

fraction of the latency. fraction of the latency. So the difference is simple. This is So the difference is simple. This is So the difference is simple. This is routing you can customize, evaluate and routing you can customize, evaluate and routing you can customize, evaluate and improve without vendor lock in. So you bring your preferences and we So you bring your preferences and we honor them. You describe a task in honor them. You describe a task in honor them. You describe a task in natural language and set what matters, natural language and set what matters, natural language and set what matters, cost, latency, and task description. You cost, latency, and task description. You cost, latency, and task description. You bring your rules and we execute them bring your rules and we execute them bring your rules and we execute them intelligently. intelligently. intelligently. Layer decision tree rules on top. Start Layer decision tree rules on top. Start Layer decision tree rules on top. Start from presets, change anything you want from presets, change anything you want from presets, change anything you want in a single line of code. And you in a single line of code. And you in a single line of code. And you validate with your own evaluations, not validate with your own evaluations, not validate with your own evaluations, not someone else's leaderboard. Route, someone else's leaderboard. Route, someone else's leaderboard. Route, evaluate, adjust, then feed that back evaluate, adjust, then feed that back evaluate, adjust, then feed that back in. That loop is key. in. That loop is key. in. That loop is key. Okay, we're going to switch gears here. Okay, we're going to switch gears here. Okay, we're going to switch gears here. We're going to do a live demo. Bear with me here. Bear with me here. All right, I'm going to show you a All right, I'm going to show you a All right, I'm going to show you a couple things. First, I'll show you couple things. First, I'll show you couple things. First, I'll show you router configuration in the UI, how to router configuration in the UI, how to router configuration in the UI, how to use it, and then how you can use use it, and then how you can use use it, and then how you can use evaluations to measure and improve your evaluations to measure and improve your evaluations to measure and improve your router's performance. And then I'll show router's performance. And then I'll show router's performance. And then I'll show you a real router that I created inside you a real router that I created inside you a real router that I created inside a coding agent workflow.

a coding agent workflow. a coding agent workflow. So I'm here in the cloud console, the So I'm here in the cloud console, the So I'm here in the cloud console, the digital ocean cloud console and you can digital ocean cloud console and you can digital ocean cloud console and you can see my routers. We have several presets. see my routers. We have several presets. see my routers. We have several presets. You can see software engineering, You can see software engineering, You can see software engineering, general writing, knowledge bases and general writing, knowledge bases and general writing, knowledge bases and document intelligence. In this case, document intelligence. In this case, document intelligence. In this case, I've actually created my own. So I I I've actually created my own. So I I I've actually created my own. So I I customized our preset software customized our preset software customized our preset software engineering. Uh if we click into this, engineering. Uh if we click into this, engineering. Uh if we click into this, we can see that I have sever several we can see that I have sever several we can see that I have sever several different tasks here. I have bug fixing, different tasks here. I have bug fixing, different tasks here. I have bug fixing, code generation, test writing, and a few code generation, test writing, and a few code generation, test writing, and a few others. This also shows that you can others. This also shows that you can others. This also shows that you can specify more than one model per task in specify more than one model per task in specify more than one model per task in the bug fixing case and code generation the bug fixing case and code generation the bug fixing case and code generation case. Um, in the code generation, I have case. Um, in the code generation, I have case. Um, in the code generation, I have GLM 5.2 and GPT 5.2. And because I GLM 5.2 and GPT 5.2. And because I GLM 5.2 and GPT 5.2. And because I really want to always route to GLM 5.2 really want to always route to GLM 5.2 really want to always route to GLM 5.2 unless it's down, I use this manual unless it's down, I use this manual unless it's down, I use this manual ranking option. So, it'll always go to ranking option. So, it'll always go to ranking option. So, it'll always go to GLM 5.2. If GLM fails, it'll fail over GLM 5.2. If GLM fails, it'll fail over GLM 5.2. If GLM fails, it'll fail over to GPD 5.2. to GPD 5.2. to GPD 5.2. In the bug fixing one, you can see a In the bug fixing one, you can see a In the bug fixing one, you can see a little bit of a different one. In this little bit of a different one. In this little bit of a different one. In this case, I have selection policy fastest. case, I have selection policy fastest. case, I have selection policy fastest. So out of this model pool, if it matches So out of this model pool, if it matches So out of this model pool, if it matches to bug fixing, it'll pick whichever to bug fixing, it'll pick whichever to bug fixing, it'll pick whichever one's been fastest in about the last 30 one's been fastest in about the last 30 one's been fastest in about the last 30 minutes.

minutes. minutes. Okay, let's do this in action a little Okay, let's do this in action a little Okay, let's do this in action a little bit. Here's our playground where I'll bit. Here's our playground where I'll bit. Here's our playground where I'll show a couple of examples side by side. show a couple of examples side by side. show a couple of examples side by side. First, I'll start with just a simple First, I'll start with just a simple First, I'll start with just a simple prompt. Write a basic Fibonacci prompt. Write a basic Fibonacci prompt. Write a basic Fibonacci function. function. function. And as this runs, we can see on the left And as this runs, we can see on the left And as this runs, we can see on the left we're writing to Opus. On the right, we're writing to Opus. On the right, we're writing to Opus. On the right, we're using our software engineering we're using our software engineering we're using our software engineering router that I just showed you. And router that I just showed you. And router that I just showed you. And you're going to see that it picks you're going to see that it picks you're going to see that it picks different models on the right. So in different models on the right. So in different models on the right. So in this case, it matched to the code this case, it matched to the code this case, it matched to the code snippets task and just used the long snippets task and just used the long snippets task and just used the long before Maverick model that I had before Maverick model that I had before Maverick model that I had configured for that one. And if we configured for that one. And if we configured for that one. And if we scroll down, I mean this is this is scroll down, I mean this is this is scroll down, I mean this is this is obvious, right? But this model is obvious, right? But this model is obvious, right? But this model is extremely fast and extremely cheap extremely fast and extremely cheap extremely fast and extremely cheap compared to Opus. Now let's say optimize my function and Now let's say optimize my function and we'll see the same thing happen. In this we'll see the same thing happen. In this we'll see the same thing happen. In this case, it matched the code performance case, it matched the code performance case, it matched the code performance optimization task using GBT 5.2. And optimization task using GBT 5.2. And optimization task using GBT 5.2. And again, it's obviously significantly again, it's obviously significantly again, it's obviously significantly faster. If we scroll down here, we can faster. If we scroll down here, we can faster. If we scroll down here, we can also see that it's significantly also see that it's significantly also see that it's significantly cheaper. We'll do one more. Write some cheaper. We'll do one more. Write some cheaper. We'll do one more. Write some unit tests.

Okay. And in this case, it matched to Okay. And in this case, it matched to Cloud 5 Summit on the test writing and Cloud 5 Summit on the test writing and Cloud 5 Summit on the test writing and code verification. code verification. code verification. And again, we're going to see faster and And again, we're going to see faster and And again, we're going to see faster and cheaper. So, it's a pattern. It matches cheaper. So, it's a pattern. It matches cheaper. So, it's a pattern. It matches my, you know, vibe check, right? It's my, you know, vibe check, right? It's my, you know, vibe check, right? It's still vibes, though. How you actually still vibes, though. How you actually still vibes, though. How you actually prove it is working it through prove it is working it through prove it is working it through evaluations. evaluations. evaluations. So, I have an evaluation that I ran So, I have an evaluation that I ran So, I have an evaluation that I ran here, comparing Opus on the left or here, comparing Opus on the left or here, comparing Opus on the left or actually on the right hand side to my actually on the right hand side to my actually on the right hand side to my router on the left hand side. You can router on the left hand side. You can router on the left hand side. You can see that the scores 90% for my router, see that the scores 90% for my router, see that the scores 90% for my router, 95% correctness for Opus are very very 95% correctness for Opus are very very 95% correctness for Opus are very very close. In fact, that's pretty much close. In fact, that's pretty much close. In fact, that's pretty much within a judge uh margin of error. But within a judge uh margin of error. But within a judge uh margin of error. But what what's really interesting is if we what what's really interesting is if we what what's really interesting is if we scroll down here, we can see that the scroll down here, we can see that the scroll down here, we can see that the router knew significantly less tokens router knew significantly less tokens router knew significantly less tokens and was significantly faster than Opus. and was significantly faster than Opus. and was significantly faster than Opus. Okay, let's jump into a real workflow Okay, let's jump into a real workflow Okay, let's jump into a real workflow here. This is where the inference router here. This is where the inference router here. This is where the inference router really becomes impactful. Here I have really becomes impactful. Here I have really becomes impactful. Here I have two terminals running open code. On the two terminals running open code. On the two terminals running open code. On the left I have a single model approach left I have a single model approach left I have a single model approach using quad opus. So I have opus set up using quad opus. So I have opus set up using quad opus. So I have opus set up or open code set up with opus. On the or open code set up with opus. On the or open code set up with opus. On the right, I've configured open code to send right, I've configured open code to send right, I've configured open code to send requests to our software engineering requests to our software engineering requests to our software engineering router that I just showed you router that I just showed you router that I just showed you configured.

configured. configured. Um, below I kind of have this customuilt Um, below I kind of have this customuilt Um, below I kind of have this customuilt open code where you'll be able to see open code where you'll be able to see open code where you'll be able to see live uh, observability essentially. So, live uh, observability essentially. So, live uh, observability essentially. So, let's go ahead and get these started. let's go ahead and get these started. let's go ahead and get these started. It's just a simple feature request It's just a simple feature request It's just a simple feature request pre-loaded into here. Build me a pre-loaded into here. Build me a pre-loaded into here. Build me a spinning wheel app. I'll run the same spinning wheel app. I'll run the same spinning wheel app. I'll run the same prompt in both. prompt in both. prompt in both. And as this runs, we can focus on the And as this runs, we can focus on the And as this runs, we can focus on the bottom panel. So, it'll start to show up bottom panel. So, it'll start to show up bottom panel. So, it'll start to show up here. Hopefully, we can see that on the here. Hopefully, we can see that on the here. Hopefully, we can see that on the screen. uh you'll be able to see token screen. uh you'll be able to see token screen. uh you'll be able to see token usage in real time, which models are usage in real time, which models are usage in real time, which models are being selected, what tasks those map to, being selected, what tasks those map to, being selected, what tasks those map to, and the cost accumulating live. So on and the cost accumulating live. So on and the cost accumulating live. So on the right, we can already see that we're the right, we can already see that we're the right, we can already see that we're starting to route to GLM 5.2 because our starting to route to GLM 5.2 because our starting to route to GLM 5.2 because our requests are starting to match the code requests are starting to match the code requests are starting to match the code generation. And on the left, of course, generation. And on the left, of course, generation. And on the left, of course, we're just routing to quadopus. I think we're just routing to quadopus. I think we're just routing to quadopus. I think open code sometimes routes to to haiku open code sometimes routes to to haiku open code sometimes routes to to haiku by itself. So that's what you see there. by itself. So that's what you see there. by itself. So that's what you see there. And we'll notice latency, too. How And we'll notice latency, too. How And we'll notice latency, too. How quickly things start to come back. In quickly things start to come back. In quickly things start to come back. In this case, it wants me to this case, it wants me to this case, it wants me to create a temporary directory. create a temporary directory. create a temporary directory. So the key difference here is that on So the key difference here is that on So the key difference here is that on the left we'll see every single request the left we'll see every single request the left we'll see every single request that I write goes to the same premium that I write goes to the same premium that I write goes to the same premium model. Cost and latency is going to stay model. Cost and latency is going to stay model. Cost and latency is going to stay high for pretty much every single task.

high for pretty much every single task. high for pretty much every single task. On the right the router is selecting On the right the router is selecting On the right the router is selecting models based on the task. models based on the task. models based on the task. [snorts] So we're optimizing both cost [snorts] So we're optimizing both cost [snorts] So we're optimizing both cost and speed. And we can see that our and speed. And we can see that our and speed. And we can see that our software engineering router already software engineering router already software engineering router already finished. And if we look here, it finished. And if we look here, it finished. And if we look here, it actually matched to two models actually matched to two models actually matched to two models throughout. So, let's go and open this throughout. So, let's go and open this throughout. So, let's go and open this up and see how it looks. Okay, this up and see how it looks. Okay, this up and see how it looks. Okay, this actually looks really solid to me in actually looks really solid to me in actually looks really solid to me in Opus 4.7 finish at a similar time. Let's Opus 4.7 finish at a similar time. Let's Opus 4.7 finish at a similar time. Let's take a look at that. We can compare take a look at that. We can compare take a look at that. We can compare them. I mean, it's this is a vibe check, them. I mean, it's this is a vibe check, them. I mean, it's this is a vibe check, right? But honestly, I would say the right? But honestly, I would say the right? But honestly, I would say the software engineering router did better software engineering router did better software engineering router did better because this is an interesting approach because this is an interesting approach because this is an interesting approach that you I'm not even sure it works too that you I'm not even sure it works too that you I'm not even sure it works too well. So, in this case, the router did a well. So, in this case, the router did a well. So, in this case, the router did a little bit better. little bit better. little bit better. So now that step is done, you know, we So now that step is done, you know, we So now that step is done, you know, we get similar outputs, but if we look get similar outputs, but if we look get similar outputs, but if we look here, here, here, the software engineering router has only the software engineering router has only the software engineering router has only spent 8 cents on the session while Opus spent 8 cents on the session while Opus spent 8 cents on the session while Opus directly has spent 25 cents. So we have directly has spent 25 cents. So we have directly has spent 25 cents. So we have a about a 3x in cost and very very a about a 3x in cost and very very a about a 3x in cost and very very similar quality so far. similar quality so far. similar quality so far. Let's try another another prompt here. Let's try another another prompt here. Let's try another another prompt here. What what comes next in a software What what comes next in a software What what comes next in a software engineering life cycle? Probably write engineering life cycle? Probably write engineering life cycle? Probably write some unit test, right?

start up this first. [clears throat] On start up this first. [clears throat] On the right we have the router again. And the right we have the router again. And the right we have the router again. And we can see that it got matched to the we can see that it got matched to the we can see that it got matched to the test writing and code verification which test writing and code verification which test writing and code verification which picked the claude 5 sonnet model because picked the claude 5 sonnet model because picked the claude 5 sonnet model because that's what I configured earlier. And that's what I configured earlier. And that's what I configured earlier. And we'll see the same pattern. It's going we'll see the same pattern. It's going we'll see the same pattern. It's going to be significantly cheaper overall to be significantly cheaper overall to be significantly cheaper overall across the entire session than going across the entire session than going across the entire session than going straight to opus. straight to opus. straight to opus. So we'll let this finish here. Okay. And that finished. Let's just Okay. And that finished. Let's just queue up one more, write some queue up one more, write some queue up one more, write some documentation in a readme, and then we'll compare the total session and then we'll compare the total session cost. Okay. And as this runs, we'll wait and Okay. And as this runs, we'll wait and see what it does. Okay, it created the read me and if we Okay, it created the read me and if we look here, we can see that the total look here, we can see that the total look here, we can see that the total session cost for the router was 14 while session cost for the router was 14 while session cost for the router was 14 while the total session cost for Opus was 44.

the total session cost for Opus was 44. the total session cost for Opus was 44. So at this point, we can see the cost is So at this point, we can see the cost is So at this point, we can see the cost is significantly lower. Latency is significantly lower. Latency is significantly lower. Latency is optimized per step and the quality optimized per step and the quality optimized per step and the quality remains pretty similar across. So you remains pretty similar across. So you remains pretty similar across. So you can see as you scale this, the cost can see as you scale this, the cost can see as you scale this, the cost performance really add up. Okay, Archa, performance really add up. Okay, Archa, performance really add up. Okay, Archa, back to you. >> Thank you so much, Tyler. And that that >> Thank you so much, Tyler. And that that was actually a live demo that we ran was actually a live demo that we ran was actually a live demo that we ran here. So, thanks to Tyler for setting it here. So, thanks to Tyler for setting it here. So, thanks to Tyler for setting it up and taking us through that. So, now up and taking us through that. So, now up and taking us through that. So, now that you've seen it work, let's look at that you've seen it work, let's look at that you've seen it work, let's look at some quick facts. Routing decision and some quick facts. Routing decision and some quick facts. Routing decision and under 200 milliseconds per request. under 200 milliseconds per request. under 200 milliseconds per request. It runs on a custom mixture of experts It runs on a custom mixture of experts It runs on a custom mixture of experts model purpose-built for routing. Zero model purpose-built for routing. Zero model purpose-built for routing. Zero application code changes needed from you application code changes needed from you application code changes needed from you to get it to adopt and it's free and to get it to adopt and it's free and to get it to adopt and it's free and included so you do not have to roll out included so you do not have to roll out included so you do not have to roll out your own router. And we open source the your own router. And we open source the your own router. And we open source the whole routing model via Plano. So you whole routing model via Plano. So you whole routing model via Plano. So you can actually check how that looks as can actually check how that looks as can actually check how that looks as well.

well. well. The last thing I wanted to talk about The last thing I wanted to talk about The last thing I wanted to talk about was a bit about um routing is the was a bit about um routing is the was a bit about um routing is the foundation layer. It's not really the foundation layer. It's not really the foundation layer. It's not really the destination. And there are three things destination. And there are three things destination. And there are three things that we usually build on top of it. The that we usually build on top of it. The that we usually build on top of it. The first one is eval to prove that the first one is eval to prove that the first one is eval to prove that the right model works with your use case and right model works with your use case and right model works with your use case and your test well. Caching so that you can your test well. Caching so that you can your test well. Caching so that you can stop paying twice or more for the same stop paying twice or more for the same stop paying twice or more for the same answer each time. And personalization so answer each time. And personalization so answer each time. And personalization so that the router learns what works for that the router learns what works for that the router learns what works for your team over time. This is a your team over time. This is a your team over time. This is a continuous improvement loop maturing continuous improvement loop maturing continuous improvement loop maturing over time. That means that the more you over time. That means that the more you over time. That means that the more you route and evaluate, the better the route and evaluate, the better the route and evaluate, the better the router does for your workload. router does for your workload. router does for your workload. So to summarize, where does this leave So to summarize, where does this leave So to summarize, where does this leave you? There is no single best model. you? There is no single best model. you? There is no single best model. There's only the right model for the There's only the right model for the There's only the right model for the request and benchmarks will only tell request and benchmarks will only tell request and benchmarks will only tell you part of the story. Your preferences you part of the story. Your preferences you part of the story. Your preferences will tell you the rest. And we built the will tell you the rest. And we built the will tell you the rest. And we built the router to honor your preferences and router to honor your preferences and router to honor your preferences and stay open so that you're never locked stay open so that you're never locked stay open so that you're never locked into a single stack. And that's how into a single stack. And that's how into a single stack. And that's how teams actually built. We are digital teams actually built. We are digital teams actually built. We are digital ocean and AI native cloud. Come find us ocean and AI native cloud. Come find us ocean and AI native cloud. Come find us at the booth and route your next at the booth and route your next at the booth and route your next workload with us. Thank you so much for workload with us. Thank you so much for workload with us. Thank you so much for being here.

being here. being here. [applause]