Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards — Dan Bjornn, Lease End

>> All right. Hello, everybody. >> All right. Hello, everybody. Thank you for coming. I'm Dan Bjorn. I'm Thank you for coming. I'm Dan Bjorn. I'm Thank you for coming. I'm Dan Bjorn. I'm a senior data scientist at Lease End. a senior data scientist at Lease End. a senior data scientist at Lease End. Lease End, we connect people who Lease End, we connect people who Lease End, we connect people who are coming to the end of their auto are coming to the end of their auto are coming to the end of their auto lease with financing options so that lease with financing options so that lease with financing options so that they can buy out their lease and keep they can buy out their lease and keep they can buy out their lease and keep their car. their car. their car. Now, as part of this, we Now, as part of this, we Now, as part of this, we built built built a an LLM-based application in late 2024 a an LLM-based application in late 2024 a an LLM-based application in late 2024 to help our customers connect with to help our customers connect with to help our customers connect with with our sales team. This application with our sales team. This application with our sales team. This application allowed them to send messages through allowed them to send messages through allowed them to send messages through text. They could ask questions about the text. They could ask questions about the text. They could ask questions about the sales process. They could schedule sales process. They could schedule sales process. They could schedule calls. They could get reminders, all of calls. They could get reminders, all of calls. They could get reminders, all of this stuff. this stuff. this stuff. Our first solution used a workflow-based Our first solution used a workflow-based Our first solution used a workflow-based approach approach approach built on top of a rag system where we built on top of a rag system where we built on top of a rag system where we searched a vector database of messages searched a vector database of messages searched a vector database of messages that we had already seen and classified that we had already seen and classified that we had already seen and classified with the customer's intent. with the customer's intent. with the customer's intent. So, for example, a message saying, "Call So, for example, a message saying, "Call So, for example, a message saying, "Call me tomorrow."

me tomorrow." me tomorrow." would be classified as the customer would be classified as the customer would be classified as the customer wants to wants to talk later. wants to wants to talk later. wants to wants to talk later. A message saying, "I've got time now." A message saying, "I've got time now." A message saying, "I've got time now." would be classified as would be classified as would be classified as the customer wants to talk right now. the customer wants to talk right now. the customer wants to talk right now. This has worked, This has worked, This has worked, but not super amazing. There's a lot of but not super amazing. There's a lot of but not super amazing. There's a lot of nuance in in messages and and nuance in in messages and and nuance in in messages and and conversation, and this rag approach just conversation, and this rag approach just conversation, and this rag approach just couldn't quite couldn't quite couldn't quite pick up on that nuance. And so, we pick up on that nuance. And so, we pick up on that nuance. And so, we started to look for for new options to started to look for for new options to started to look for for new options to improve this. improve this. improve this. And naturally, being a data scientist, And naturally, being a data scientist, And naturally, being a data scientist, my first thought was, "Hey, let's start my first thought was, "Hey, let's start my first thought was, "Hey, let's start fine-tuning." This seemed like a fun fine-tuning." This seemed like a fun fine-tuning." This seemed like a fun thing to do and I was sure that this was thing to do and I was sure that this was thing to do and I was sure that this was the right call. the right call. the right call. Um there's a few reasons for that. First Um there's a few reasons for that. First Um there's a few reasons for that. First of all, of all, of all, uh we need a better accuracy. uh we need a better accuracy. uh we need a better accuracy. Uh our entire system Uh our entire system Uh our entire system uh was built upon us getting the user's uh was built upon us getting the user's uh was built upon us getting the user's intent correct. Did they want to talk intent correct. Did they want to talk intent correct. Did they want to talk now? Did they want to schedule a call? now? Did they want to schedule a call? now? Did they want to schedule a call? Did they want to opt out? Did they want to opt out? Did they want to opt out? All of this hinged on that decision and All of this hinged on that decision and All of this hinged on that decision and so we needed to make sure that we got so we needed to make sure that we got so we needed to make sure that we got that first and foremost.

that first and foremost. that first and foremost. Next, Next, Next, uh we could use smaller models with uh we could use smaller models with uh we could use smaller models with fine-tuning and so this would lower the fine-tuning and so this would lower the fine-tuning and so this would lower the cost and also lower latency. So, this is cost and also lower latency. So, this is cost and also lower latency. So, this is really important for us because we were really important for us because we were really important for us because we were uh uh uh responding to thousands of messages a responding to thousands of messages a responding to thousands of messages a day in real time and so it it uh would day in real time and so it it uh would day in real time and so it it uh would help us scale a lot. help us scale a lot. help us scale a lot. Then next, like I said, we were Then next, like I said, we were Then next, like I said, we were classifying the intent of the user and classifying the intent of the user and classifying the intent of the user and so this was a very narrow structured so this was a very narrow structured so this was a very narrow structured task that we were trying to do and so it task that we were trying to do and so it task that we were trying to do and so it lent itself very nicely to supervised lent itself very nicely to supervised lent itself very nicely to supervised fine-tuning. fine-tuning. fine-tuning. Uh we would bucket uh that conversation Uh we would bucket uh that conversation Uh we would bucket uh that conversation in one of six different categories and in one of six different categories and in one of six different categories and the model would learn the differences the model would learn the differences the model would learn the differences between those. between those. between those. Uh so, seemed like a great option there. Uh so, seemed like a great option there. Uh so, seemed like a great option there. Lastly, Lastly, Lastly, uh uh uh I believe that this would help us have a I believe that this would help us have a I believe that this would help us have a little bit more control over our destiny little bit more control over our destiny little bit more control over our destiny with the the model providers. with the the model providers. with the the model providers. Uh the idea was that we had the data Uh the idea was that we had the data Uh the idea was that we had the data and all we would need to do is pass that and all we would need to do is pass that and all we would need to do is pass that into a new model, into a new model, into a new model, go through the fine-tuning process and go through the fine-tuning process and go through the fine-tuning process and we could get similar results no matter we could get similar results no matter we could get similar results no matter uh what we decided to use. So, we could uh what we decided to use. So, we could uh what we decided to use. So, we could be model agnostic.

be model agnostic. be model agnostic. So, this was the approach that we took So, this was the approach that we took So, this was the approach that we took um um um and I built a pipeline to collect and I built a pipeline to collect and I built a pipeline to collect examples, examples, examples, run LLM as judge uh classifications to run LLM as judge uh classifications to run LLM as judge uh classifications to label our data, I'd uh manually review label our data, I'd uh manually review label our data, I'd uh manually review that, create holdout sets, that, create holdout sets, that, create holdout sets, go through the fine-tuning process, go through the fine-tuning process, go through the fine-tuning process, check my metrics. check my metrics. check my metrics. This was a data scientist's dream. This was a data scientist's dream. This was a data scientist's dream. And uh And uh And uh the numbers sure helped. Within a year, the numbers sure helped. Within a year, the numbers sure helped. Within a year, this application had helped us bring in this application had helped us bring in this application had helped us bring in $12 million of revenue at a 50x ROI. $12 million of revenue at a 50x ROI. $12 million of revenue at a 50x ROI. Um Um Um it was pretty awesome. it was pretty awesome. it was pretty awesome. But But But uh the whole time it was quietly uh the whole time it was quietly uh the whole time it was quietly accumulating debt underneath that we accumulating debt underneath that we accumulating debt underneath that we didn't see. didn't see. didn't see. So, I want to show a couple examples of So, I want to show a couple examples of So, I want to show a couple examples of how how how this application could get things wrong. this application could get things wrong. this application could get things wrong. Uh first of all, Uh first of all, Uh first of all, uh the confused confirmer is a situation uh the confused confirmer is a situation uh the confused confirmer is a situation where where where um when customers um when customers um when customers set up an appointment with a sales rep, set up an appointment with a sales rep, set up an appointment with a sales rep, we send them a confirmation message to we send them a confirmation message to we send them a confirmation message to let them know that it's been scheduled let them know that it's been scheduled let them know that it's been scheduled and give them the details of that.

and give them the details of that. and give them the details of that. Uh so, a conversation may look like Uh so, a conversation may look like Uh so, a conversation may look like this. this. this. We reach out and say, "Hi Tracy, just We reach out and say, "Hi Tracy, just We reach out and say, "Hi Tracy, just confirming your Lease End call with your confirming your Lease End call with your confirming your Lease End call with your advisor is set for Thursday at 2:00 p.m. advisor is set for Thursday at 2:00 p.m. advisor is set for Thursday at 2:00 p.m. We'll call you then." We'll call you then." We'll call you then." Tracy then sends us a message back Tracy then sends us a message back Tracy then sends us a message back saying, "Sounds good." saying, "Sounds good." saying, "Sounds good." And then our LLM responds with, "Great, And then our LLM responds with, "Great, And then our LLM responds with, "Great, I'm calling you right now." I'm calling you right now." I'm calling you right now." Uh Uh Uh it's not what we want. We just confirmed it's not what we want. We just confirmed it's not what we want. We just confirmed a an appointment for a following day, a an appointment for a following day, a an appointment for a following day, and then all of a sudden we start and then all of a sudden we start and then all of a sudden we start calling them. This calling them. This calling them. This led to frustrated customers and some led to frustrated customers and some led to frustrated customers and some missed opportunities. missed opportunities. missed opportunities. Uh the next one Uh the next one Uh the next one um I've come to lovingly call the um I've come to lovingly call the um I've come to lovingly call the overeager puppy. overeager puppy. overeager puppy. Um Um Um the the conversation looks like this. the the conversation looks like this. the the conversation looks like this. So, first, "Hi James, this is Alex with So, first, "Hi James, this is Alex with So, first, "Hi James, this is Alex with Lease End reaching out about your Lease End reaching out about your Lease End reaching out about your upcoming lease maturity." upcoming lease maturity." upcoming lease maturity." James then says, "Hi, good morning." James then says, "Hi, good morning." James then says, "Hi, good morning." And And And "Good morning, I'm giving you a call." "Good morning, I'm giving you a call." "Good morning, I'm giving you a call." Um Um Um just like a puppy that gets so excited just like a puppy that gets so excited just like a puppy that gets so excited that somebody's giving it attention.

that somebody's giving it attention. that somebody's giving it attention. Our model decided to to give a call Our model decided to to give a call Our model decided to to give a call right there. right there. right there. Um obviously this is not what James Um obviously this is not what James Um obviously this is not what James wanted. This actually did happen in wanted. This actually did happen in wanted. This actually did happen in production. production. production. Um Um Um very embarrassing there. very embarrassing there. very embarrassing there. Uh but this is a these are a couple Uh but this is a these are a couple Uh but this is a these are a couple examples of of where it went wrong and examples of of where it went wrong and examples of of where it went wrong and and don't get me wrong, and don't get me wrong, and don't get me wrong, the the app did well. The revenue the the app did well. The revenue the the app did well. The revenue numbers show that that it was working, numbers show that that it was working, numbers show that that it was working, but it could also mess up pretty but it could also mess up pretty but it could also mess up pretty spectacularly. spectacularly. spectacularly. Um the big issue wasn't Um the big issue wasn't Um the big issue wasn't how to fix it, but how to make it the how to fix it, but how to make it the how to fix it, but how to make it the fix manageable. fix manageable. fix manageable. The the fine-tuning process was pretty The the fine-tuning process was pretty The the fine-tuning process was pretty complex. complex. complex. Uh first we needed to gather examples of Uh first we needed to gather examples of Uh first we needed to gather examples of the problems that we started to see. the problems that we started to see. the problems that we started to see. Um then we needed to ask ourselves, Um then we needed to ask ourselves, Um then we needed to ask ourselves, uh do we have enough examples for uh to uh do we have enough examples for uh to uh do we have enough examples for uh to go through fine-tuning? If not, we go through fine-tuning? If not, we go through fine-tuning? If not, we synthesized those examples. Uh we passed synthesized those examples. Uh we passed synthesized those examples. Uh we passed it through an LLM. It created some some it through an LLM. It created some some it through an LLM. It created some some possible examples there.

possible examples there. possible examples there. We'd have to validate those, which was a We'd have to validate those, which was a We'd have to validate those, which was a very manual process uh because we wanted very manual process uh because we wanted very manual process uh because we wanted to make sure it had the best training to make sure it had the best training to make sure it had the best training data possible. data possible. data possible. And then once we had enough, uh we And then once we had enough, uh we And then once we had enough, uh we labeled those with the the labeled those with the the labeled those with the the categorization bins and we validated categorization bins and we validated categorization bins and we validated validated those through a manual review. validated those through a manual review. validated those through a manual review. Surprisingly, the fine-tuning process Surprisingly, the fine-tuning process Surprisingly, the fine-tuning process was the shortest part of all of this. Uh was the shortest part of all of this. Uh was the shortest part of all of this. Uh normally it took about an hour depending normally it took about an hour depending normally it took about an hour depending on the size of the data that we had, on the size of the data that we had, on the size of the data that we had, but uh but uh but uh we never got it on the first iteration. we never got it on the first iteration. we never got it on the first iteration. Uh normally what happened was we would Uh normally what happened was we would Uh normally what happened was we would uh uh uh we would fine-tune and we'd evaluate we would fine-tune and we'd evaluate we would fine-tune and we'd evaluate this and uh we fixed the problem that we this and uh we fixed the problem that we this and uh we fixed the problem that we were just trying to solve, were just trying to solve, were just trying to solve, but then we caused regressions in other but then we caused regressions in other but then we caused regressions in other things. And so this turned into kind of things. And so this turned into kind of things. And so this turned into kind of a whack-a-mole process where we would a whack-a-mole process where we would a whack-a-mole process where we would solve something new, but then other old solve something new, but then other old solve something new, but then other old issues kept popping up that we had to to issues kept popping up that we had to to issues kept popping up that we had to to whack down. whack down. whack down. Um this whole process took about a week Um this whole process took about a week Um this whole process took about a week to gather the data, label everything, go to gather the data, label everything, go to gather the data, label everything, go through the fine-tuning process, and through the fine-tuning process, and through the fine-tuning process, and iterate, and then deploy. So, it was iterate, and then deploy. So, it was iterate, and then deploy. So, it was costly.

costly. costly. Um therefore, we needed to triage Um therefore, we needed to triage Um therefore, we needed to triage all of these issues that we ran into. We all of these issues that we ran into. We all of these issues that we ran into. We asked ourselves three questions before asked ourselves three questions before asked ourselves three questions before we did any any retraining. we did any any retraining. we did any any retraining. How frequent is the issue? How frequent is the issue? How frequent is the issue? Is it something that customers are Is it something that customers are Is it something that customers are seeing every day? Is it one-off? Um seeing every day? Is it one-off? Um seeing every day? Is it one-off? Um One big exception to this was if it was One big exception to this was if it was One big exception to this was if it was hurting the customer experience too hurting the customer experience too hurting the customer experience too much. much. much. So, for example of this would be uh So, for example of this would be uh So, for example of this would be uh somebody repeatedly stating what their somebody repeatedly stating what their somebody repeatedly stating what their uh their preference for a call time is, uh their preference for a call time is, uh their preference for a call time is, and then the uh and then the uh and then the uh the model ignoring that. the model ignoring that. the model ignoring that. Another one would be a customer Another one would be a customer Another one would be a customer scheduling a call, we tell them that scheduling a call, we tell them that scheduling a call, we tell them that we've scheduled it for them, but we we've scheduled it for them, but we we've scheduled it for them, but we don't return the payload in in the don't return the payload in in the don't return the payload in in the proper way, proper way, proper way, and so the the call never gets and so the the call never gets and so the the call never gets scheduled, and so we don't follow up scheduled, and so we don't follow up scheduled, and so we don't follow up with them. So, these kinds of things with them. So, these kinds of things with them. So, these kinds of things needed to be fixed right away, but needed to be fixed right away, but needed to be fixed right away, but before we did that, uh before we did that, uh before we did that, uh we asked the last question. Is there we asked the last question. Is there we asked the last question. Is there anything that we can do anything that we can do anything that we can do in order to prevent a retrain? Can we in order to prevent a retrain? Can we in order to prevent a retrain? Can we have some kind of a band-aid fix to get have some kind of a band-aid fix to get have some kind of a band-aid fix to get out there so we don't have to go through out there so we don't have to go through out there so we don't have to go through a whole week-long process uh a whole week-long process uh a whole week-long process uh for one or two issues.

for one or two issues. for one or two issues. And so, we we ranked our own bugs uh And so, we we ranked our own bugs uh And so, we we ranked our own bugs uh based on how much customer pain we could based on how much customer pain we could based on how much customer pain we could tolerate at the moment. tolerate at the moment. tolerate at the moment. Um so, not a great situation to be in Um so, not a great situation to be in Um so, not a great situation to be in with a production system. with a production system. with a production system. This led to what I've come to call the This led to what I've come to call the This led to what I've come to call the calcification tax. calcification tax. calcification tax. Uh the more we use the model, the more Uh the more we use the model, the more Uh the more we use the model, the more rigid everything became. rigid everything became. rigid everything became. This manifested in a couple different This manifested in a couple different This manifested in a couple different ways. First, ways. First, ways. First, we were locked into our model. You we were locked into our model. You we were locked into our model. You remember when I said that uh remember when I said that uh remember when I said that uh fine-tuning would give us uh fine-tuning would give us uh fine-tuning would give us uh more freedom in what model we did. That more freedom in what model we did. That more freedom in what model we did. That was not the case. was not the case. was not the case. Um Um Um within providers, there's nuance between within providers, there's nuance between within providers, there's nuance between one model version to another, and so one model version to another, and so one model version to another, and so that changes the the training data that that changes the the training data that that changes the the training data that you need to provide it. Um across model you need to provide it. Um across model you need to provide it. Um across model providers, uh it's extremely different. providers, uh it's extremely different. providers, uh it's extremely different. The structure of the data you need to The structure of the data you need to The structure of the data you need to pass to it, it's different, the amount pass to it, it's different, the amount pass to it, it's different, the amount of the training data to get good of the training data to get good of the training data to get good results, the way to interact with the results, the way to interact with the results, the way to interact with the training interface.

training interface. training interface. All of this caused a lot of complexity, All of this caused a lot of complexity, All of this caused a lot of complexity, and so it was just too costly for us to and so it was just too costly for us to and so it was just too costly for us to switch. switch. switch. And so um And so um And so um to we kept it the same model for to we kept it the same model for to we kept it the same model for consistency because we already had a lot consistency because we already had a lot consistency because we already had a lot to do with uh with each retraining to do with uh with each retraining to do with uh with each retraining process, and we couldn't afford to process, and we couldn't afford to process, and we couldn't afford to upgrade the model. upgrade the model. upgrade the model. So, uh So, uh So, uh the other way that this locked in was the other way that this locked in was the other way that this locked in was architecture. And we built this app in architecture. And we built this app in architecture. And we built this app in uh late 2024 uh late 2024 uh late 2024 uh when workflows were kind of the um uh when workflows were kind of the um uh when workflows were kind of the um gold standard if you wanted good uh gold standard if you wanted good uh gold standard if you wanted good uh production results, and uh the AI world production results, and uh the AI world production results, and uh the AI world moves very fast, and we couldn't adapt moves very fast, and we couldn't adapt moves very fast, and we couldn't adapt to that because we were so locked into to that because we were so locked into to that because we were so locked into this, just trying to keep it running. this, just trying to keep it running. this, just trying to keep it running. And we couldn't take advantage of the And we couldn't take advantage of the And we couldn't take advantage of the new architectures um new architectures um new architectures um and and improve performance that way. and and improve performance that way. and and improve performance that way. So, earlier this year I had an aha So, earlier this year I had an aha So, earlier this year I had an aha moment. Um we started using Claude Code moment. Um we started using Claude Code moment. Um we started using Claude Code for our coding tasks, and for our coding tasks, and for our coding tasks, and I noticed that I noticed that I noticed that we never needed to change the model we never needed to change the model we never needed to change the model depending on what task we're using. Um depending on what task we're using. Um depending on what task we're using. Um we just changed the skill, the resources we just changed the skill, the resources we just changed the skill, the resources that we passed it, the context.

that we passed it, the context. that we passed it, the context. Um you drop in the better better Um you drop in the better better Um you drop in the better better context, you get better results. context, you get better results. context, you get better results. And I thought, And I thought, And I thought, "Why can't we do this with our messaging "Why can't we do this with our messaging "Why can't we do this with our messaging app?" app?" app?" Um Um Um this was obviously difficult for me to this was obviously difficult for me to this was obviously difficult for me to admit cuz I was the champion for admit cuz I was the champion for admit cuz I was the champion for fine-tuning and uh fine-tuning and uh fine-tuning and uh luckily, we were able to piggyback on a luckily, we were able to piggyback on a luckily, we were able to piggyback on a project that was already happening um project that was already happening um project that was already happening um and so we and so we and so we migrated our workflow approach to a migrated our workflow approach to a migrated our workflow approach to a series of skills, and tools, and series of skills, and tools, and series of skills, and tools, and resources that the skills could could resources that the skills could could resources that the skills could could load into or load up um and get that load into or load up um and get that load into or load up um and get that context. context. context. And so we pushed this as one of our And so we pushed this as one of our And so we pushed this as one of our first production tests of our our new first production tests of our our new first production tests of our our new agentic framework that that was being agentic framework that that was being agentic framework that that was being built already. built already. built already. Now, uh I want to compare the process Now, uh I want to compare the process Now, uh I want to compare the process before and after our rebuild. Uh before, before and after our rebuild. Uh before, before and after our rebuild. Uh before, we already went through the kind of the we already went through the kind of the we already went through the kind of the training cycle, but there was this training cycle, but there was this training cycle, but there was this triage cycle beforehand where we needed triage cycle beforehand where we needed triage cycle beforehand where we needed to make sure that we had reached a to make sure that we had reached a to make sure that we had reached a critical mass of problems before we critical mass of problems before we critical mass of problems before we would even attempt to uh fine-tune again would even attempt to uh fine-tune again would even attempt to uh fine-tune again to improve everything.

to improve everything. to improve everything. Like I said, this took about a week, so Like I said, this took about a week, so Like I said, this took about a week, so it was a long process, costly. Uh after it was a long process, costly. Uh after it was a long process, costly. Uh after the rebuild, the rebuild, the rebuild, um it was a simple process of you find a um it was a simple process of you find a um it was a simple process of you find a problem, problem, problem, you adjust the simp system prompt or the you adjust the simp system prompt or the you adjust the simp system prompt or the skill that was affected, skill that was affected, skill that was affected, we validated performance on a curated we validated performance on a curated we validated performance on a curated set that we had been collecting over the set that we had been collecting over the set that we had been collecting over the time that this was in production. We time that this was in production. We time that this was in production. We iterate a few times, and then we deploy iterate a few times, and then we deploy iterate a few times, and then we deploy that simply by that simply by that simply by uploading MD files to an S3 bucket. Um uploading MD files to an S3 bucket. Um uploading MD files to an S3 bucket. Um this whole process from discovering a this whole process from discovering a this whole process from discovering a problem to deploying the fix, we reduced problem to deploying the fix, we reduced problem to deploying the fix, we reduced down to less than an hour. down to less than an hour. down to less than an hour. So, it extremely improved all of this, So, it extremely improved all of this, So, it extremely improved all of this, and we could be far more reactive, give and we could be far more reactive, give and we could be far more reactive, give our our customers way better our our customers way better our our customers way better performance, or better experience there. performance, or better experience there. performance, or better experience there. Now, I'll be honest, it did cost us a Now, I'll be honest, it did cost us a Now, I'll be honest, it did cost us a little bit more per message. We were little bit more per message. We were little bit more per message. We were using better models. Um using better models. Um using better models. Um so, the API costs were a little higher.

so, the API costs were a little higher. so, the API costs were a little higher. But, But, But, accuracy went way up. accuracy went way up. accuracy went way up. I said before that accuracy was the key I said before that accuracy was the key I said before that accuracy was the key to to getting all of this right. Um and to to getting all of this right. Um and to to getting all of this right. Um and we did that. Accuracy uh was far better we did that. Accuracy uh was far better we did that. Accuracy uh was far better with this than it ever was with with this than it ever was with with this than it ever was with fine-tuning. fine-tuning. fine-tuning. Um next, like I said, we reduced Um next, like I said, we reduced Um next, like I said, we reduced our uh our fixed process from days down our uh our fixed process from days down our uh our fixed process from days down to minutes. to minutes. to minutes. Next, we were able to unfreeze our model Next, we were able to unfreeze our model Next, we were able to unfreeze our model and finally get that freedom from a and finally get that freedom from a and finally get that freedom from a vendor that we never had with vendor that we never had with vendor that we never had with fine-tuning. Um the our agentic fine-tuning. Um the our agentic fine-tuning. Um the our agentic framework was built framework was built framework was built model agnostic, so we could use OpenAI, model agnostic, so we could use OpenAI, model agnostic, so we could use OpenAI, we can use uh Anthropic, we can use any we can use uh Anthropic, we can use any we can use uh Anthropic, we can use any other model that we want. The important other model that we want. The important other model that we want. The important part is the context that we're providing part is the context that we're providing part is the context that we're providing to that model. to that model. to that model. And then lastly, And then lastly, And then lastly, while it cost us a little more per while it cost us a little more per while it cost us a little more per message, the total cost went down message, the total cost went down message, the total cost went down because we were spending far less time because we were spending far less time because we were spending far less time trying to keep it up and running, and trying to keep it up and running, and trying to keep it up and running, and fine-tuning to uh to keep it working fine-tuning to uh to keep it working fine-tuning to uh to keep it working properly.

properly. properly. So, So, So, before you fine-tune, I'd ask you, can before you fine-tune, I'd ask you, can before you fine-tune, I'd ask you, can you cross your reasons off of this list? you cross your reasons off of this list? you cross your reasons off of this list? So, I thought we would get better So, I thought we would get better So, I thought we would get better accuracy. accuracy. accuracy. Uh the rebuild Uh the rebuild Uh the rebuild beat the fine-tuned model. Um beat the fine-tuned model. Um beat the fine-tuned model. Um I thought we would get lower cost at the I thought we would get lower cost at the I thought we would get lower cost at the volume we were doing. Um I was looking volume we were doing. Um I was looking volume we were doing. Um I was looking at the wrong costs. at the wrong costs. at the wrong costs. We we paid more per message, but the We we paid more per message, but the We we paid more per message, but the total cost ended up going down with our total cost ended up going down with our total cost ended up going down with our rebuild. rebuild. rebuild. Lower latency. We We did see marginal Lower latency. We We did see marginal Lower latency. We We did see marginal gains on the these smaller models, but gains on the these smaller models, but gains on the these smaller models, but they were so small that in practice it they were so small that in practice it they were so small that in practice it really didn't make any difference. really didn't make any difference. really didn't make any difference. And then maybe you've got a narrow or And then maybe you've got a narrow or And then maybe you've got a narrow or structured task. structured task. structured task. Our textbook case still became tech Our textbook case still became tech Our textbook case still became tech debt. debt. debt. Um Um Um And lastly, vendor control. It's not as And lastly, vendor control. It's not as And lastly, vendor control. It's not as simple as just plugging the data in. simple as just plugging the data in. simple as just plugging the data in. The other two situations where you you The other two situations where you you The other two situations where you you might have privacy and data control or might have privacy and data control or might have privacy and data control or you need some off-line off-line you need some off-line off-line you need some off-line off-line solution.

solution. solution. I would say this these are the I would say this these are the I would say this these are the situations where a fine-tuned model may situations where a fine-tuned model may situations where a fine-tuned model may be useful, be useful, be useful, but you need to be cautious. There are but you need to be cautious. There are but you need to be cautious. There are other solutions out there. other solutions out there. other solutions out there. Um Um Um but but but um um um you need to make sure that it's not uh you need to make sure that it's not uh you need to make sure that it's not uh not causing issues in the long run. not causing issues in the long run. not causing issues in the long run. So, finally, So, finally, So, finally, fine-tune only when you literally cannot fine-tune only when you literally cannot fine-tune only when you literally cannot call a frontier model. And even then, call a frontier model. And even then, call a frontier model. And even then, your decision still has to beat the the your decision still has to beat the the your decision still has to beat the the tax. tax. tax. Thank you.