Why Off-the-Shelf AI Doesn't Understand Money — Udi Menkes, Intuit

So, I have a three-year-old daughter and So, I have a three-year-old daughter and she's abs absolutely adorable and the she's abs absolutely adorable and the she's abs absolutely adorable and the parents here in the room know how parents here in the room know how parents here in the room know how insightful that age can be and she has a insightful that age can be and she has a insightful that age can be and she has a complete theory about money by now. And complete theory about money by now. And complete theory about money by now. And I'll give you an example. So, a couple I'll give you an example. So, a couple I'll give you an example. So, a couple weeks ago, I was driving the car. I was weeks ago, I was driving the car. I was weeks ago, I was driving the car. I was coming into park and there was something coming into park and there was something coming into park and there was something in my dead end and I scratched the car. in my dead end and I scratched the car. in my dead end and I scratched the car. I went out. I'm like, "Oh man, I can't I went out. I'm like, "Oh man, I can't I went out. I'm like, "Oh man, I can't believe I scratched the car." And then I believe I scratched the car." And then I believe I scratched the car." And then I hear my daughter from the back and she hear my daughter from the back and she hear my daughter from the back and she was like, "Daddy, what's happened?" And, was like, "Daddy, what's happened?" And, was like, "Daddy, what's happened?" And, you know, I'm explaining it to her and you know, I'm explaining it to her and you know, I'm explaining it to her and then she says, "What's the problem? Just then she says, "What's the problem? Just then she says, "What's the problem? Just buy another one." buy another one." buy another one." So anyway, um I want to ask you today So anyway, um I want to ask you today So anyway, um I want to ask you today with a raise of hand, who uses LLMs, has with a raise of hand, who uses LLMs, has with a raise of hand, who uses LLMs, has used LLMs for getting financial advice, used LLMs for getting financial advice, used LLMs for getting financial advice, a recommendation on something in the a recommendation on something in the a recommendation on something in the financial world. Great. Almost everyone. financial world. Great. Almost everyone. financial world. Great. Almost everyone. Wait, keep your hand up if you trusted Wait, keep your hand up if you trusted Wait, keep your hand up if you trusted the answer and you actually followed the the answer and you actually followed the the answer and you actually followed the advice.

advice. advice. Okay. Okay. Okay. a lot of hands are going down and that's a lot of hands are going down and that's a lot of hands are going down and that's the core problem. So I had the same the core problem. So I had the same the core problem. So I had the same thing a couple a couple uh months ago. I thing a couple a couple uh months ago. I thing a couple a couple uh months ago. I had a big decision I was looking to had a big decision I was looking to had a big decision I was looking to take. Should I invest in you know real take. Should I invest in you know real take. Should I invest in you know real estate niche or in the stock market estate niche or in the stock market estate niche or in the stock market niche on a specific area and you know I niche on a specific area and you know I niche on a specific area and you know I do AI for finance for a living. So I do AI for finance for a living. So I do AI for finance for a living. So I went the full-blown way. context, brain, went the full-blown way. context, brain, went the full-blown way. context, brain, um, the books, the knowledge, all my um, the books, the knowledge, all my um, the books, the knowledge, all my finances combined, all the latest finances combined, all the latest finances combined, all the latest models, and it gave me a recommendation. models, and it gave me a recommendation. models, and it gave me a recommendation. You should do A with great reasoning. You should do A with great reasoning. You should do A with great reasoning. And then I changed just a little bit, And then I changed just a little bit, And then I changed just a little bit, one of the assumptions, and it one of the assumptions, and it one of the assumptions, and it completely flipped. You should do B, completely flipped. You should do B, completely flipped. You should do B, never do A. And then I tweaked one more never do A. And then I tweaked one more never do A. And then I tweaked one more small thing, and it went all the way small thing, and it went all the way small thing, and it went all the way back to A. back to A. back to A. And at that moment, I understood that And at that moment, I understood that And at that moment, I understood that the advice sounds good. It sounds sound, the advice sounds good. It sounds sound, the advice sounds good. It sounds sound, but I can't really trust it. And by the but I can't really trust it. And by the but I can't really trust it. And by the end of the talk today, you will end of the talk today, you will end of the talk today, you will understand why off-the-shelf LLMs don't understand why off-the-shelf LLMs don't understand why off-the-shelf LLMs don't understand money and what you need to do understand money and what you need to do understand money and what you need to do about it.

So, I'm going to show you a couple of So, I'm going to show you a couple of real examples from a study we're doing real examples from a study we're doing real examples from a study we're doing at into it on thousands and thousands of at into it on thousands and thousands of at into it on thousands and thousands of businesses around a 100,000 situations businesses around a 100,000 situations businesses around a 100,000 situations and time frames. This is an example of a and time frames. This is an example of a and time frames. This is an example of a small business, a new landlord that is small business, a new landlord that is small business, a new landlord that is building a rental property business. His building a rental property business. His building a rental property business. His first property and he's down. He's he's first property and he's down. He's he's first property and he's down. He's he's in negative cash flow. there's a open in negative cash flow. there's a open in negative cash flow. there's a open loan and the profit is basically loan and the profit is basically loan and the profit is basically trending into the red and a question trending into the red and a question trending into the red and a question comes up. How do I improve my profit? comes up. How do I improve my profit? comes up. How do I improve my profit? And a frontier model gives the following And a frontier model gives the following And a frontier model gives the following response. Go and acquire a second rental response. Go and acquire a second rental response. Go and acquire a second rental property because that'll bring more property because that'll bring more property because that'll bring more income and compensate for the deficit. income and compensate for the deficit. income and compensate for the deficit. And that model had all of the business's And that model had all of the business's And that model had all of the business's data. data. data. Now that's very risky for someone in the Now that's very risky for someone in the Now that's very risky for someone in the negative in the red to be doing. On the negative in the red to be doing. On the negative in the red to be doing. On the other hand, a model that is grounded in other hand, a model that is grounded in other hand, a model that is grounded in real outcomes and what I mean by that is real outcomes and what I mean by that is real outcomes and what I mean by that is a model that has seen similar situations a model that has seen similar situations a model that has seen similar situations of such businesses what they did and of such businesses what they did and of such businesses what they did and what was the outcome actually what was the outcome actually what was the outcome actually recommended to raise prices on the recommended to raise prices on the recommended to raise prices on the existing tenant by 5 to 10%.

existing tenant by 5 to 10%. existing tenant by 5 to 10%. And to do it before the renewal. And to do it before the renewal. And to do it before the renewal. Now, some of you are thinking that it's Now, some of you are thinking that it's Now, some of you are thinking that it's just a matter of context. Just give it just a matter of context. Just give it just a matter of context. Just give it more context. more context. more context. And the thing is that this advice is And the thing is that this advice is And the thing is that this advice is coming based off on real situations of coming based off on real situations of coming based off on real situations of similar businesses and would actually similar businesses and would actually similar businesses and would actually move them into profitability in this move them into profitability in this move them into profitability in this case. case. case. And this is not a one-off. I can go on And this is not a one-off. I can go on And this is not a one-off. I can go on and on showing you a lot of examples. and on showing you a lot of examples. and on showing you a lot of examples. This is second one. This is an egg This is second one. This is an egg This is second one. This is an egg supplier where one customer is 70% of supplier where one customer is 70% of supplier where one customer is 70% of the revenue and one vendor is almost all the revenue and one vendor is almost all the revenue and one vendor is almost all of its cost. Same question, how do I of its cost. Same question, how do I of its cost. Same question, how do I improve my profit? A frontier model says improve my profit? A frontier model says improve my profit? A frontier model says raise your prices 15 to 20%. Now you understand that is very risky Now you understand that is very risky because you might lose almost all your because you might lose almost all your because you might lose almost all your revenue. On the other hand, the same revenue. On the other hand, the same revenue. On the other hand, the same grounded model in real outcomes went grounded model in real outcomes went grounded model in real outcomes went actually to the cost site and actually to the cost site and actually to the cost site and recommended to negotiate recommended to negotiate recommended to negotiate um negotiate the vendor cost pricing for um negotiate the vendor cost pricing for um negotiate the vendor cost pricing for a 5 to 10% reduction. So it went to the a 5 to 10% reduction. So it went to the a 5 to 10% reduction. So it went to the cost side. It took into account the cost side. It took into account the cost side. It took into account the constraints. So what we saw are two constraints. So what we saw are two constraints. So what we saw are two examples of frontiers. I'm talking about examples of frontiers. I'm talking about examples of frontiers. I'm talking about leading models in the world today that leading models in the world today that leading models in the world today that had all the business context actually had all the business context actually had all the business context actually give advice that could be very harmful give advice that could be very harmful give advice that could be very harmful for the business.

for the business. for the business. And that's what I call the fluent bluff. And that's what I call the fluent bluff. And that's what I call the fluent bluff. The fluent bluff is a generic fluent and The fluent bluff is a generic fluent and The fluent bluff is a generic fluent and confident answer that frontier LLMs can confident answer that frontier LLMs can confident answer that frontier LLMs can give you around money because of what give you around money because of what give you around money because of what they learned on the internet, blogs, they learned on the internet, blogs, they learned on the internet, blogs, books, advice columns, what people wrote books, advice columns, what people wrote books, advice columns, what people wrote about money, but not based on what about money, but not based on what about money, but not based on what actually happened. And I would argue actually happened. And I would argue actually happened. And I would argue that almost every answer that you see that almost every answer that you see that almost every answer that you see related to money and finances is such related to money and finances is such related to money and finances is such and we're soon going to release re the and we're soon going to release re the and we're soon going to release re the research I talked about but I'll give research I talked about but I'll give research I talked about but I'll give you a highlight from there you a highlight from there you a highlight from there across these 100,000 businesses in time across these 100,000 businesses in time across these 100,000 businesses in time frames 40% of the time the essence of frames 40% of the time the essence of frames 40% of the time the essence of the advice that the frontier models were the advice that the frontier models were the advice that the frontier models were giving was acquire a new customer giving was acquire a new customer giving was acquire a new customer which which is you know everyone would which which is you know everyone would which which is you know everyone would wish they could do that and 14 more wish they could do that and 14 more wish they could do that and 14 more additional percent were increase additional percent were increase additional percent were increase basically the revenue from your product.

basically the revenue from your product. basically the revenue from your product. So combined more than half of the So combined more than half of the So combined more than half of the essence of the advice that was given by essence of the advice that was given by essence of the advice that was given by the frontier models was acquire new the frontier models was acquire new the frontier models was acquire new customers and try to increase revenue customers and try to increase revenue customers and try to increase revenue from the profit. from the profit. from the profit. And this is not just me saying this is a And this is not just me saying this is a And this is not just me saying this is a very interesting research coming that very interesting research coming that very interesting research coming that just came out a couple weeks ago from just came out a couple weeks ago from just came out a couple weeks ago from researchers at Princeton. What they researchers at Princeton. What they researchers at Princeton. What they tried to do is simulate and answer the tried to do is simulate and answer the tried to do is simulate and answer the question can the leading models drive question can the leading models drive question can the leading models drive long horizon business decisions. long horizon business decisions. long horizon business decisions. And what they did is they gave the And what they did is they gave the And what they did is they gave the models a harness with tools and data and models a harness with tools and data and models a harness with tools and data and everything they would need to take everything they would need to take everything they would need to take decisions across a simulation of 500 decisions across a simulation of 500 decisions across a simulation of 500 days. days. days. Can they turn a profit? Each model got a Can they turn a profit? Each model got a Can they turn a profit? Each model got a million dollars to start with and guess million dollars to start with and guess million dollars to start with and guess what happened? what happened? what happened? Most of the models drove the company Most of the models drove the company Most of the models drove the company bankrupt and it didn't even take 500 bankrupt and it didn't even take 500 bankrupt and it didn't even take 500 days. days. days. And the interesting part is that they And the interesting part is that they And the interesting part is that they also ran a simple rules-based system and also ran a simple rules-based system and also ran a simple rules-based system and that rules-based system that rules-based system that rules-based system outbeat almost all of the models.

outbeat almost all of the models. outbeat almost all of the models. Even the very very few models that were Even the very very few models that were Even the very very few models that were able to generate some profit, it was in able to generate some profit, it was in able to generate some profit, it was in a specific in instance and when you a specific in instance and when you a specific in instance and when you rerun that they also actually drove rerun that they also actually drove rerun that they also actually drove bankruptcy. So how can it be that a bankruptcy. So how can it be that a bankruptcy. So how can it be that a simple rule-based system outbeats the simple rule-based system outbeats the simple rule-based system outbeats the frontier models today on real business frontier models today on real business frontier models today on real business decisions? And here's what I want you to think And here's what I want you to think about. A frontier model has read about about. A frontier model has read about about. A frontier model has read about money, money, money, but a grounded model in real outcome has but a grounded model in real outcome has but a grounded model in real outcome has actually watched what happens. actually watched what happens. actually watched what happens. And let me be precise with my argue And let me be precise with my argue And let me be precise with my argue here. Even if you take all of a here. Even if you take all of a here. Even if you take all of a company's data and for example we at company's data and for example we at company's data and for example we at into it have all the financial data from into it have all the financial data from into it have all the financial data from QuickBooks for example the general QuickBooks for example the general QuickBooks for example the general ledger the P&L the cash flows everything ledger the P&L the cash flows everything ledger the P&L the cash flows everything the invoices of the business and you the invoices of the business and you the invoices of the business and you give it to a frontier LLM it's still give it to a frontier LLM it's still give it to a frontier LLM it's still just one group of data points on a just one group of data points on a just one group of data points on a company company company and that's a difference between sounding and that's a difference between sounding and that's a difference between sounding right and actually being right right and actually being right right and actually being right so I'm Udy Menquez and I've been in the so I'm Udy Menquez and I've been in the so I'm Udy Menquez and I've been in the AI and finance world for the past 15 AI and finance world for the past 15 AI and finance world for the past 15 years. I started in the AI science world years. I started in the AI science world years. I started in the AI science world leading AI and data teams and shifted leading AI and data teams and shifted leading AI and data teams and shifted into product management becoming an into product management becoming an into product management becoming an about four years ago an AI princ AI about four years ago an AI princ AI about four years ago an AI princ AI product manager at in it long before by product manager at in it long before by product manager at in it long before by the way it was cool to become an AIPM the way it was cool to become an AIPM the way it was cool to become an AIPM um and today I'm a principal product

um and today I'm a principal product um and today I'm a principal product manager at into it. lead financial manager at into it. lead financial manager at into it. lead financial intelligence and advisory systems that intelligence and advisory systems that intelligence and advisory systems that help into its customers to take better help into its customers to take better help into its customers to take better decisions and grow their business. And decisions and grow their business. And decisions and grow their business. And the question that I fixate on on a daily the question that I fixate on on a daily the question that I fixate on on a daily basis is not which model is the best now basis is not which model is the best now basis is not which model is the best now that I I can use. It's what do we that I I can use. It's what do we that I I can use. It's what do we fundamentally have that no model access fundamentally have that no model access fundamentally have that no model access can replicate and how can we can I can replicate and how can we can I can replicate and how can we can I transform that into AI native transform that into AI native transform that into AI native transformative and delightful transformative and delightful transformative and delightful experiences for our customers. experiences for our customers. experiences for our customers. Now I want to develop some intuition Now I want to develop some intuition Now I want to develop some intuition for from three different angles on why for from three different angles on why for from three different angles on why these models bluff. So the first angle these models bluff. So the first angle these models bluff. So the first angle is around context is not experience. And is around context is not experience. And is around context is not experience. And I'll give you another real example. This I'll give you another real example. This I'll give you another real example. This is an apparel company where 80% is an apparel company where 80% is an apparel company where 80% of the cost is coming from this one of the cost is coming from this one of the cost is coming from this one vendor. And the textbook answer is go vendor. And the textbook answer is go vendor. And the textbook answer is go cut your biggest cost. Right? Textbook cut your biggest cost. Right? Textbook cut your biggest cost. Right? Textbook book answer. But the issue here is you book answer. But the issue here is you book answer. But the issue here is you cut this cost. That same vendor was cut this cost. That same vendor was cut this cost. That same vendor was actually enabling the generation of 97% actually enabling the generation of 97% actually enabling the generation of 97% of that company's revenue. So cut that of that company's revenue. So cut that of that company's revenue. So cut that biggest cost biggest cost biggest cost and you lose almost all the revenue. And and you lose almost all the revenue. And and you lose almost all the revenue. And that can be great margins on zero that can be great margins on zero that can be great margins on zero dollars.

dollars. dollars. And think about it. If I give you an And think about it. If I give you an And think about it. If I give you an option to work with two two different option to work with two two different option to work with two two different adviserss, one adviser is a very adviserss, one adviser is a very adviserss, one adviser is a very experienced one, years of experience experienced one, years of experience experienced one, years of experience working with businesses guiding them and working with businesses guiding them and working with businesses guiding them and another adviser which is very very another adviser which is very very another adviser which is very very smart. They know all the textbook. smart. They know all the textbook. smart. They know all the textbook. They're fresh. They read everything. They're fresh. They read everything. They're fresh. They read everything. They know AI in and out. but you don't They know AI in and out. but you don't They know AI in and out. but you don't have experience, I would bet you would have experience, I would bet you would have experience, I would bet you would always go with the experienced one. always go with the experienced one. always go with the experienced one. And that's the same thing with the And that's the same thing with the And that's the same thing with the models in what I just showed you. models in what I just showed you. models in what I just showed you. And it turns out experience is very hard And it turns out experience is very hard And it turns out experience is very hard to measure. to measure. to measure. And I'll give you an example. So let's And I'll give you an example. So let's And I'll give you an example. So let's look at a restaurant. A restaurant, look at a restaurant. A restaurant, look at a restaurant. A restaurant, let's say, raises price and after 6 let's say, raises price and after 6 let's say, raises price and after 6 months becomes much more profitable. Is months becomes much more profitable. Is months becomes much more profitable. Is it because they raise prices or is it it because they raise prices or is it it because they raise prices or is it because they're just naturally because they're just naturally because they're just naturally successful? And the challenge is successful? And the challenge is successful? And the challenge is obviously you can't run the business obviously you can't run the business obviously you can't run the business twice, right? twice, right? twice, right? So what you do is you take two groups of So what you do is you take two groups of So what you do is you take two groups of similar companies, similar businesses similar companies, similar businesses similar companies, similar businesses that have the same propensity to raise that have the same propensity to raise that have the same propensity to raise prices, the same likelihood to raise prices, the same likelihood to raise prices, the same likelihood to raise prices. One group raised prices while prices. One group raised prices while prices. One group raised prices while the other didn't. And then we look after the other didn't. And then we look after the other didn't. And then we look after some time at the results. So, the group some time at the results. So, the group some time at the results. So, the group that raised prices actually gained that raised prices actually gained that raised prices actually gained $4,200 a day profit. And the group that $4,200 a day profit. And the group that $4,200 a day profit. And the group that didn't raise prices actually gained didn't raise prices actually gained didn't raise prices actually gained $2,800 a day. So, the question is, what $2,800 a day. So, the question is, what $2,800 a day. So, the question is, what is the impact of raising prices? So, a

is the impact of raising prices? So, a is the impact of raising prices? So, a naive answer would be the difference, naive answer would be the difference, naive answer would be the difference, right? 1,400 is the impact of raising right? 1,400 is the impact of raising right? 1,400 is the impact of raising prices. But actually, you need to prices. But actually, you need to prices. But actually, you need to account for the fact that the companies account for the fact that the companies account for the fact that the companies that raise prices are actually naturally that raise prices are actually naturally that raise prices are actually naturally more successful businesses, which is more successful businesses, which is more successful businesses, which is also why they could raise the prices. also why they could raise the prices. also why they could raise the prices. So, the real difference is more like So, the real difference is more like So, the real difference is more like $1,150 $1,150 $1,150 for this illustration. And we measure for this illustration. And we measure for this illustration. And we measure the impact of actions on the outcome the impact of actions on the outcome the impact of actions on the outcome through a measure called Kate, through a measure called Kate, through a measure called Kate, conditional average treatment error, conditional average treatment error, conditional average treatment error, which looks at that connection. which looks at that connection. which looks at that connection. And here's what I want all the AI and And here's what I want all the AI and And here's what I want all the AI and finance leaders here in the room to pay finance leaders here in the room to pay finance leaders here in the room to pay attention to. attention to. attention to. So find where you can see a lot of So find where you can see a lot of So find where you can see a lot of different situations across entities different situations across entities different situations across entities that you have in your data. In our case, that you have in your data. In our case, that you have in your data. In our case, it's businesses what they did and verify it's businesses what they did and verify it's businesses what they did and verify the outcomes and how things turned out. the outcomes and how things turned out. the outcomes and how things turned out. If you can see that in the data because If you can see that in the data because If you can see that in the data because that's the one thing that Frontier that's the one thing that Frontier that's the one thing that Frontier off-the-shelf models do not have.

off-the-shelf models do not have. off-the-shelf models do not have. And And And don't get me wrong, Frontier models are don't get me wrong, Frontier models are don't get me wrong, Frontier models are amazing and we actually use them. And amazing and we actually use them. And amazing and we actually use them. And I'll I'll show you how we use them. So, I'll I'll show you how we use them. So, I'll I'll show you how we use them. So, we use them to generate hypothesis we use them to generate hypothesis we use them to generate hypothesis candidates for actions we would suggest candidates for actions we would suggest candidates for actions we would suggest a business to do. But then we would use a business to do. But then we would use a business to do. But then we would use a model that we trained using a model that we trained using a model that we trained using reinforcement learning in order to reinforcement learning in order to reinforcement learning in order to figure out which one of those is figure out which one of those is figure out which one of those is actually a right move to do versus a actually a right move to do versus a actually a right move to do versus a mistake that could drive the business mistake that could drive the business mistake that could drive the business down. down. down. Now how we do it a little bit into our Now how we do it a little bit into our Now how we do it a little bit into our approach is we look at what we call we approach is we look at what we call we approach is we look at what we call we actually create from the data millions actually create from the data millions actually create from the data millions of business trajectories. So we have of business trajectories. So we have of business trajectories. So we have data add into it across our products data add into it across our products data add into it across our products QuickBooks, Turboax, Credit Karma, QuickBooks, Turboax, Credit Karma, QuickBooks, Turboax, Credit Karma, Mailchimp. So think about a business and Mailchimp. So think about a business and Mailchimp. So think about a business and the financial data. There's the general the financial data. There's the general the financial data. There's the general ledger, the P&L, the cash flow ledger, the P&L, the cash flow ledger, the P&L, the cash flow statements, the invoices like I statements, the invoices like I statements, the invoices like I mentioned before. So we take all of that mentioned before. So we take all of that mentioned before. So we take all of that data and we create what we call business data and we create what we call business data and we create what we call business states. A state of a business is think states. A state of a business is think states. A state of a business is think about a very detailed summary at a given about a very detailed summary at a given about a very detailed summary at a given point of time and then we derive point of time and then we derive point of time and then we derive actions. So for example in your general actions. So for example in your general actions. So for example in your general ledger I can look and see that you have ledger I can look and see that you have ledger I can look and see that you have invested in a campaign in a marketing invested in a campaign in a marketing invested in a campaign in a marketing campaign or you paid someone. So I know campaign or you paid someone. So I know campaign or you paid someone. So I know you're paying payroll. I know how much you're paying payroll. I know how much you're paying payroll. I know how much um um your hiring costs are and so on.

um um your hiring costs are and so on. um um your hiring costs are and so on. So we derive all of these actions So we derive all of these actions So we derive all of these actions and we look at what are the outcomes in and we look at what are the outcomes in and we look at what are the outcomes in different time frames and outcomes can different time frames and outcomes can different time frames and outcomes can be increase in profit and revenue in be increase in profit and revenue in be increase in profit and revenue in cash flow in time combination of those. cash flow in time combination of those. cash flow in time combination of those. So we create millions of vectors of So we create millions of vectors of So we create millions of vectors of state, action and outcome and then we state, action and outcome and then we state, action and outcome and then we train an R model to be able to train an R model to be able to train an R model to be able to understand in given situations of understand in given situations of understand in given situations of similar businesses which actions lead to similar businesses which actions lead to similar businesses which actions lead to the best outcomes. the best outcomes. the best outcomes. And then the third step is we actually And then the third step is we actually And then the third step is we actually train an LLM to be able to generate that train an LLM to be able to generate that train an LLM to be able to generate that better advice. better advice. better advice. That's where opinions are going in and That's where opinions are going in and That's where opinions are going in and evidence is going out. So the model you evidence is going out. So the model you evidence is going out. So the model you saw in the examples at the beginning saw in the examples at the beginning saw in the examples at the beginning were actually a model that we developed were actually a model that we developed were actually a model that we developed with researchers at into it across with researchers at into it across with researchers at into it across millions of small and medium businesses. millions of small and medium businesses. millions of small and medium businesses. And we actually tested it head-to-head And we actually tested it head-to-head And we actually tested it head-to-head with all of the leading models in the with all of the leading models in the with all of the leading models in the world. And we were able with a midsize world. And we were able with a midsize world. And we were able with a midsize cheaper model to outperform the frontier cheaper model to outperform the frontier cheaper model to outperform the frontier models models models because of the grounding that I just because of the grounding that I just because of the grounding that I just showed you. And the interesting part as showed you. And the interesting part as showed you. And the interesting part as a product person you would think that a product person you would think that a product person you would think that it's all about the model size and the it's all about the model size and the it's all about the model size and the bigger and better model obviously I bigger and better model obviously I bigger and better model obviously I would have a lot better chance. But it would have a lot better chance. But it would have a lot better chance. But it doesn't turn out to be true. And the doesn't turn out to be true. And the doesn't turn out to be true. And the moat here is that it's not about the moat here is that it's not about the moat here is that it's not about the model access, it's about the data itself model access, it's about the data itself model access, it's about the data itself that you have.

And then we went ahead and built an And then we went ahead and built an experience out of it. And this is an AI experience out of it. And this is an AI experience out of it. And this is an AI business advisor that is currently in business advisor that is currently in business advisor that is currently in beta with research in a research preview beta with research in a research preview beta with research in a research preview with our customers where we use the LLM with our customers where we use the LLM with our customers where we use the LLM that I just described that we created to that I just described that we created to that I just described that we created to proactively raise opportunities for proactively raise opportunities for proactively raise opportunities for businesses at every given point of time. businesses at every given point of time. businesses at every given point of time. Here's what you should do. Here is why. Here's what you should do. Here is why. Here's what you should do. Here is why. Grounded in who is like you, who's Grounded in who is like you, who's Grounded in who is like you, who's similar to you, what they did, and why similar to you, what they did, and why similar to you, what they did, and why we're actually recommending you to do we're actually recommending you to do we're actually recommending you to do it. And you can drill down into it, it. And you can drill down into it, it. And you can drill down into it, understand, and create action plans that understand, and create action plans that understand, and create action plans that will lead to your business actually will lead to your business actually will lead to your business actually growing in the right direction. growing in the right direction. growing in the right direction. Now, I want to take a step back and zoom Now, I want to take a step back and zoom Now, I want to take a step back and zoom out because this isn't just about money. out because this isn't just about money. out because this isn't just about money. We are entering the era of outcomedriven We are entering the era of outcomedriven We are entering the era of outcomedriven AI. And the question stops being which AI. And the question stops being which AI. And the question stops being which model is better and starts becoming how model is better and starts becoming how model is better and starts becoming how can we steer AI to actually make it can we steer AI to actually make it can we steer AI to actually make it achieve the outcomes we want in our achieve the outcomes we want in our achieve the outcomes we want in our domains. And it doesn't matter if you're domains. And it doesn't matter if you're domains. And it doesn't matter if you're building an anti-fraud system or a building an anti-fraud system or a building an anti-fraud system or a health care system, logistics developer health care system, logistics developer health care system, logistics developer tools.

tools. tools. The winners in my opinion are going to The winners in my opinion are going to The winners in my opinion are going to be those with the best system of be those with the best system of be those with the best system of records, creating unique data sets out records, creating unique data sets out records, creating unique data sets out of them and then training the models to of them and then training the models to of them and then training the models to achieve the outcomes. And as the product person here, it's not And as the product person here, it's not just about the science. The science is just about the science. The science is just about the science. The science is very important. But a great adviser, very important. But a great adviser, very important. But a great adviser, think about the great adviserss and think about the great adviserss and think about the great adviserss and mentors that you had in your life. They mentors that you had in your life. They mentors that you had in your life. They understand you, right? They understand understand you, right? They understand understand you, right? They understand your preferences, what you like, what your preferences, what you like, what your preferences, what you like, what you don't like. So, a great advisory you don't like. So, a great advisory you don't like. So, a great advisory experience needs to have two things. It experience needs to have two things. It experience needs to have two things. It needs to have great grounded science, needs to have great grounded science, needs to have great grounded science, the best science, but also it needs to the best science, but also it needs to the best science, but also it needs to understand you, what you prefer, and it understand you, what you prefer, and it understand you, what you prefer, and it needs to even make you feel as if you needs to even make you feel as if you needs to even make you feel as if you were part of the decision to create a were part of the decision to create a were part of the decision to create a trusted experience. So, three things I want you to remember So, three things I want you to remember today. today. today. Every model has read about your domain, Every model has read about your domain, Every model has read about your domain, but none has actually watched but none has actually watched but none has actually watched the plays and the moves and their the plays and the moves and their the plays and the moves and their outcomes. And that gap is the whole outcomes. And that gap is the whole outcomes. And that gap is the whole game. Now, in coding agents and coding game. Now, in coding agents and coding game. Now, in coding agents and coding models, we're seeing it very advanced. A models, we're seeing it very advanced. A models, we're seeing it very advanced. A lot of verified outcomes and creating lot of verified outcomes and creating lot of verified outcomes and creating models that actually lead to better models that actually lead to better models that actually lead to better outcomes in coding, but it's very much outcomes in coding, but it's very much outcomes in coding, but it's very much unexplored in the financial domain and unexplored in the financial domain and unexplored in the financial domain and in other domains as well. And you don't in other domains as well. And you don't in other domains as well. And you don't close the gap with bigger models. You close the gap with bigger models. You close the gap with bigger models. You close the gap with experience, embedding

close the gap with experience, embedding close the gap with experience, embedding experience into the model by looking at experience into the model by looking at experience into the model by looking at verified outcomes in your data. What verified outcomes in your data. What verified outcomes in your data. What actually worked at scale. actually worked at scale. actually worked at scale. So off-the-shelf models don't understand So off-the-shelf models don't understand So off-the-shelf models don't understand money, but grounded in real outcomes, it money, but grounded in real outcomes, it money, but grounded in real outcomes, it does. And that's what we were able to does. And that's what we were able to does. And that's what we were able to figure out. figure out. figure out. So, here's the one thing I want you to So, here's the one thing I want you to So, here's the one thing I want you to do tomorrow. do tomorrow. do tomorrow. Well, actually, you know what? Go ahead Well, actually, you know what? Go ahead Well, actually, you know what? Go ahead and enjoy Fourth of July weekend. But and enjoy Fourth of July weekend. But and enjoy Fourth of July weekend. But right after that, right after that, right after that, look in your data where you can see look in your data where you can see look in your data where you can see situations situations situations across entities and outcomes you can across entities and outcomes you can across entities and outcomes you can verify. verify. verify. Get deep into that data and start Get deep into that data and start Get deep into that data and start grounding your AI in that. Think about grounding your AI in that. Think about grounding your AI in that. Think about those angles those angles those angles and that's for you to build. and that's for you to build. and that's for you to build. Thank you very much. Thank you for Thank you very much. Thank you for Thank you very much. Thank you for listening to me. Happy to connect listening to me. Happy to connect listening to me. Happy to connect LinkedIn, Twitter, in the hallway. Thank LinkedIn, Twitter, in the hallway. Thank LinkedIn, Twitter, in the hallway. Thank you very much.