How Kepler Built Verifiable AI for Financial Services — Vinoo Ganesh

>> Hey everyone, thank you for being here. >> Hey everyone, thank you for being here. My name is Venu Ganesh and I'm the CEO My name is Venu Ganesh and I'm the CEO My name is Venu Ganesh and I'm the CEO and co-founder of Kepler. and co-founder of Kepler. and co-founder of Kepler. Today, I'm going to talk to you about Today, I'm going to talk to you about Today, I'm going to talk to you about how we built verifiable AI for financial how we built verifiable AI for financial how we built verifiable AI for financial services. services. services. First, a little about me. My career has First, a little about me. My career has First, a little about me. My career has been working in fairly been working in fairly been working in fairly difficult places to work in terms of difficult places to work in terms of difficult places to work in terms of numerical accuracy and verifiability. numerical accuracy and verifiability. numerical accuracy and verifiability. Began my career at Palantir where I led Began my career at Palantir where I led Began my career at Palantir where I led the compute platform as well as a lot of the compute platform as well as a lot of the compute platform as well as a lot of our USG engagements. Built and sold a our USG engagements. Built and sold a our USG engagements. Built and sold a alt data startup. Then was head of alt data startup. Then was head of alt data startup. Then was head of business engineering at Citadel. I have business engineering at Citadel. I have business engineering at Citadel. I have some Citadel colleagues here in the some Citadel colleagues here in the some Citadel colleagues here in the audience as well. audience as well. audience as well. I've advised a bunch of startups and I've advised a bunch of startups and I've advised a bunch of startups and been lucky that all of them have reached been lucky that all of them have reached been lucky that all of them have reached pretty positive outcomes. pretty positive outcomes. pretty positive outcomes. So this whole talk is a distilled So this whole talk is a distilled So this whole talk is a distilled version of a case study Anthropic did on version of a case study Anthropic did on version of a case study Anthropic did on Kepler. If you scan that QR code, we're Kepler. If you scan that QR code, we're Kepler. If you scan that QR code, we're the only company they've ever done a the only company they've ever done a the only company they've ever done a case study of, which is kind of cool. case study of, which is kind of cool. case study of, which is kind of cool. And so this is a distilled version of And so this is a distilled version of And so this is a distilled version of that. That has a lot more information that. That has a lot more information that. That has a lot more information about how we were able to do what we do, about how we were able to do what we do, about how we were able to do what we do, why it's been pretty impactful in why it's been pretty impactful in why it's been pretty impactful in financial services, and really where we financial services, and really where we financial services, and really where we go from here.

go from here. go from here. So my only goal with this whole talk is So my only goal with this whole talk is So my only goal with this whole talk is to convince you that AI is going to to convince you that AI is going to to convince you that AI is going to start doing some very powerful things in start doing some very powerful things in start doing some very powerful things in terms of producing work product. So terms of producing work product. So terms of producing work product. So everything that I tell you is everything that I tell you is everything that I tell you is inevitable. Whether it's Kepler or inevitable. Whether it's Kepler or inevitable. Whether it's Kepler or whether it's anyone else, we are already whether it's anyone else, we are already whether it's anyone else, we are already on this journey and this trajectory. on this journey and this trajectory. on this journey and this trajectory. So we all better be ready. So we all better be ready. So we all better be ready. So I first want to observe every talk So I first want to observe every talk So I first want to observe every talk that we've seen in the financial that we've seen in the financial that we've seen in the financial services space so far has been about services space so far has been about services space so far has been about producing more. Like how do I token max? producing more. Like how do I token max? producing more. Like how do I token max? How do I get AI to do more? Very little How do I get AI to do more? Very little How do I get AI to do more? Very little of it is about how to actually make AI of it is about how to actually make AI of it is about how to actually make AI trustworthy or produce trustworthy trustworthy or produce trustworthy trustworthy or produce trustworthy products. products. products. And what's interesting is even terms And what's interesting is even terms And what's interesting is even terms like trust and verify-ability have like trust and verify-ability have like trust and verify-ability have actually been largely abused. Evals are actually been largely abused. Evals are actually been largely abused. Evals are not verifiable. You cannot take a not verifiable. You cannot take a not verifiable. You cannot take a non-deterministic LLM and eval your way non-deterministic LLM and eval your way non-deterministic LLM and eval your way to something deterministic. These are to something deterministic. These are to something deterministic. These are probability machines. probability machines. probability machines. And the kind of underlying reason for And the kind of underlying reason for And the kind of underlying reason for this is that AI has made a writing this is that AI has made a writing this is that AI has made a writing problem a reading problem. problem a reading problem. problem a reading problem. We can produce insane amounts of We can produce insane amounts of We can produce insane amounts of content, whether that's code, whether content, whether that's code, whether content, whether that's code, whether it's marketing, whether it's like a DCF it's marketing, whether it's like a DCF it's marketing, whether it's like a DCF in record time.

in record time. in record time. But we can't easily verify this. And But we can't easily verify this. And But we can't easily verify this. And that's because for years the hardest that's because for years the hardest that's because for years the hardest part about this whole process was part about this whole process was part about this whole process was actually producing the work. actually producing the work. actually producing the work. Edge and alpha came from people like Edge and alpha came from people like Edge and alpha came from people like Citadel being able to hire hundreds of Citadel being able to hire hundreds of Citadel being able to hire hundreds of analysts who could scour the internet analysts who could scour the internet analysts who could scour the internet and understand where any source of alpha and understand where any source of alpha and understand where any source of alpha could exist. So it really came from this could exist. So it really came from this could exist. So it really came from this idea of being able to consume content. idea of being able to consume content. idea of being able to consume content. The problem is when a model reads The problem is when a model reads The problem is when a model reads everything, you have the most real everything, you have the most real everything, you have the most real version of alpha decay that you possibly version of alpha decay that you possibly version of alpha decay that you possibly can. There is no edge if everyone can can. There is no edge if everyone can can. There is no edge if everyone can look at Tegus and get all the same look at Tegus and get all the same look at Tegus and get all the same information. information. information. So the hard part now is trusting what So the hard part now is trusting what So the hard part now is trusting what actually got produced by the model. actually got produced by the model. actually got produced by the model. And this is kind of funny. This is not And this is kind of funny. This is not And this is kind of funny. This is not necessarily a finance problem. necessarily a finance problem. necessarily a finance problem. Every system that exists has some form Every system that exists has some form Every system that exists has some form of this. Software, we run CICD. We do of this. Software, we run CICD. We do of this. Software, we run CICD. We do unit tests. We do integration tests. We unit tests. We do integration tests. We unit tests. We do integration tests. We have code reviews. When a doctor writes have code reviews. When a doctor writes have code reviews. When a doctor writes a prescription, a pharmacist fills that a prescription, a pharmacist fills that a prescription, a pharmacist fills that prescription. So if it says 10,000 prescription. So if it says 10,000 prescription. So if it says 10,000 milligrams of a medication, someone milligrams of a medication, someone milligrams of a medication, someone catches that. We have a pilot and a catches that. We have a pilot and a catches that. We have a pilot and a co-pilot. We have an EMT that's a co-pilot. We have an EMT that's a co-pilot. We have an EMT that's a primary EMT and a secondary.

primary EMT and a secondary. primary EMT and a secondary. In finance, we have maybe an overworked In finance, we have maybe an overworked In finance, we have maybe an overworked VP as a verification layer, but that VP as a verification layer, but that VP as a verification layer, but that concept doesn't really exist. concept doesn't really exist. concept doesn't really exist. And now I'm going to say something even And now I'm going to say something even And now I'm going to say something even more aggressive. more aggressive. more aggressive. Uh Uh Uh the reason that people buy products like the reason that people buy products like the reason that people buy products like Bloomberg and FactSet is to displace Bloomberg and FactSet is to displace Bloomberg and FactSet is to displace culpability. culpability. culpability. When you buy a tool like that, you know When you buy a tool like that, you know When you buy a tool like that, you know that information free. It exists in SEC that information free. It exists in SEC that information free. It exists in SEC filings. But, you believe that because a filings. But, you believe that because a filings. But, you believe that because a bunch of contractors or folks overseas bunch of contractors or folks overseas bunch of contractors or folks overseas vetted this data and stuck it in a vetted this data and stuck it in a vetted this data and stuck it in a central instance, at least if it's central instance, at least if it's central instance, at least if it's wrong, everyone on Wall Street has the wrong, everyone on Wall Street has the wrong, everyone on Wall Street has the same incorrect information. same incorrect information. same incorrect information. And that's interesting in certain ways. And that's interesting in certain ways. And that's interesting in certain ways. It's also kind of a scary proposition. It's also kind of a scary proposition. It's also kind of a scary proposition. And that's because every one of these And that's because every one of these And that's because every one of these tools is read-only. tools is read-only. tools is read-only. You as an analyst look at Bloomberg, you You as an analyst look at Bloomberg, you You as an analyst look at Bloomberg, you look at FactSet, you consume look at FactSet, you consume look at FactSet, you consume information, and you produce the work information, and you produce the work information, and you produce the work product. And that's where the biggest product. And that's where the biggest product. And that's where the biggest gap and biggest wall to real meaningful gap and biggest wall to real meaningful gap and biggest wall to real meaningful adoption across Wall Street has actually adoption across Wall Street has actually adoption across Wall Street has actually been. been. been. And so, why does this actually matter? And so, why does this actually matter? And so, why does this actually matter? We can't use AI properly in this We can't use AI properly in this We can't use AI properly in this ecosystem. There is a reason that ecosystem. There is a reason that ecosystem. There is a reason that analysts are still working till 4:00 analysts are still working till 4:00 analysts are still working till 4:00 a.m., and there's a reason folks in a.m., and there's a reason folks in a.m., and there's a reason folks in investment banks are actually dying investment banks are actually dying investment banks are actually dying because of the hours they're putting in.

because of the hours they're putting in. because of the hours they're putting in. Because AI can produce a very confident Because AI can produce a very confident Because AI can produce a very confident answer, but when it comes to producing answer, but when it comes to producing answer, but when it comes to producing any kind of meaningful work product, any kind of meaningful work product, any kind of meaningful work product, we're totally lacking. we're totally lacking. we're totally lacking. Not only that, we have the SEC, we have Not only that, we have the SEC, we have Not only that, we have the SEC, we have the OCC, we have a number of these the OCC, we have a number of these the OCC, we have a number of these regulatory agencies who exist to make regulatory agencies who exist to make regulatory agencies who exist to make sure that you are not insider trading, sure that you are not insider trading, sure that you are not insider trading, or you're not producing a trading or you're not producing a trading or you're not producing a trading decision that can't be justified or decision that can't be justified or decision that can't be justified or backed by some set of primitives or some backed by some set of primitives or some backed by some set of primitives or some set of information that you can reliably set of information that you can reliably set of information that you can reliably say, "I made this decision because of say, "I made this decision because of say, "I made this decision because of these sources of information." these sources of information." these sources of information." And so, in this ecosystem, the biggest And so, in this ecosystem, the biggest And so, in this ecosystem, the biggest challenge is how do I get AI to jump challenge is how do I get AI to jump challenge is how do I get AI to jump from producing the search technologies from producing the search technologies from producing the search technologies that it's doing right now to producing that it's doing right now to producing that it's doing right now to producing work product. That can mean a fairness work product. That can mean a fairness work product. That can mean a fairness opinion, it can mean a DCF, it can mean opinion, it can mean a DCF, it can mean opinion, it can mean a DCF, it can mean an investment memo, it can mean looking an investment memo, it can mean looking an investment memo, it can mean looking at every SIM that your firm had 5 years at every SIM that your firm had 5 years at every SIM that your firm had 5 years ago and figuring out why your IRR number ago and figuring out why your IRR number ago and figuring out why your IRR number wasn't what it should have been, or wasn't what it should have been, or wasn't what it should have been, or anything else. anything else. anything else. And so, the whole industry has solved And so, the whole industry has solved And so, the whole industry has solved this by citing things.

this by citing things. this by citing things. When you search the internet with Claude When you search the internet with Claude When you search the internet with Claude or ChatGPT, it gives you a list of or ChatGPT, it gives you a list of or ChatGPT, it gives you a list of sources that pulled information from. sources that pulled information from. sources that pulled information from. What's ironic is you can't easily curate What's ironic is you can't easily curate What's ironic is you can't easily curate those sources. So, if you find my random those sources. So, if you find my random those sources. So, if you find my random Substack that says Palantir is going to Substack that says Palantir is going to Substack that says Palantir is going to be $3,000 a share, and you trade off of be $3,000 a share, and you trade off of be $3,000 a share, and you trade off of that, please go do that because it'll be that, please go do that because it'll be that, please go do that because it'll be very helpful for me. But, very helpful for me. But, very helpful for me. But, that's not a real vetted source. Seeking that's not a real vetted source. Seeking that's not a real vetted source. Seeking Alpha and some of these blogs or Reddit Alpha and some of these blogs or Reddit Alpha and some of these blogs or Reddit posts, they're data points, but they're posts, they're data points, but they're posts, they're data points, but they're not real vetted sources. not real vetted sources. not real vetted sources. So, showing where you got the So, showing where you got the So, showing where you got the information from is only half the information from is only half the information from is only half the battle. battle. battle. And that's where we're limited right And that's where we're limited right And that's where we're limited right now. The citation is effectively an now. The citation is effectively an now. The citation is effectively an after-the-fact audit. after-the-fact audit. after-the-fact audit. Now, a verification is a deterministic, Now, a verification is a deterministic, Now, a verification is a deterministic, repeatable, numerically verifiable repeatable, numerically verifiable repeatable, numerically verifiable mechanism that we can use to produce mechanism that we can use to produce mechanism that we can use to produce validity that a number is right. That validity that a number is right. That validity that a number is right. That was a lot of buzzwords. A verification was a lot of buzzwords. A verification was a lot of buzzwords. A verification just means I can prove deterministically just means I can prove deterministically just means I can prove deterministically this number is right. So, when I extract this number is right. So, when I extract this number is right. So, when I extract a revenue number from a 10-K, I can a revenue number from a 10-K, I can a revenue number from a 10-K, I can deterministically prove that is the deterministically prove that is the deterministically prove that is the correct number from that 10-K.

correct number from that 10-K. correct number from that 10-K. So, these are two sides of kind of the So, these are two sides of kind of the So, these are two sides of kind of the same game. same game. same game. One is showing where you got the sources One is showing where you got the sources One is showing where you got the sources from, and one is verifying that the from, and one is verifying that the from, and one is verifying that the information that you pulled out is information that you pulled out is information that you pulled out is actually correct. actually correct. actually correct. This becomes challenging. This becomes challenging. This becomes challenging. It becomes challenging because It becomes challenging because It becomes challenging because verification is not a outcome. verification is not a outcome. verification is not a outcome. It lives in the path or the set of steps It lives in the path or the set of steps It lives in the path or the set of steps that are required for you to produce that are required for you to produce that are required for you to produce information that is valid for your information that is valid for your information that is valid for your individual firm. individual firm. individual firm. Finance is one of the rare industries Finance is one of the rare industries Finance is one of the rare industries where two people can be have the same where two people can be have the same where two people can be have the same information, and one can be long a information, and one can be long a information, and one can be long a stock, and one can be short that stock stock, and one can be short that stock stock, and one can be short that stock with the exact same data. with the exact same data. with the exact same data. And so, the idea of verification is not And so, the idea of verification is not And so, the idea of verification is not actually ground truth. actually ground truth. actually ground truth. It is verifying that you got an output It is verifying that you got an output It is verifying that you got an output that respects the nouns and verbs or the that respects the nouns and verbs or the that respects the nouns and verbs or the rules of your organization. rules of your organization. rules of your organization. If a desk at Citadel, like a TMT desk at If a desk at Citadel, like a TMT desk at If a desk at Citadel, like a TMT desk at Citadel, believes that a particular Citadel, believes that a particular Citadel, believes that a particular stock is going to go up into the right, stock is going to go up into the right, stock is going to go up into the right, another TMT desk at Citadel may have the another TMT desk at Citadel may have the another TMT desk at Citadel may have the exact opposite belief, and they may have exact opposite belief, and they may have exact opposite belief, and they may have the same verification mechanisms that the same verification mechanisms that the same verification mechanisms that produce vastly different outcomes.

produce vastly different outcomes. produce vastly different outcomes. And so, this really comes on to And so, this really comes on to And so, this really comes on to something simple. What are the sources something simple. What are the sources something simple. What are the sources that we trust? What are the that we trust? What are the that we trust? What are the transformations that we apply on those transformations that we apply on those transformations that we apply on those sources to produce information that we sources to produce information that we sources to produce information that we care about? And how do we make sure care about? And how do we make sure care about? And how do we make sure that's codified in a way that makes that's codified in a way that makes that's codified in a way that makes logical sense? logical sense? logical sense? And so, the whole point of this is And so, the whole point of this is And so, the whole point of this is simple. simple. simple. This came from a 10K is not the the This came from a 10K is not the the This came from a 10K is not the the validation as a whole. validation as a whole. validation as a whole. Your job now is to figure out as an Your job now is to figure out as an Your job now is to figure out as an individual, how do I use AI in a individual, how do I use AI in a individual, how do I use AI in a verifiable way? verifiable way? verifiable way? And here's the honest answer. You don't. And here's the honest answer. You don't. And here's the honest answer. You don't. AI is great at doing non-deterministic AI is great at doing non-deterministic AI is great at doing non-deterministic tasks. It can solve problems in a way tasks. It can solve problems in a way tasks. It can solve problems in a way that's novel. It can figure out exactly that's novel. It can figure out exactly that's novel. It can figure out exactly how to do EBITDA adjustments, but it how to do EBITDA adjustments, but it how to do EBITDA adjustments, but it can't be the one responsible for doing can't be the one responsible for doing can't be the one responsible for doing the mathematical adjustments. the mathematical adjustments. the mathematical adjustments. Because it's a probability machine. It Because it's a probability machine. It Because it's a probability machine. It is great at next token prediction. is great at next token prediction. is great at next token prediction. So, the second contention of this whole So, the second contention of this whole So, the second contention of this whole talk is that you cannot use AI to talk is that you cannot use AI to talk is that you cannot use AI to produce verifiable work product in produce verifiable work product in produce verifiable work product in finance without augmenting it with a finance without augmenting it with a finance without augmenting it with a deterministic substrate.

deterministic substrate. deterministic substrate. Which effectively means, if you're a Which effectively means, if you're a Which effectively means, if you're a portfolio manager at Citadel, you have portfolio manager at Citadel, you have portfolio manager at Citadel, you have access to a number of deterministic access to a number of deterministic access to a number of deterministic tools that you use to make your trading tools that you use to make your trading tools that you use to make your trading decisions. decisions. decisions. We need to model AI like that PM. It We need to model AI like that PM. It We need to model AI like that PM. It needs to exist in grounding, needs to needs to exist in grounding, needs to needs to exist in grounding, needs to exist with exist with exist with a certain risk threshold and a certain risk threshold and a certain risk threshold and verifiability. verifiability. verifiability. So, here's how we did it. Here's what So, here's how we did it. Here's what So, here's how we did it. Here's what Anthropic was excited about. We have Anthropic was excited about. We have Anthropic was excited about. We have three tenants that we use to ensure that three tenants that we use to ensure that three tenants that we use to ensure that numerical accuracy is a tenant of the numerical accuracy is a tenant of the numerical accuracy is a tenant of the Kepler platform, and that allows us to Kepler platform, and that allows us to Kepler platform, and that allows us to produce pretty powerful, pretty produce pretty powerful, pretty produce pretty powerful, pretty verifiably reliable information. verifiably reliable information. verifiably reliable information. The first is atomic provenance, and I'll The first is atomic provenance, and I'll The first is atomic provenance, and I'll talk through all of these. The second is talk through all of these. The second is talk through all of these. The second is scope determinism, and the third is scope determinism, and the third is scope determinism, and the third is derivation chains, uh which we'll all derivation chains, uh which we'll all derivation chains, uh which we'll all talk about. talk about. talk about. But the core premise is this. There are But the core premise is this. There are But the core premise is this. There are certain things humans should never do. certain things humans should never do. certain things humans should never do. And I will tell you right now, I don't And I will tell you right now, I don't And I will tell you right now, I don't believe a human should sit there and believe a human should sit there and believe a human should sit there and look at a PDF 10K, 10Q, 8K, or earnings look at a PDF 10K, 10Q, 8K, or earnings look at a PDF 10K, 10Q, 8K, or earnings call transcript and on one screen take call transcript and on one screen take call transcript and on one screen take that number and put it into an Excel that number and put it into an Excel that number and put it into an Excel model on the other screen. Humans were model on the other screen. Humans were model on the other screen. Humans were not built for that. That's not like not built for that. That's not like not built for that. That's not like there's a reason we don't have databases there's a reason we don't have databases there's a reason we don't have databases just written in our heads.

just written in our heads. just written in our heads. And so, let's break this down for how And so, let's break this down for how And so, let's break this down for how you can actually use AI to produce work you can actually use AI to produce work you can actually use AI to produce work product. product. product. Let's talk about provenance. Let's talk about provenance. Let's talk about provenance. Provenance as a whole just means writing Provenance as a whole just means writing Provenance as a whole just means writing down exactly where you got the down exactly where you got the down exactly where you got the information. information. information. Now, Now, Now, uh uh uh have a lot of respect for everyone else have a lot of respect for everyone else have a lot of respect for everyone else who got here on stage, but we did a who got here on stage, but we did a who got here on stage, but we did a model we trained a model that was really model we trained a model that was really model we trained a model that was really good at extracting information. It good at extracting information. It good at extracting information. It outperformed foundation models, it's 94% outperformed foundation models, it's 94% outperformed foundation models, it's 94% great. It's in the article. great. It's in the article. great. It's in the article. Who here would trade off of something Who here would trade off of something Who here would trade off of something that's 94% accurate? that's 94% accurate? that's 94% accurate? Right. So, fine-tuning your way on Right. So, fine-tuning your way on Right. So, fine-tuning your way on probabilistic solutions still is it's probabilistic solutions still is it's probabilistic solutions still is it's really cool and like TechCrunch will be really cool and like TechCrunch will be really cool and like TechCrunch will be really excited about it, but like no one really excited about it, but like no one really excited about it, but like no one else really cares about it. else really cares about it. else really cares about it. And that's because a wrong number is And that's because a wrong number is And that's because a wrong number is still wrong if you're in that still wrong if you're in that still wrong if you're in that unfortunate 6%. unfortunate 6%. unfortunate 6%. So, with atomic provenance, what we do So, with atomic provenance, what we do So, with atomic provenance, what we do is the model writes effectively a is the model writes effectively a is the model writes effectively a reference to the number. reference to the number. reference to the number. It cannot write the number or manipulate It cannot write the number or manipulate It cannot write the number or manipulate the number in any way. It doesn't even the number in any way. It doesn't even the number in any way. It doesn't even understand what that number is. understand what that number is. understand what that number is. We have tools that are really good at We have tools that are really good at We have tools that are really good at understanding numbers. They're understanding numbers. They're understanding numbers. They're databases. They are systems that can databases. They are systems that can databases. They are systems that can codify information and read them and codify information and read them and codify information and read them and write them with appropriate fidelity.

write them with appropriate fidelity. write them with appropriate fidelity. So, that's the first piece of how we do So, that's the first piece of how we do So, that's the first piece of how we do things. Anytime a model makes a things. Anytime a model makes a things. Anytime a model makes a decision, it makes the decision to decision, it makes the decision to decision, it makes the decision to figure out exactly where it got the figure out exactly where it got the figure out exactly where it got the number from and hand off to something number from and hand off to something number from and hand off to something that can write that number. that can write that number. that can write that number. We then run it through a deterministic We then run it through a deterministic We then run it through a deterministic check where any kind of a wrong number, check where any kind of a wrong number, check where any kind of a wrong number, if we can't verify it independently, we if we can't verify it independently, we if we can't verify it independently, we strip it out. strip it out. strip it out. That number will never make it to That number will never make it to That number will never make it to someone if it doesn't follow the someone if it doesn't follow the someone if it doesn't follow the deterministic check, the Providence deterministic check, the Providence deterministic check, the Providence ledger, and most importantly, ledger, and most importantly, ledger, and most importantly, uh the whole cycle kind of repeating at uh the whole cycle kind of repeating at uh the whole cycle kind of repeating at least a couple of times. least a couple of times. least a couple of times. So, this is not me saying, "Have 10 So, this is not me saying, "Have 10 So, this is not me saying, "Have 10 models and each individual have OpenAI models and each individual have OpenAI models and each individual have OpenAI check chat check Anthropic and have check chat check Anthropic and have check chat check Anthropic and have Anthropic check XAI." These are not Anthropic check XAI." These are not Anthropic check XAI." These are not probabilistic systems evaluating each probabilistic systems evaluating each probabilistic systems evaluating each other's work. There's a core canonical other's work. There's a core canonical other's work. There's a core canonical process of extracting a number, process of extracting a number, process of extracting a number, persisting it, and making sure that persisting it, and making sure that persisting it, and making sure that process actually occurred properly. process actually occurred properly. process actually occurred properly. And this is when I say atomic And this is when I say atomic And this is when I say atomic traditional atomicity in like database traditional atomicity in like database traditional atomicity in like database land. land. land. Second, scope determinism. Second, scope determinism. Second, scope determinism. This is This was a super controversial This is This was a super controversial This is This was a super controversial idea like a year ago, and VCs were like, idea like a year ago, and VCs were like, idea like a year ago, and VCs were like, "This is crazy." Now, this half of this "This is crazy." Now, this half of this "This is crazy." Now, this half of this talk has been This session has been talk has been This session has been talk has been This session has been about this. Um about this. Um about this. Um the model is really good at reasoning the model is really good at reasoning the model is really good at reasoning and planning. Intelligence is and planning. Intelligence is and planning. Intelligence is commoditized. GLM 52 shows it. You can commoditized. GLM 52 shows it. You can commoditized. GLM 52 shows it. You can download it off of Hugging Face right download it off of Hugging Face right download it off of Hugging Face right now. You have something as powerful as now. You have something as powerful as now. You have something as powerful as Opus 48.

Opus 48. Opus 48. Now, what the model cannot do is math. Now, what the model cannot do is math. Now, what the model cannot do is math. And why would it? Why would I run 1 + 1 And why would it? Why would I run 1 + 1 And why would it? Why would I run 1 + 1 through a multi-billion parameter model through a multi-billion parameter model through a multi-billion parameter model instead of one CPU cycle? instead of one CPU cycle? instead of one CPU cycle? Unless you're companies that are giving Unless you're companies that are giving Unless you're companies that are giving bonuses on people token maxing, which is bonuses on people token maxing, which is bonuses on people token maxing, which is another Polarys thing. another Polarys thing. another Polarys thing. >> [snorts] >> [snorts] >> [snorts] >> Um and so, what the model does is the >> Um and so, what the model does is the >> Um and so, what the model does is the model decides what to compute. It never model decides what to compute. It never model decides what to compute. It never does the computation itself. does the computation itself. does the computation itself. And so, from the Kepler platform And so, from the Kepler platform And so, from the Kepler platform perspective, what we do is we split the perspective, what we do is we split the perspective, what we do is we split the deterministic pieces of the model, which deterministic pieces of the model, which deterministic pieces of the model, which are none, from the non-deterministic are none, from the non-deterministic are none, from the non-deterministic pieces of the model, which are all of pieces of the model, which are all of pieces of the model, which are all of them. And we give the model the right them. And we give the model the right them. And we give the model the right tooling and technology to calculate the tooling and technology to calculate the tooling and technology to calculate the deterministic pieces. Now, let's get deterministic pieces. Now, let's get deterministic pieces. Now, let's get really concrete. A model can read really concrete. A model can read really concrete. A model can read something like, something like, something like, "Okay, I need to understand what net "Okay, I need to understand what net "Okay, I need to understand what net margin is. I know the right pieces of margin is. I know the right pieces of margin is. I know the right pieces of information to go to to get that data, information to go to to get that data, information to go to to get that data, but I can't be the entity that's but I can't be the entity that's but I can't be the entity that's actually running the code behind the actually running the code behind the actually running the code behind the scenes to pull that number out of a PDF scenes to pull that number out of a PDF scenes to pull that number out of a PDF or parsing the XBRL behind the scenes to or parsing the XBRL behind the scenes to or parsing the XBRL behind the scenes to pull that information out. That is pull that information out. That is pull that information out. That is code." code." code." Now, the deterministic pieces of the Now, the deterministic pieces of the Now, the deterministic pieces of the platform pull that information out and platform pull that information out and platform pull that information out and persist it outside of anything the model persist it outside of anything the model persist it outside of anything the model understands.

understands. understands. And with those two together, we can And with those two together, we can And with those two together, we can actually produce a numerically accurate actually produce a numerically accurate actually produce a numerically accurate answer to the question of like, what was answer to the question of like, what was answer to the question of like, what was the net What was the net margin of this the net What was the net margin of this the net What was the net margin of this stock or this company last quarter? stock or this company last quarter? stock or this company last quarter? This also is a lot cheaper because This also is a lot cheaper because This also is a lot cheaper because again, I don't need the model to do a again, I don't need the model to do a again, I don't need the model to do a bunch of stuff it shouldn't be doing. Now, the last piece here is how do we Now, the last piece here is how do we actually do reconciliation? actually do reconciliation? actually do reconciliation? When someone asks about a ratio like a When someone asks about a ratio like a When someone asks about a ratio like a gross margin or anything else that gross margin or anything else that gross margin or anything else that doesn't exist in a filing, therefore, I doesn't exist in a filing, therefore, I doesn't exist in a filing, therefore, I can't just go look up what the gross can't just go look up what the gross can't just go look up what the gross margin is or what the set of EBITDA margin is or what the set of EBITDA margin is or what the set of EBITDA adjustments were. adjustments were. adjustments were. The other thing that's really The other thing that's really The other thing that's really complicated is everyone calculates these complicated is everyone calculates these complicated is everyone calculates these ratios and these multiples differently. ratios and these multiples differently. ratios and these multiples differently. Everyone does enterprise value Everyone does enterprise value Everyone does enterprise value calculations differently. Things that calculations differently. Things that calculations differently. Things that are considered recurring or are considered recurring or are considered recurring or non-recurring may be unique. non-recurring may be unique. non-recurring may be unique. So, not only do we have to codify that So, not only do we have to codify that So, not only do we have to codify that in the processes that are run, but we in the processes that are run, but we in the processes that are run, but we need some kind of a chain of events to need some kind of a chain of events to need some kind of a chain of events to figure out what went into producing an figure out what went into producing an figure out what went into producing an individual number and what went into individual number and what went into individual number and what went into producing an outcome. producing an outcome. producing an outcome. This is not any different than the chain This is not any different than the chain This is not any different than the chain of events that an analyst does on a desk of events that an analyst does on a desk of events that an analyst does on a desk at a hedge fund to make a risk-reward or at a hedge fund to make a risk-reward or at a hedge fund to make a risk-reward or a trading decision. It's just done by a trading decision. It's just done by a trading decision. It's just done by the model in a way that can be replayed the model in a way that can be replayed the model in a way that can be replayed and rewound.

and rewound. and rewound. And so, that leads us to something And so, that leads us to something And so, that leads us to something fairly simple here. fairly simple here. fairly simple here. Uh Uh Uh we have a system that knows which data we have a system that knows which data we have a system that knows which data points it's allowed to produce, meaning points it's allowed to produce, meaning points it's allowed to produce, meaning from structured filings, from numerical from structured filings, from numerical from structured filings, from numerical data, from any other ecosystem, it knows data, from any other ecosystem, it knows data, from any other ecosystem, it knows what it's allowed to produce, what it's allowed to produce, what it's allowed to produce, and it knows what it's never allowed to and it knows what it's never allowed to and it knows what it's never allowed to produce. So, if it's pulling things out produce. So, if it's pulling things out produce. So, if it's pulling things out of prose, raw tables, or anything else, of prose, raw tables, or anything else, of prose, raw tables, or anything else, it doesn't do that extraction. it doesn't do that extraction. it doesn't do that extraction. What this allows us to do What this allows us to do What this allows us to do is this allows us to do things like is this allows us to do things like is this allows us to do things like consolidate financial statements in consolidate financial statements in consolidate financial statements in seconds with every number tied back to seconds with every number tied back to seconds with every number tied back to its individual source. its individual source. its individual source. Which means Which means Which means not picking any companies, but the not picking any companies, but the not picking any companies, but the CapIQ's, the Deloopa's that are all CapIQ's, the Deloopa's that are all CapIQ's, the Deloopa's that are all using contractors for this, using contractors for this, using contractors for this, we don't need to do that anymore. we don't need to do that anymore. we don't need to do that anymore. We can actually create a financial model We can actually create a financial model We can actually create a financial model in a numerically accurate way that in a numerically accurate way that in a numerically accurate way that allows you to build work product. We can allows you to build work product. We can allows you to build work product. We can build a DCF in the format that you build a DCF in the format that you build a DCF in the format that you actually want. In knowledge, the model actually want. In knowledge, the model actually want. In knowledge, the model will not hallucinate that a row exists will not hallucinate that a row exists will not hallucinate that a row exists that shouldn't exist. that shouldn't exist. that shouldn't exist. Now, the really crazy thing and why I Now, the really crazy thing and why I Now, the really crazy thing and why I say this is inevitable is everything say this is inevitable is everything say this is inevitable is everything that I'm telling you generally that I'm telling you generally that I'm telling you generally generalizes past finance.

generalizes past finance. generalizes past finance. We're picking numbers here because We're picking numbers here because We're picking numbers here because finance cares about numbers. finance cares about numbers. finance cares about numbers. But you can imagine a world where every But you can imagine a world where every But you can imagine a world where every court case, if you're a Harvey or a court case, if you're a Harvey or a court case, if you're a Harvey or a Legora, runs through the same process. A Legora, runs through the same process. A Legora, runs through the same process. A pre-processing step that understands pre-processing step that understands pre-processing step that understands that we can extract entities like case A that we can extract entities like case A that we can extract entities like case A versus case B and store that versus case B and store that versus case B and store that deterministically so we don't deterministically so we don't deterministically so we don't hallucinate citations. hallucinate citations. hallucinate citations. Or every drug discovery formulation that Or every drug discovery formulation that Or every drug discovery formulation that exists in NIH white papers such that we exists in NIH white papers such that we exists in NIH white papers such that we never miss a compound or anything else. never miss a compound or anything else. never miss a compound or anything else. And so, the kind of uh interesting And so, the kind of uh interesting And so, the kind of uh interesting dimension that we're entering is we're dimension that we're entering is we're dimension that we're entering is we're an ecosystem right now where we're an ecosystem right now where we're an ecosystem right now where we're almost like pre-SSL almost like pre-SSL almost like pre-SSL in the e-commerce ecosystem. in the e-commerce ecosystem. in the e-commerce ecosystem. Where like what's the TAM of e-commerce? Where like what's the TAM of e-commerce? Where like what's the TAM of e-commerce? Trillions, but how many people were Trillions, but how many people were Trillions, but how many people were comfortable putting their credit card comfortable putting their credit card comfortable putting their credit card number on the internet before there was number on the internet before there was number on the internet before there was security? security? security? Zero. Zero. Zero. So, we're in the last step. AI can now So, we're in the last step. AI can now So, we're in the last step. AI can now produce verifiable work product across a produce verifiable work product across a produce verifiable work product across a number of industries. Meaning the rag number of industries. Meaning the rag number of industries. Meaning the rag platforms of the past are really, really platforms of the past are really, really platforms of the past are really, really helpful and really cool in codifying helpful and really cool in codifying helpful and really cool in codifying workflows, but there's a reason that the workflows, but there's a reason that the workflows, but there's a reason that the foundational labs are going after every foundational labs are going after every foundational labs are going after every one of these. There's a reason that one of these. There's a reason that one of these. There's a reason that Claude for science is not a Claude for science is not a Claude for science is not a deterministic system, but still a deterministic system, but still a deterministic system, but still a rag-based system.

rag-based system. rag-based system. And so, the And so, the And so, the piece that should be really exciting piece that should be really exciting piece that should be really exciting about this whole thing is there is a about this whole thing is there is a about this whole thing is there is a piece of this that no one has built yet piece of this that no one has built yet piece of this that no one has built yet in a variety of disciplines and a in a variety of disciplines and a in a variety of disciplines and a variety of industries. variety of industries. variety of industries. We're really good at consuming tokens. We're really good at consuming tokens. We're really good at consuming tokens. In fact, there's a club here for people In fact, there's a club here for people In fact, there's a club here for people that consumed a billion plus tokens. that consumed a billion plus tokens. that consumed a billion plus tokens. They're walking around with gold cards. They're walking around with gold cards. They're walking around with gold cards. It's kind of funny, actually. It's kind of funny, actually. It's kind of funny, actually. But, like, that's kind of hilarious, But, like, that's kind of hilarious, But, like, that's kind of hilarious, right? Like, in what time in history has right? Like, in what time in history has right? Like, in what time in history has an employee been rewarded for your an employee been rewarded for your an employee been rewarded for your company to pay another vendor for how company to pay another vendor for how company to pay another vendor for how much money you're spending? much money you're spending? much money you're spending? And so, token maxing, I think, is now And so, token maxing, I think, is now And so, token maxing, I think, is now being thought of as not the right being thought of as not the right being thought of as not the right approach here to actually solve your approach here to actually solve your approach here to actually solve your problems. problems. problems. So, the natural thing will become a rush So, the natural thing will become a rush So, the natural thing will become a rush to the bottom, which is an optimization to the bottom, which is an optimization to the bottom, which is an optimization problem, which we've seen over and over problem, which we've seen over and over problem, which we've seen over and over again. again. again. If you rewind time, I actually sat on If you rewind time, I actually sat on If you rewind time, I actually sat on this stage 4 years ago, maybe not this this stage 4 years ago, maybe not this this stage 4 years ago, maybe not this particular room, talking about like particular room, talking about like particular room, talking about like Snowflake Summit and Databricks Summit, Snowflake Summit and Databricks Summit, Snowflake Summit and Databricks Summit, where the idea was all of a sudden where the idea was all of a sudden where the idea was all of a sudden people wanted ROI on top of like to people wanted ROI on top of like to people wanted ROI on top of like to understand the ROI of their Snowflake understand the ROI of their Snowflake understand the ROI of their Snowflake investment or their Databricks investment or their Databricks investment or their Databricks investment. And we started doing cost investment. And we started doing cost investment. And we started doing cost optimization. We started figuring out optimization. We started figuring out optimization. We started figuring out how to make sure every dollar of capital how to make sure every dollar of capital how to make sure every dollar of capital we put into Snowflake and into we put into Snowflake and into we put into Snowflake and into Databricks went to actually producing a Databricks went to actually producing a Databricks went to actually producing a pipeline that people were using.

pipeline that people were using. pipeline that people were using. That same trend is about to start. And That same trend is about to start. And That same trend is about to start. And we're figuring out right now, how do we we're figuring out right now, how do we we're figuring out right now, how do we use the right tool for the right job? use the right tool for the right job? use the right tool for the right job? And sometimes you don't need a And sometimes you don't need a And sometimes you don't need a multi-billion parameter model when one multi-billion parameter model when one multi-billion parameter model when one CPU cycle will just work. CPU cycle will just work. CPU cycle will just work. So, kind of wrapping this up, So, kind of wrapping this up, So, kind of wrapping this up, AI has made producing work completely, I AI has made producing work completely, I AI has made producing work completely, I say nearly free, but like, thank you say nearly free, but like, thank you say nearly free, but like, thank you VCs, heavily subsidized. VCs, heavily subsidized. VCs, heavily subsidized. The reading problem is still very open. The reading problem is still very open. The reading problem is still very open. And you verifying a data point is not And you verifying a data point is not And you verifying a data point is not enough anymore. enough anymore. enough anymore. The system has to be able to track or The system has to be able to track or The system has to be able to track or track its own provenance and actually track its own provenance and actually track its own provenance and actually ensure that the numbers that you're ensure that the numbers that you're ensure that the numbers that you're pulling represent your own unique pulling represent your own unique pulling represent your own unique company philosophies. company philosophies. company philosophies. Citations got us like 50% of the way Citations got us like 50% of the way Citations got us like 50% of the way there, but the next half of the there, but the next half of the there, but the next half of the verifiability and provability is going verifiability and provability is going verifiability and provability is going to be how we start using this in real to be how we start using this in real to be how we start using this in real like valuable, verifiable work. like valuable, verifiable work. like valuable, verifiable work. And the interesting thing here is the And the interesting thing here is the And the interesting thing here is the work product itself is the proof. work product itself is the proof. work product itself is the proof. In code, we have I mean unit tests, we In code, we have I mean unit tests, we In code, we have I mean unit tests, we have every single pull request and every have every single pull request and every have every single pull request and every commit and every code review on that commit and every code review on that commit and every code review on that pull request stored in perpetuity. There pull request stored in perpetuity. There pull request stored in perpetuity. There are companies here trying to mine that are companies here trying to mine that are companies here trying to mine that information to create a representation information to create a representation information to create a representation of your on your ontology right now on a of your on your ontology right now on a of your on your ontology right now on a company specific basis.

company specific basis. company specific basis. We need that same ecosystem in finance. We need that same ecosystem in finance. We need that same ecosystem in finance. So, let's just say this, all these So, let's just say this, all these So, let's just say this, all these problems are solved at this point. The problems are solved at this point. The problems are solved at this point. The last remaining mile is going to be that last remaining mile is going to be that last remaining mile is going to be that personalization. personalization. personalization. So, the second version of this in 2027 So, the second version of this in 2027 So, the second version of this in 2027 hopefully one of us will be on stage hopefully one of us will be on stage hopefully one of us will be on stage talking about how we're now able to talking about how we're now able to talking about how we're now able to build verifiable ontologies that build verifiable ontologies that build verifiable ontologies that actually proxy our investment processes actually proxy our investment processes actually proxy our investment processes instead of saying how do I not spend a instead of saying how do I not spend a instead of saying how do I not spend a trillion tokens to solve this individual trillion tokens to solve this individual trillion tokens to solve this individual problem. problem. problem. Um so, Um so, Um so, we're also we're also we're also I mean Susanna, we're both from Kepler. I mean Susanna, we're both from Kepler. I mean Susanna, we're both from Kepler. We're growing pretty quickly. Obligatory We're growing pretty quickly. Obligatory We're growing pretty quickly. Obligatory come join us if these problems are come join us if these problems are come join us if these problems are interesting to you. interesting to you. interesting to you. And yeah, I think we're starting in And yeah, I think we're starting in And yeah, I think we're starting in finance now. We'll be in a lot of finance now. We'll be in a lot of finance now. We'll be in a lot of different dimensions pretty quickly. Um different dimensions pretty quickly. Um different dimensions pretty quickly. Um so, thank you so much and happy to so, thank you so much and happy to so, thank you so much and happy to answer any questions. answer any questions. answer any questions. >> [applause] Yes, your question is about provenance Yes, your question is about provenance and from the provenance ledger, where and from the provenance ledger, where and from the provenance ledger, where does the number actually come from? At does the number actually come from? At does the number actually come from? At its core, the number comes from three its core, the number comes from three its core, the number comes from three different things. It comes from different things. It comes from different things. It comes from extracted information from the filings.

extracted information from the filings. extracted information from the filings. It comes from either a mathematical It comes from either a mathematical It comes from either a mathematical calculation that we do to derivative calculation that we do to derivative calculation that we do to derivative production, so like a ratio or something production, so like a ratio or something production, so like a ratio or something else. else. else. Or it comes from your internal documents Or it comes from your internal documents Or it comes from your internal documents or other internal pieces of information or other internal pieces of information or other internal pieces of information that operate in conjunction with that that operate in conjunction with that that operate in conjunction with that external data to produce that individual external data to produce that individual external data to produce that individual data point. So, anytime the model data point. So, anytime the model data point. So, anytime the model effectively effectively effectively is responsible for telling some entity is responsible for telling some entity is responsible for telling some entity to do an IO operation, that's part of to do an IO operation, that's part of to do an IO operation, that's part of the provenance chain. the provenance chain. the provenance chain. Yeah. What are your customers most excited What are your customers most excited about in your product? about in your product? about in your product? >> Yeah, so the question is um what are our >> Yeah, so the question is um what are our >> Yeah, so the question is um what are our customers most excited about? It's customers most excited about? It's customers most excited about? It's funny. Uh everyone wants AI like funny. Uh everyone wants AI like funny. Uh everyone wants AI like the dream is the AI portfolio manager. the dream is the AI portfolio manager. the dream is the AI portfolio manager. The portfolio managers don't want the AI The portfolio managers don't want the AI The portfolio managers don't want the AI portfolio manager. Like they want the AI portfolio manager. Like they want the AI portfolio manager. Like they want the AI analyst. And so the thing they're most analyst. And so the thing they're most analyst. And so the thing they're most excited about is a way of rapidly excited about is a way of rapidly excited about is a way of rapidly producing, rapidly doing kind of the producing, rapidly doing kind of the producing, rapidly doing kind of the repeatable painful tasks that their repeatable painful tasks that their repeatable painful tasks that their analysts are doing. analysts are doing. analysts are doing. Things like uh analyst actually sits Things like uh analyst actually sits Things like uh analyst actually sits there and listens to an earnings call there and listens to an earnings call there and listens to an earnings call transcript. If we didn't have to have an transcript. If we didn't have to have an transcript. If we didn't have to have an analyst do that, it would be amazing. Or analyst do that, it would be amazing. Or analyst do that, it would be amazing. Or an analyst sits there with like 15 tabs an analyst sits there with like 15 tabs an analyst sits there with like 15 tabs and they're opening every 8-K and 10-K and they're opening every 8-K and 10-K and they're opening every 8-K and 10-K over the last, you know, whatever years over the last, you know, whatever years over the last, you know, whatever years to create the V0 of a financial model.

to create the V0 of a financial model. to create the V0 of a financial model. So they're most excited about getting So they're most excited about getting So they're most excited about getting their analyst time back. That's the their analyst time back. That's the their analyst time back. That's the honest answer. honest answer. honest answer. And these are expensive analysts. Like And these are expensive analysts. Like And these are expensive analysts. Like these these these Some of these folks make 6-700k a year Some of these folks make 6-700k a year Some of these folks make 6-700k a year to do this. Cool. I'm happy to answer any more Cool. I'm happy to answer any more questions outside as actually Yeah, questions outside as actually Yeah, questions outside as actually Yeah, outside as well if you want to defer it outside as well if you want to defer it outside as well if you want to defer it time. time. time. Yeah, the gentleman in the back.