Trading Desks to Clinical Trials: Parallels in Applied Vertical AI — Ayush Bhardwaj, Allos AI

So, my name is Ayush Bhardwaj and I did So, my name is Ayush Bhardwaj and I did applied AI for a hedge fund. And now I applied AI for a hedge fund. And now I applied AI for a hedge fund. And now I do everything tech plus applied AI for a do everything tech plus applied AI for a do everything tech plus applied AI for a pharma tech startup cuz you know the way pharma tech startup cuz you know the way pharma tech startup cuz you know the way startups are. You have to do everything, startups are. You have to do everything, startups are. You have to do everything, wear multiple hats. wear multiple hats. wear multiple hats. So, So, So, before I start the session, I would like before I start the session, I would like before I start the session, I would like to do a small survey. Can I get a raise to do a small survey. Can I get a raise to do a small survey. Can I get a raise of hands for all the engineers in the of hands for all the engineers in the of hands for all the engineers in the room? room? room? Okay, that's a tough room. Now, can I Okay, that's a tough room. Now, can I Okay, that's a tough room. Now, can I get a raise of hands for managers? get a raise of hands for managers? get a raise of hands for managers? Okay, just to be clear, managing AI Okay, just to be clear, managing AI Okay, just to be clear, managing AI agent does not count. agent does not count. agent does not count. You have to manage people. Okay, we have You have to manage people. Okay, we have You have to manage people. Okay, we have few managers as well. few managers as well. few managers as well. Interesting. So, they will help me like Interesting. So, they will help me like Interesting. So, they will help me like fine-tune my talk a bit. fine-tune my talk a bit. fine-tune my talk a bit. So, today my aim is to take you through So, today my aim is to take you through So, today my aim is to take you through the journey of how do you actually build the journey of how do you actually build the journey of how do you actually build and iterate in applied vertical AI? and iterate in applied vertical AI? and iterate in applied vertical AI? And my experience is from the hedge fund And my experience is from the hedge fund And my experience is from the hedge fund and the pharma tech company. So, before and the pharma tech company. So, before and the pharma tech company. So, before delving deep into the recipe, I'll just delving deep into the recipe, I'll just delving deep into the recipe, I'll just like take you through what do I even like take you through what do I even like take you through what do I even mean by applied vertical AI cuz I don't mean by applied vertical AI cuz I don't mean by applied vertical AI cuz I don't know if it sounds like a very weird know if it sounds like a very weird know if it sounds like a very weird term. It's like the vertical word is term. It's like the vertical word is term. It's like the vertical word is kind of forced. I won't lie, it is. I kind of forced. I won't lie, it is. I kind of forced. I won't lie, it is. I coined this term probably. So, applied coined this term probably. So, applied coined this term probably. So, applied AI is like built for So, what applied AI is like built for So, what applied AI is like built for So, what applied vertical AI is essentially applied AI vertical AI is essentially applied AI vertical AI is essentially applied AI but built for one very specific but built for one very specific but built for one very specific industry.

industry. industry. It's It's aim is to simulate a job of a It's It's aim is to simulate a job of a It's It's aim is to simulate a job of a person in that particular industry in a person in that particular industry in a person in that particular industry in a sense. So, an example of applied AI is sense. So, an example of applied AI is sense. So, an example of applied AI is Google Translate which is like general Google Translate which is like general Google Translate which is like general purpose, helps you translate. It could purpose, helps you translate. It could purpose, helps you translate. It could be used in education tech and it can be used in education tech and it can be used in education tech and it can have like tons and various sorts of uses have like tons and various sorts of uses have like tons and various sorts of uses is whereas Elos, which is my employer, is whereas Elos, which is my employer, is whereas Elos, which is my employer, the pharma tech company, we specifically the pharma tech company, we specifically the pharma tech company, we specifically build drugs with AI. So, that's a very build drugs with AI. So, that's a very build drugs with AI. So, that's a very specific use case. specific use case. specific use case. Another examples of applied vertical AI Another examples of applied vertical AI Another examples of applied vertical AI field could be the legal tech firms that field could be the legal tech firms that field could be the legal tech firms that are now coming up with. You must I'm are now coming up with. You must I'm are now coming up with. You must I'm sure you must have heard about them. So, sure you must have heard about them. So, sure you must have heard about them. So, those are like another the examples of those are like another the examples of those are like another the examples of applied vertical AI. applied vertical AI. applied vertical AI. So, So, So, when I left the hedge fund, right? So, I when I left the hedge fund, right? So, I when I left the hedge fund, right? So, I was expecting that the world would was expecting that the world would was expecting that the world would change for me cuz you know, hedge funds change for me cuz you know, hedge funds change for me cuz you know, hedge funds are like really fast and really pressure are like really fast and really pressure are like really fast and really pressure sensitive. Whereas, pharma is like, sensitive. Whereas, pharma is like, sensitive. Whereas, pharma is like, "Okay, we're going to take 15 years, but "Okay, we're going to take 15 years, but "Okay, we're going to take 15 years, but we're going to do it right." Hedge fund we're going to do it right." Hedge fund we're going to do it right." Hedge fund was all about like, "You need to do it was all about like, "You need to do it was all about like, "You need to do it fast and mostly right. It does not fast and mostly right. It does not fast and mostly right. It does not matter if we lose at one paradigm as matter if we lose at one paradigm as matter if we lose at one paradigm as long as we are overall winning." long as we are overall winning." long as we are overall winning." Whereas, a pharma firm is like, "We have Whereas, a pharma firm is like, "We have Whereas, a pharma firm is like, "We have to be absolutely right. You can take a to be absolutely right. You can take a to be absolutely right. You can take a week more." And it was true. It's it's a week more." And it was true. It's it's a week more." And it was true. It's it's a completely different world. But, completely different world. But, completely different world. But, to your surprise and to mine as well, to your surprise and to mine as well, to your surprise and to mine as well, nothing changed, actually.

nothing changed, actually. nothing changed, actually. My job increased, but the core part of My job increased, but the core part of My job increased, but the core part of my job, applied AI, remained the exact my job, applied AI, remained the exact my job, applied AI, remained the exact same and I cannot uh same and I cannot uh same and I cannot uh express how surprised I was cuz I express how surprised I was cuz I express how surprised I was cuz I thought that it'll be a complete thought that it'll be a complete thought that it'll be a complete different thing, but apparently it was different thing, but apparently it was different thing, but apparently it was not. So, uh not. So, uh not. So, uh So, then I spoke to other people as well So, then I spoke to other people as well So, then I spoke to other people as well across legal AI and the people coming up across legal AI and the people coming up across legal AI and the people coming up with the prop tech firms, which is with the prop tech firms, which is with the prop tech firms, which is essentially the real estate tech firms. essentially the real estate tech firms. essentially the real estate tech firms. And I realized that everyone is kind of And I realized that everyone is kind of And I realized that everyone is kind of building the applied vertical AI in a building the applied vertical AI in a building the applied vertical AI in a very similar way. I could see some steps very similar way. I could see some steps very similar way. I could see some steps that could be essentially abstracted that could be essentially abstracted that could be essentially abstracted out. And that's what we'll do today. out. And that's what we'll do today. out. And that's what we'll do today. So, uh before again delving the deep So, uh before again delving the deep So, uh before again delving the deep into that, I received a few reach outs into that, I received a few reach outs into that, I received a few reach outs saying, saying, saying, "Are people actually putting agents into "Are people actually putting agents into "Are people actually putting agents into production?" production?" production?" And I was like, this is such a wrong And I was like, this is such a wrong And I was like, this is such a wrong question to ask. Everyone is putting question to ask. Everyone is putting question to ask. Everyone is putting agents into production, even like agents into production, even like agents into production, even like 15-year-old 15-year-old 15-year-old 15-year-old kids these days. The 15-year-old kids these days. The 15-year-old kids these days. The question to ask is whether they actually question to ask is whether they actually question to ask is whether they actually work, whether they actually make or save work, whether they actually make or save work, whether they actually make or save money, whether they justify their ROI, money, whether they justify their ROI, money, whether they justify their ROI, whether uh they're making way more than whether uh they're making way more than whether uh they're making way more than the amount we are investing into it like the amount we are investing into it like the amount we are investing into it like end-to-end. And I can say from my end-to-end. And I can say from my end-to-end. And I can say from my anecdotal experience, yes. At the both anecdotal experience, yes. At the both anecdotal experience, yes. At the both places I worked, the agent either saved places I worked, the agent either saved places I worked, the agent either saved the money or made more money.

the money or made more money. the money or made more money. So, with that, let's get started. So, with that, let's get started. So, with that, let's get started. So, So, So, the recipe I'll take you through a the recipe I'll take you through a the recipe I'll take you through a series of seven steps, roughly, and try series of seven steps, roughly, and try series of seven steps, roughly, and try to like make this process as simple as to like make this process as simple as to like make this process as simple as possible. So, the first step is possible. So, the first step is possible. So, the first step is formulate the problem. So, this is formulate the problem. So, this is formulate the problem. So, this is sounds like very trivial, but a lot of sounds like very trivial, but a lot of sounds like very trivial, but a lot of people, specifically startups, get this people, specifically startups, get this people, specifically startups, get this wrong. They just try to do too much at wrong. They just try to do too much at wrong. They just try to do too much at once. Whereas, from what I have learned once. Whereas, from what I have learned once. Whereas, from what I have learned and what I think a lot of colleagues and what I think a lot of colleagues and what I think a lot of colleagues would agree, you need to pick a very would agree, you need to pick a very would agree, you need to pick a very narrow task. You just cannot ask it to narrow task. You just cannot ask it to narrow task. You just cannot ask it to do everything. A good example for this do everything. A good example for this do everything. A good example for this could be, let's say if you build could be, let's say if you build could be, let's say if you build something in finance, you won't ask it something in finance, you won't ask it something in finance, you won't ask it to like, "Hey, can you fetch me top to like, "Hey, can you fetch me top to like, "Hey, can you fetch me top three market opportunities that I could three market opportunities that I could three market opportunities that I could invest in?" No, that won't work. You invest in?" No, that won't work. You invest in?" No, that won't work. You have to be like very specific. have to be like very specific. have to be like very specific. Like you pick a market, you say, "Let's Like you pick a market, you say, "Let's Like you pick a market, you say, "Let's take the US equities." Then you pick an take the US equities." Then you pick an take the US equities." Then you pick an industry, let's take IT. And then you industry, let's take IT. And then you industry, let's take IT. And then you ask it to like rank stocks based on some ask it to like rank stocks based on some ask it to like rank stocks based on some parameters like capital expenditure or parameters like capital expenditure or parameters like capital expenditure or let's say the AI um let's say the AI um let's say the AI um uh investments. So, you pick like very uh investments. So, you pick like very uh investments. So, you pick like very specific things, and then you specific things, and then you specific things, and then you uh sort of formulate a very narrow job uh sort of formulate a very narrow job uh sort of formulate a very narrow job for the AI agent to do. And you can for the AI agent to do. And you can for the AI agent to do. And you can build like n number of AI agent. Last I build like n number of AI agent. Last I build like n number of AI agent. Last I checked, there was no tax on building checked, there was no tax on building checked, there was no tax on building more AI agents. So, why do you want your more AI agents. So, why do you want your more AI agents. So, why do you want your single agent to do everything?

single agent to do everything? single agent to do everything? So, this is important, and this is in So, this is important, and this is in So, this is important, and this is in the same uh in the pharma context is the the same uh in the pharma context is the the same uh in the pharma context is the exact same. We just break down the exact same. We just break down the exact same. We just break down the process into steps, and then ask really process into steps, and then ask really process into steps, and then ask really pointed questions with the agent. We pointed questions with the agent. We pointed questions with the agent. We model our agent for a task. model our agent for a task. model our agent for a task. So, once we have our problem right off So, once we have our problem right off So, once we have our problem right off the way, we know what we're trying to the way, we know what we're trying to the way, we know what we're trying to solve, the next step is identify the solve, the next step is identify the solve, the next step is identify the data. And I cannot stress this enough. data. And I cannot stress this enough. data. And I cannot stress this enough. This is a really, really, really This is a really, really, really This is a really, really, really important step, cuz everyone has news important step, cuz everyone has news important step, cuz everyone has news data. Everyone has like seller side data. Everyone has like seller side data. Everyone has like seller side reports from JP Morgan, Morgan Stanley. reports from JP Morgan, Morgan Stanley. reports from JP Morgan, Morgan Stanley. Uh everyone has the arXiv preprint Uh everyone has the arXiv preprint Uh everyone has the arXiv preprint server or PubChem or your research server or PubChem or your research server or PubChem or your research papers, right? But what actually makes papers, right? But what actually makes papers, right? But what actually makes your application better than let's say your application better than let's say your application better than let's say ChatGPT or Claude? It is your ChatGPT or Claude? It is your ChatGPT or Claude? It is your proprietary data. proprietary data. proprietary data. So, the thing with proprietary data is So, the thing with proprietary data is So, the thing with proprietary data is it's really expensive to buy, and most it's really expensive to buy, and most it's really expensive to buy, and most people won't sell it to you. So, you people won't sell it to you. So, you people won't sell it to you. So, you need to curate it by yourself. Imagine need to curate it by yourself. Imagine need to curate it by yourself. Imagine your organization has been working for 3 your organization has been working for 3 your organization has been working for 3 years, right? They already have a lot of years, right? They already have a lot of years, right? They already have a lot of data. It's just unstructured. And in the data. It's just unstructured. And in the data. It's just unstructured. And in the age of LLMs, I think this is a very age of LLMs, I think this is a very age of LLMs, I think this is a very fairly easy task to make unstructured fairly easy task to make unstructured fairly easy task to make unstructured data into structured data. Like a LLM data into structured data. Like a LLM data into structured data. Like a LLM workflow could do it overnight. So, workflow could do it overnight. So, workflow could do it overnight. So, to give you a great example of the to give you a great example of the to give you a great example of the proprietary data that finance industry proprietary data that finance industry proprietary data that finance industry has, it's the trade thesis, which is has, it's the trade thesis, which is has, it's the trade thesis, which is like what trade work and why it worked.

like what trade work and why it worked. like what trade work and why it worked. And in pharma, it is the data for failed And in pharma, it is the data for failed And in pharma, it is the data for failed experiments. For successful experiments experiments. For successful experiments experiments. For successful experiments data, yes, you can get it, but failed data, yes, you can get it, but failed data, yes, you can get it, but failed experiments, that's relatively hard to experiments, that's relatively hard to experiments, that's relatively hard to get. So, now we have the problem, we get. So, now we have the problem, we get. So, now we have the problem, we have the data. What's the third step? have the data. What's the third step? have the data. What's the third step? That is to model the problem, like write That is to model the problem, like write That is to model the problem, like write the prompt. the prompt. the prompt. So, while writing prompt, So, while writing prompt, So, while writing prompt, you like what we should aim is to model you like what we should aim is to model you like what we should aim is to model it after the person who you are trying it after the person who you are trying it after the person who you are trying to replace. I mean, that's the to replace. I mean, that's the to replace. I mean, that's the hypothesis, but yeah, no offense, we're hypothesis, but yeah, no offense, we're hypothesis, but yeah, no offense, we're not trying to replace anyone with AI, not trying to replace anyone with AI, not trying to replace anyone with AI, but that's the ideology behind writing but that's the ideology behind writing but that's the ideology behind writing prompts. Encode how a person would solve prompts. Encode how a person would solve prompts. Encode how a person would solve this job into multiple steps. So, it's this job into multiple steps. So, it's this job into multiple steps. So, it's just like a like a mental model. So, just like a like a mental model. So, just like a like a mental model. So, this is again fairly simple. Next thing, this is again fairly simple. Next thing, this is again fairly simple. Next thing, observability, I'm sure you have been in observability, I'm sure you have been in observability, I'm sure you have been in this conference at 3 years and this this conference at 3 years and this this conference at 3 years and this word, I think I don't know, you'll be word, I think I don't know, you'll be word, I think I don't know, you'll be hearing about like a thousandth time. hearing about like a thousandth time. hearing about like a thousandth time. There are tons of observability There are tons of observability There are tons of observability provider. provider. provider. If you can't see it, you can fix it. So, If you can't see it, you can fix it. So, If you can't see it, you can fix it. So, you need observability to see the you need observability to see the you need observability to see the traces, understand what your traces, understand what your traces, understand what your uh AI application is doing, and debug uh AI application is doing, and debug uh AI application is doing, and debug it.

it. it. So, sorry, but all of this was the easy So, sorry, but all of this was the easy So, sorry, but all of this was the easy part, to be honest. All of this fits one part, to be honest. All of this fits one part, to be honest. All of this fits one screen. The mythical 10x engineers can screen. The mythical 10x engineers can screen. The mythical 10x engineers can do this stuff in minutes. do this stuff in minutes. do this stuff in minutes. Like literally, this is the code you Like literally, this is the code you Like literally, this is the code you precisely need to build an AI agent. So, precisely need to build an AI agent. So, precisely need to build an AI agent. So, that's why it's not the moat. Uh of that's why it's not the moat. Uh of that's why it's not the moat. Uh of course, except your proprietary data. course, except your proprietary data. course, except your proprietary data. So, what do you do now? So, what do you do now? So, what do you do now? What do you do after doing the first What do you do after doing the first What do you do after doing the first four steps, four steps, four steps, which is observability, which is observability, which is observability, and prompts, and like uh getting the and prompts, and like uh getting the and prompts, and like uh getting the data right, and everything? UI trade. data right, and everything? UI trade. data right, and everything? UI trade. Now, the thing with iteration is like Now, the thing with iteration is like Now, the thing with iteration is like when I joined the hedge fund, I thought when I joined the hedge fund, I thought when I joined the hedge fund, I thought how hard it can be. I mean, everyone can how hard it can be. I mean, everyone can how hard it can be. I mean, everyone can iterate. I mean, we have been iterating iterate. I mean, we have been iterating iterate. I mean, we have been iterating our whole life for each of the task. our whole life for each of the task. our whole life for each of the task. But, But, But, to be honest, to be honest, to be honest, I could build it, but I just could not I could build it, but I just could not I could build it, but I just could not tell if it worked cuz tell if it worked cuz tell if it worked cuz I'm not a trader. I'm not someone who I'm not a trader. I'm not someone who I'm not a trader. I'm not someone who has a PhD in biology or chemistry. I has a PhD in biology or chemistry. I has a PhD in biology or chemistry. I just don't understand what the model is just don't understand what the model is just don't understand what the model is saying, what is the output of my AI saying, what is the output of my AI saying, what is the output of my AI agent is. And since most of you are agent is. And since most of you are agent is. And since most of you are engineers, you would relate. You can engineers, you would relate. You can engineers, you would relate. You can instantly tell that Sonnet 5 sucks instantly tell that Sonnet 5 sucks instantly tell that Sonnet 5 sucks because you have your own training. You because you have your own training. You because you have your own training. You understand, okay, this code is not great understand, okay, this code is not great understand, okay, this code is not great code. Whereas, some X model, let's say code. Whereas, some X model, let's say code. Whereas, some X model, let's say Fable 5, you see, okay, this is great Fable 5, you see, okay, this is great Fable 5, you see, okay, this is great but not as great as the high base cuz but not as great as the high base cuz but not as great as the high base cuz you've been trained for this for life.

you've been trained for this for life. you've been trained for this for life. You have You have You have a mental model to judge these things. a mental model to judge these things. a mental model to judge these things. But, you just do not have the same kind But, you just do not have the same kind But, you just do not have the same kind of mental model when it comes to like of mental model when it comes to like of mental model when it comes to like predicting trade thesis is or doing like predicting trade thesis is or doing like predicting trade thesis is or doing like really specific task that vertically our really specific task that vertically our really specific task that vertically our industry does. And this is also the industry does. And this is also the industry does. And this is also the place where like a lot of vertical AI place where like a lot of vertical AI place where like a lot of vertical AI projects quietly die because on the projects quietly die because on the projects quietly die because on the surface it looks like you have made it, surface it looks like you have made it, surface it looks like you have made it, you have built it, let's put this into you have built it, let's put this into you have built it, let's put this into production and start selling it. But, no production and start selling it. But, no production and start selling it. But, no one would buy it the same way you won't one would buy it the same way you won't one would buy it the same way you won't use an inferior coding model. use an inferior coding model. use an inferior coding model. So, as an engineer when I ran into this, So, as an engineer when I ran into this, So, as an engineer when I ran into this, I just couldn't accept honestly. I I just couldn't accept honestly. I I just couldn't accept honestly. I thought, no, there's certainly more that thought, no, there's certainly more that thought, no, there's certainly more that I can do. We don't need other people. I can do. We don't need other people. I can do. We don't need other people. So, I thought I could So, I thought I could So, I thought I could LLM as a judge my way out of it. LLM as a judge my way out of it. LLM as a judge my way out of it. >> [sighs and laughter] >> [sighs and laughter] >> [sighs and laughter] >> And this was a really, really stupid >> And this was a really, really stupid >> And this was a really, really stupid mistake to be honest cuz what LLM is mistake to be honest cuz what LLM is mistake to be honest cuz what LLM is essentially doing, it's it's predicting essentially doing, it's it's predicting essentially doing, it's it's predicting the next probable word. So, if you see, the next probable word. So, if you see, the next probable word. So, if you see, it's just like jargoning its way out. It it's just like jargoning its way out. It it's just like jargoning its way out. It does not understand what alpha means. It does not understand what alpha means. It does not understand what alpha means. It does not understand how to actually does not understand how to actually does not understand how to actually create value unless you have like taught create value unless you have like taught create value unless you have like taught it some way. And whereas a human can it some way. And whereas a human can it some way. And whereas a human can just tell it instantly what's just tell it instantly what's just tell it instantly what's and what's not.

and what's not. and what's not. So, So, So, I'll just try to dwell a bit more deeper I'll just try to dwell a bit more deeper I'll just try to dwell a bit more deeper on why you can just iterate. on why you can just iterate. on why you can just iterate. So, first thing is that model cannot So, first thing is that model cannot So, first thing is that model cannot verify itself, specifically in these verify itself, specifically in these verify itself, specifically in these fields, because reinforcement learning fields, because reinforcement learning fields, because reinforcement learning via verifiable rewards is really good at via verifiable rewards is really good at via verifiable rewards is really good at math and code because you have like math and code because you have like math and code because you have like answer keys, you can verify your code is answer keys, you can verify your code is answer keys, you can verify your code is uh compiling or not, and there are tons uh compiling or not, and there are tons uh compiling or not, and there are tons of stuff you can just model uh the of stuff you can just model uh the of stuff you can just model uh the complete thing around this. But, when in complete thing around this. But, when in complete thing around this. But, when in these fields, there is just no way to these fields, there is just no way to these fields, there is just no way to model it. And And let's say if any error model it. And And let's say if any error model it. And And let's say if any error gets in, it's just compounds with every gets in, it's just compounds with every gets in, it's just compounds with every stuff. And that's what LeCun seems to stuff. And that's what LeCun seems to stuff. And that's what LeCun seems to think as well. And now, the more think as well. And now, the more think as well. And now, the more important part that we touched upon important part that we touched upon important part that we touched upon previously, the data. previously, the data. previously, the data. So, the interesting thing with pharma So, the interesting thing with pharma So, the interesting thing with pharma and finance is the data was never there. and finance is the data was never there. and finance is the data was never there. And I'll explain to you why. So, any institutional manager holding So, any institutional manager holding over $100 million in qualifying US over $100 million in qualifying US over $100 million in qualifying US equities are forced to publicly file equities are forced to publicly file equities are forced to publicly file their holdings, long position holdings, their holdings, long position holdings, their holdings, long position holdings, every quarter. And once a hedge fund every quarter. And once a hedge fund every quarter. And once a hedge fund does this, does this, does this, this is the percentage decrease in their this is the percentage decrease in their this is the percentage decrease in their returns because everyone just sees those returns because everyone just sees those returns because everyone just sees those reverse engineers and takes away their reverse engineers and takes away their reverse engineers and takes away their moat.

moat. moat. And when it comes to pharma, right? So, And when it comes to pharma, right? So, And when it comes to pharma, right? So, this is the number of uh so, by law, you this is the number of uh so, by law, you this is the number of uh so, by law, you are like required to disclose every are like required to disclose every are like required to disclose every clinical trial pass or failure you have clinical trial pass or failure you have clinical trial pass or failure you have done. But, 30% of the funds, which is done. But, 30% of the funds, which is done. But, 30% of the funds, which is like nearly 1/3 of firms, never do. And like nearly 1/3 of firms, never do. And like nearly 1/3 of firms, never do. And in like 2026, FDA had to like publicly in like 2026, FDA had to like publicly in like 2026, FDA had to like publicly remind over, I don't know, about 2,000 remind over, I don't know, about 2,000 remind over, I don't know, about 2,000 sponsors sponsors sponsors that they are, I mean, doing injustice that they are, I mean, doing injustice that they are, I mean, doing injustice by not uh releasing unfavorable results by not uh releasing unfavorable results by not uh releasing unfavorable results because this is the exact data which because this is the exact data which because this is the exact data which helps the model thing, which helps your helps the model thing, which helps your helps the model thing, which helps your LLM actually reason through these LLM actually reason through these LLM actually reason through these complex and niche industries. And they complex and niche industries. And they complex and niche industries. And they hide it because for them, it's like a hide it because for them, it's like a hide it because for them, it's like a chicken laying golden eggs. Why would chicken laying golden eggs. Why would chicken laying golden eggs. Why would they sell their chicken? So, naturally, they sell their chicken? So, naturally, they sell their chicken? So, naturally, neither OpenAI nor Anthropic has that neither OpenAI nor Anthropic has that neither OpenAI nor Anthropic has that has this data because it's like has this data because it's like has this data because it's like gatekeeper. You just cannot hire a gatekeeper. You just cannot hire a gatekeeper. You just cannot hire a trader for $100 an hour and have them trader for $100 an hour and have them trader for $100 an hour and have them annotate that stuff because there's like annotate that stuff because there's like annotate that stuff because there's like lots of NDAs and lots of NDAs and lots of NDAs and they definitely earn more. So, okay, now they definitely earn more. So, okay, now they definitely earn more. So, okay, now I have told you about tens of problems, I have told you about tens of problems, I have told you about tens of problems, right? Now, you would naturally think, right? Now, you would naturally think, right? Now, you would naturally think, okay, yeah, right, then what do we do?

okay, yeah, right, then what do we do? okay, yeah, right, then what do we do? How do we build a startup in like a How do we build a startup in like a How do we build a startup in like a vertical space space? vertical space space? vertical space space? So, very self-explanatory, you hire the very self-explanatory, you hire the person who you want to sell it to cuz person who you want to sell it to cuz person who you want to sell it to cuz there is, to be honest, no other way there is, to be honest, no other way there is, to be honest, no other way around. I have tried a lot of stuff. You around. I have tried a lot of stuff. You around. I have tried a lot of stuff. You just need to hire the user. just need to hire the user. just need to hire the user. In finance in a hedge fund, this was In finance in a hedge fund, this was In finance in a hedge fund, this was very easy because the user was kind of very easy because the user was kind of very easy because the user was kind of like my boss, the trader. We worked like my boss, the trader. We worked like my boss, the trader. We worked together, but in the PharmaTech startup, together, but in the PharmaTech startup, together, but in the PharmaTech startup, it was very weird. We were like a bunch it was very weird. We were like a bunch it was very weird. We were like a bunch of young engineers and we were like, oh, of young engineers and we were like, oh, of young engineers and we were like, oh, we need a 20-year-old scientist in our we need a 20-year-old scientist in our we need a 20-year-old scientist in our company to tell us what to do? Yeah, I company to tell us what to do? Yeah, I company to tell us what to do? Yeah, I guess we do. And then we hired someone, guess we do. And then we hired someone, guess we do. And then we hired someone, right? And that someone actually changed right? And that someone actually changed right? And that someone actually changed the trajectory of our tools. Our tools the trajectory of our tools. Our tools the trajectory of our tools. Our tools started making sense. When we pitched to started making sense. When we pitched to started making sense. When we pitched to the other pharma companies, the big the other pharma companies, the big the other pharma companies, the big ones, the big pharma, they started ones, the big pharma, they started ones, the big pharma, they started liking our tools because it's kind of liking our tools because it's kind of liking our tools because it's kind of spoke their language versus the normal spoke their language versus the normal spoke their language versus the normal jargonish LLM language. So, jargonish LLM language. So, jargonish LLM language. So, once you have hired the user, let's say, once you have hired the user, let's say, once you have hired the user, let's say, then what would you make that user do? then what would you make that user do? then what would you make that user do? You try to build a learning loop out of You try to build a learning loop out of You try to build a learning loop out of it.

it. it. The domain expert can start at the like The domain expert can start at the like The domain expert can start at the like a very, very low level, the ground a very, very low level, the ground a very, very low level, the ground level, where they just think about level, where they just think about level, where they just think about prompts. Okay, yeah, I mean, let's not prompts. Okay, yeah, I mean, let's not prompts. Okay, yeah, I mean, let's not ask ask ask LLM to do this. Let's ask a very LLM to do this. Let's ask a very LLM to do this. Let's ask a very specific query again. They'll help you specific query again. They'll help you specific query again. They'll help you curate data. Just like engineers know curate data. Just like engineers know curate data. Just like engineers know which conferences are which which conferences are which which conferences are which are not, which research paper sites are are not, which research paper sites are are not, which research paper sites are great, which are not, which are like top great, which are not, which are like top great, which are not, which are like top leaders in engineering, which is which leaders in engineering, which is which leaders in engineering, which is which are just like influencers. Similarly, a are just like influencers. Similarly, a are just like influencers. Similarly, a pharma expert or let's say a trader pharma expert or let's say a trader pharma expert or let's say a trader knows which sources are more reliable knows which sources are more reliable knows which sources are more reliable than the other. So, they help you create than the other. So, they help you create than the other. So, they help you create their data. They help you like refine their data. They help you like refine their data. They help you like refine your prompts better, and they try to your prompts better, and they try to your prompts better, and they try to create like thinking models of how they create like thinking models of how they create like thinking models of how they would think about a problem. Cuz I mean, would think about a problem. Cuz I mean, would think about a problem. Cuz I mean, let's say if you if you follow five let's say if you if you follow five let's say if you if you follow five steps to solve a problem, right? You steps to solve a problem, right? You steps to solve a problem, right? You just cannot do it in in any random just cannot do it in in any random just cannot do it in in any random order. There has to be a logical flow. order. There has to be a logical flow. order. There has to be a logical flow. There has to be a natural flow. That's There has to be a natural flow. That's There has to be a natural flow. That's So, that's what they So, that's what they So, that's what they uh try to curate like decompose a uh try to curate like decompose a uh try to curate like decompose a problem, gradually refine, and then problem, gradually refine, and then problem, gradually refine, and then finally judge. So, the person finally judge. So, the person finally judge. So, the person who sort of has lived through the who sort of has lived through the who sort of has lived through the complete of the industry that they're complete of the industry that they're complete of the industry that they're trying to revolutionize, their judgment trying to revolutionize, their judgment trying to revolutionize, their judgment is now like turning into agents. So, is now like turning into agents. So, is now like turning into agents. So, that's what's happening behind the loop.

that's what's happening behind the loop. that's what's happening behind the loop. So, uh to do this there are like again So, uh to do this there are like again So, uh to do this there are like again multiple ways. I mean, each of these multiple ways. I mean, each of these multiple ways. I mean, each of these could have been a hour-long session on could have been a hour-long session on could have been a hour-long session on its own. And I wish I could take, but its own. And I wish I could take, but its own. And I wish I could take, but these are like few ways that I these are like few ways that I these are like few ways that I identified. Uh I'll just like take uh identified. Uh I'll just like take uh identified. Uh I'll just like take uh you through them like really quickly in you through them like really quickly in you through them like really quickly in the interest of time. So, supervised the interest of time. So, supervised the interest of time. So, supervised fine-tuning I think most of you would fine-tuning I think most of you would fine-tuning I think most of you would know where like model mimics human nest know where like model mimics human nest know where like model mimics human nest demonstrations. Uh reinforcement demonstrations. Uh reinforcement demonstrations. Uh reinforcement learning from human feedback is like a learning from human feedback is like a learning from human feedback is like a kind of uh a very efficient way where kind of uh a very efficient way where kind of uh a very efficient way where human preferences train a reward model. human preferences train a reward model. human preferences train a reward model. Then rubrics as a reward is I I like to Then rubrics as a reward is I I like to Then rubrics as a reward is I I like to call it reinforcement learning from AI call it reinforcement learning from AI call it reinforcement learning from AI feedback. feedback. feedback. This is because that you can human can This is because that you can human can This is because that you can human can just create a rubric, and then AI will just create a rubric, and then AI will just create a rubric, and then AI will just like grade itself based on that just like grade itself based on that just like grade itself based on that rubric, and that improve its own rubric, and that improve its own rubric, and that improve its own processes. But again, there is a slight processes. But again, there is a slight processes. But again, there is a slight chance that you might run into an echo chance that you might run into an echo chance that you might run into an echo chamber with rubrics as rewards. And the chamber with rubrics as rewards. And the chamber with rubrics as rewards. And the cheapest of all, and I think the highest cheapest of all, and I think the highest cheapest of all, and I think the highest ROI is the error analysis. Whereas the ROI is the error analysis. Whereas the ROI is the error analysis. Whereas the observability part that you set up observability part that you set up observability part that you set up earlier, you just analyze the logs plain earlier, you just analyze the logs plain earlier, you just analyze the logs plain and simple. You understand where model and simple. You understand where model and simple. You understand where model is going wrong, and then you just try to is going wrong, and then you just try to is going wrong, and then you just try to correct it. So, correct it. So, correct it. So, this is where you have like don't have this is where you have like don't have this is where you have like don't have to touch any weights, and the most to touch any weights, and the most to touch any weights, and the most highest ROI way to get the impact from a highest ROI way to get the impact from a highest ROI way to get the impact from a like start on. And once you understand like start on. And once you understand like start on. And once you understand like uh what more you could do, or if like uh what more you could do, or if like uh what more you could do, or if error analysis is solving or not, you error analysis is solving or not, you error analysis is solving or not, you can just gradually climb up the ladder, can just gradually climb up the ladder, can just gradually climb up the ladder, and probably uh later on go to the and probably uh later on go to the and probably uh later on go to the ultimate reinforcement learning from ultimate reinforcement learning from ultimate reinforcement learning from human feedback cuz that's I think in our human feedback cuz that's I think in our human feedback cuz that's I think in our industry kind of the golden standard industry kind of the golden standard industry kind of the golden standard these days that you need to do RLHF to

these days that you need to do RLHF to these days that you need to do RLHF to actually get some edge. actually get some edge. actually get some edge. But uh certainly there are some pitfalls But uh certainly there are some pitfalls But uh certainly there are some pitfalls of it. Like for example, of it. Like for example, of it. Like for example, now there's GLM 5.2, right? now there's GLM 5.2, right? now there's GLM 5.2, right? You fine-tuned it, right? You fine-tuned it, right? You fine-tuned it, right? Uh Alibaba Cloud or let's say Deep Seek Uh Alibaba Cloud or let's say Deep Seek Uh Alibaba Cloud or let's say Deep Seek will release a newer model, then you will release a newer model, then you will release a newer model, then you have to fine-tune that too as well. So have to fine-tune that too as well. So have to fine-tune that too as well. So there is a cost. It's not cheap. there is a cost. It's not cheap. there is a cost. It's not cheap. So once you have done all this, you just So once you have done all this, you just So once you have done all this, you just create a loop and you just like go on to create a loop and you just like go on to create a loop and you just like go on to that loop. You hired one user, you hire that loop. You hired one user, you hire that loop. You hired one user, you hire more users, they ask more queries, the more users, they ask more queries, the more users, they ask more queries, the scoping increases, the data increases. scoping increases, the data increases. scoping increases, the data increases. At this point you're kind of generating At this point you're kind of generating At this point you're kind of generating your own data. The exercise you have your own data. The exercise you have your own data. The exercise you have been doing in loop, right? That exercise been doing in loop, right? That exercise been doing in loop, right? That exercise itself is generating a very I would say itself is generating a very I would say itself is generating a very I would say a crazy data set of what works and what a crazy data set of what works and what a crazy data set of what works and what does not work. And this loop never does not work. And this loop never does not work. And this loop never stops. Once you feel confident enough in stops. Once you feel confident enough in stops. Once you feel confident enough in your application, you just ship it, your application, you just ship it, your application, you just ship it, provide it to the external paying users, provide it to the external paying users, provide it to the external paying users, and then you see the magic of it that it and then you see the magic of it that it and then you see the magic of it that it actually works. So actually works. So actually works. So I just pulled the stat from Stanford AI I just pulled the stat from Stanford AI I just pulled the stat from Stanford AI Index report cuz it's a really nice Index report cuz it's a really nice Index report cuz it's a really nice report that gives you an idea of what report that gives you an idea of what report that gives you an idea of what the state of AI is. And this says like the state of AI is. And this says like the state of AI is. And this says like 80% 89% of enterprise AI agents never 80% 89% of enterprise AI agents never 80% 89% of enterprise AI agents never reach production. Again, I disagree.

reach production. Again, I disagree. reach production. Again, I disagree. Every AI reaches production, but it just Every AI reaches production, but it just Every AI reaches production, but it just fails to work or like justify its own fails to work or like justify its own fails to work or like justify its own cost. So that's the real thing. You can cost. So that's the real thing. You can cost. So that's the real thing. You can just build and ship AI agents whenever just build and ship AI agents whenever just build and ship AI agents whenever you want, but you need to justify ROI. you want, but you need to justify ROI. you want, but you need to justify ROI. And finance and pharma are two such And finance and pharma are two such And finance and pharma are two such industries where if it does not make industries where if it does not make industries where if it does not make money, it's shown the door. money, it's shown the door. money, it's shown the door. Simple. They won't like wait and say, Simple. They won't like wait and say, Simple. They won't like wait and say, "Okay, maybe it'll work in 2 years. "Okay, maybe it'll work in 2 years. "Okay, maybe it'll work in 2 years. Maybe the cost will be lower in by the Maybe the cost will be lower in by the Maybe the cost will be lower in by the third year." No. It has to instantly third year." No. It has to instantly third year." No. It has to instantly make money. It has to like hit the make money. It has to like hit the make money. It has to like hit the ground running. And if it does not, ground running. And if it does not, ground running. And if it does not, shown the door instantly. So just to shown the door instantly. So just to shown the door instantly. So just to summarize the seven steps that I feel summarize the seven steps that I feel summarize the seven steps that I feel are like good enough to give you an are like good enough to give you an are like good enough to give you an abstraction of how the vertical AI abstraction of how the vertical AI abstraction of how the vertical AI industry moves. industry moves. industry moves. You formulate the problem statement, you You formulate the problem statement, you You formulate the problem statement, you source your data sources, you prompt it source your data sources, you prompt it source your data sources, you prompt it well, you define those prompts, you well, you define those prompts, you well, you define those prompts, you observe how your tool is performing, you observe how your tool is performing, you observe how your tool is performing, you don't iterate yet, you hire the user. don't iterate yet, you hire the user. don't iterate yet, you hire the user. And this user or users now play with the And this user or users now play with the And this user or users now play with the tool as much as possible. They like kind tool as much as possible. They like kind tool as much as possible. They like kind of form a learning loop, an endless of form a learning loop, an endless of form a learning loop, an endless learning loop that goes on and at a learning loop that goes on and at a learning loop that goes on and at a point when you feel yeah, it's it's point when you feel yeah, it's it's point when you feel yeah, it's it's really delivering that alpha over let's really delivering that alpha over let's really delivering that alpha over let's say Claude and ChatGPT, you just ship say Claude and ChatGPT, you just ship say Claude and ChatGPT, you just ship it, you start earning money.

it, you start earning money. it, you start earning money. So, So, So, one more interesting thing. So, HITL is one more interesting thing. So, HITL is one more interesting thing. So, HITL is like kind of a thing everyone is like like kind of a thing everyone is like like kind of a thing everyone is like yeah, let's add human in the loop. I yeah, let's add human in the loop. I yeah, let's add human in the loop. I would say not yet. Finance and pharma would say not yet. Finance and pharma would say not yet. Finance and pharma are still those two industries where are still those two industries where are still those two industries where it's AITL, AI in the loop cuz everything it's AITL, AI in the loop cuz everything it's AITL, AI in the loop cuz everything is like done by the expert, but the AI is like done by the expert, but the AI is like done by the expert, but the AI assistant really helps save time. Like assistant really helps save time. Like assistant really helps save time. Like for example, uh for example, uh for example, uh it may take an X amount for a trader to it may take an X amount for a trader to it may take an X amount for a trader to form different trade thesis, and AI can form different trade thesis, and AI can form different trade thesis, and AI can just give him five candidate trade just give him five candidate trade just give him five candidate trade thesis, but which one would actually thesis, but which one would actually thesis, but which one would actually work in the market and which won't is work in the market and which won't is work in the market and which won't is the discussion the discussion still lies the discussion the discussion still lies the discussion the discussion still lies with the trader. And same for pharma with the trader. And same for pharma with the trader. And same for pharma when you're like picking drug when you're like picking drug when you're like picking drug candidates, which one to pick, the candidates, which one to pick, the candidates, which one to pick, the expert still does it, but you just like expert still does it, but you just like expert still does it, but you just like reduce the time of expert by a lot lot. reduce the time of expert by a lot lot. reduce the time of expert by a lot lot. So, and and it will stay this way for So, and and it will stay this way for So, and and it will stay this way for really long. So, really long. So, really long. So, uh for the models to actually make good uh for the models to actually make good uh for the models to actually make good decisions, they don't need to do decisions, they don't need to do decisions, they don't need to do correlation, they need to do causation. correlation, they need to do causation. correlation, they need to do causation. And And And as Ya as Jan LeCun puts it, these are as Ya as Jan LeCun puts it, these are as Ya as Jan LeCun puts it, these are like text statistics, not real-world like text statistics, not real-world like text statistics, not real-world models. You cannot just pattern match models. You cannot just pattern match models. You cannot just pattern match with past and use future to predict to with past and use future to predict to with past and use future to predict to it. And so, we are like kind of not it. And so, we are like kind of not it. And so, we are like kind of not there yet. That's what I call as the AGI there yet. That's what I call as the AGI there yet. That's what I call as the AGI line. Once we are there, yeah, probably line. Once we are there, yeah, probably line. Once we are there, yeah, probably then models will just like make drugs.

then models will just like make drugs. then models will just like make drugs. You will have vibe coded drugs. Someone You will have vibe coded drugs. Someone You will have vibe coded drugs. Someone would be vibe coding market, but yeah, would be vibe coding market, but yeah, would be vibe coding market, but yeah, not yet. not yet. not yet. So, So, So, a final takeaway that I would call if if a final takeaway that I would call if if a final takeaway that I would call if if if there's one thing you are taking away if there's one thing you are taking away if there's one thing you are taking away from this talk, this is it. from this talk, this is it. from this talk, this is it. Model infra ecosystem, everyone selling Model infra ecosystem, everyone selling Model infra ecosystem, everyone selling you tons of stuff at this conference is you tons of stuff at this conference is you tons of stuff at this conference is just commodity. Everyone has it. If you just commodity. Everyone has it. If you just commodity. Everyone has it. If you have it, everyone has it. Everyone can have it, everyone has it. Everyone can have it, everyone has it. Everyone can pay X number of dollars for a pay X number of dollars for a pay X number of dollars for a subscription. But, what is moat and no subscription. But, what is moat and no subscription. But, what is moat and no one will come and sell it to you. You one will come and sell it to you. You one will come and sell it to you. You won't have to curate it on your own is won't have to curate it on your own is won't have to curate it on your own is the domain expertise. You need your the domain expertise. You need your the domain expertise. You need your data. You need other people's data. That data. You need other people's data. That data. You need other people's data. That is just not out there on the internet. is just not out there on the internet. is just not out there on the internet. And that that's what will form your And that that's what will form your And that that's what will form your moat. So, thank you for your time. I moat. So, thank you for your time. I moat. So, thank you for your time. I think you enjoyed the talk and yeah, let think you enjoyed the talk and yeah, let think you enjoyed the talk and yeah, let me know if you have any questions. We me know if you have any questions. We me know if you have any questions. We can meet outside. Thank you. can meet outside. Thank you. can meet outside. Thank you. >> [applause]