How to Build AI You Can Trust, With Adyen's Dhruv Ghulati

Hi, welcome to Blueprint Plus+ a webinar Hi, welcome to Blueprint Plus+ a webinar series in intersection of business and series in intersection of business and series in intersection of business and technology and our guest today is Duf technology and our guest today is Duf technology and our guest today is Duf Galati a product leader at Aden. Hi Galati a product leader at Aden. Hi Galati a product leader at Aden. Hi Droo, nice to see you here. So uh let's Droo, nice to see you here. So uh let's Droo, nice to see you here. So uh let's start with a little background. Could start with a little background. Could start with a little background. Could you tell a bit about yourself, how you you tell a bit about yourself, how you you tell a bit about yourself, how you got here and what you're working on got here and what you're working on got here and what you're working on right now? right now? right now? >> Sure. Um yeah so my background I >> Sure. Um yeah so my background I >> Sure. Um yeah so my background I actually started off my career in actually started off my career in actually started off my career in finance and banking. Uh I my first job finance and banking. Uh I my first job finance and banking. Uh I my first job was at Meil Lynch on the trading floor. was at Meil Lynch on the trading floor. was at Meil Lynch on the trading floor. So I have I had nothing to do with So I have I had nothing to do with So I have I had nothing to do with technology but this was around 2014 I technology but this was around 2014 I technology but this was around 2014 I got very interested in in tech and got got very interested in in tech and got got very interested in in tech and got into AI back then. Yeah. And then my into AI back then. Yeah. And then my into AI back then. Yeah. And then my career has spanned kind of being an career has spanned kind of being an career has spanned kind of being an early stage founder. Um I was in London early stage founder. Um I was in London early stage founder. Um I was in London and I was working on a company that was and I was working on a company that was and I was working on a company that was detecting misinformation, disinformation detecting misinformation, disinformation detecting misinformation, disinformation online. And then after exiting my first online. And then after exiting my first online. And then after exiting my first company um I decided to tune in and hone company um I decided to tune in and hone company um I decided to tune in and hone in some of the skill sets that I wanted in some of the skill sets that I wanted in some of the skill sets that I wanted to gain which is around AI product to gain which is around AI product to gain which is around AI product management because I think management because I think management because I think >> I felt that to be a great technical >> I felt that to be a great technical >> I felt that to be a great technical leader um and and be a great leader um and and be a great leader um and and be a great entrepreneur actually you have to entrepreneur actually you have to entrepreneur actually you have to understand product. Then I moved to understand product. Then I moved to understand product. Then I moved to Amsterdam uh where I was leading out um Amsterdam uh where I was leading out um Amsterdam uh where I was leading out um the Uber AI team on product um in the Uber AI team on product um in the Uber AI team on product um in Amsterdam. So I was the first site lead Amsterdam. So I was the first site lead Amsterdam. So I was the first site lead here. Um, so worked on a lot of here. Um, so worked on a lot of here. Um, so worked on a lot of interesting AI problems across mobility, interesting AI problems across mobility, interesting AI problems across mobility, Uber Eatats, delivery, forecasting Uber Eatats, delivery, forecasting Uber Eatats, delivery, forecasting problems, and we can talk talk a bit problems, and we can talk talk a bit problems, and we can talk talk a bit about those. Um, and currently I'm a about those. Um, and currently I'm a about those. Um, and currently I'm a principal PM at Aden. So I'm I'm leading principal PM at Aden. So I'm I'm leading principal PM at Aden. So I'm I'm leading out some of their their goals of out some of their their goals of out some of their their goals of becoming and and building out some of becoming and and building out some of becoming and and building out some of their their AI products u for payments.

their their AI products u for payments. their their AI products u for payments. >> Mhm. >> Mhm. >> Mhm. >> Yeah. And you've been telling about the >> Yeah. And you've been telling about the >> Yeah. And you've been telling about the uh your first startup basically uh your first startup basically uh your first startup basically matter. Uh so how do you feel nowadays matter. Uh so how do you feel nowadays matter. Uh so how do you feel nowadays uh about the fact that you've started uh about the fact that you've started uh about the fact that you've started using um machine learning methods and using um machine learning methods and using um machine learning methods and PhD folks were uh in your company PhD folks were uh in your company PhD folks were uh in your company working on facteing but nowadays machine working on facteing but nowadays machine working on facteing but nowadays machine learning capabilities are more kind of a learning capabilities are more kind of a learning capabilities are more kind of a contributing into that problem. I'd say contributing into that problem. I'd say contributing into that problem. I'd say like a lot of my my my career has been like a lot of my my my career has been like a lot of my my my career has been about I guess like my company was about I guess like my company was about I guess like my company was detecting information that you just detecting information that you just detecting information that you just don't want to have on your website or on don't want to have on your website or on don't want to have on your website or on your ad network and it was kind of your ad network and it was kind of your ad network and it was kind of detecting bad things, right? Same thing detecting bad things, right? Same thing detecting bad things, right? Same thing with uh when I was at Onfido I was with uh when I was at Onfido I was with uh when I was at Onfido I was detecting counterfeits and forged detecting counterfeits and forged detecting counterfeits and forged documents. So um now even at at uh at documents. So um now even at at uh at documents. So um now even at at uh at Aden I'm working on chargeback fraud Aden I'm working on chargeback fraud Aden I'm working on chargeback fraud detection. So I've always been detecting detection. So I've always been detecting detection. So I've always been detecting kind of fraud bad things and I think kind of fraud bad things and I think kind of fraud bad things and I think that to understand and be able to do that to understand and be able to do that to understand and be able to do that effectively you have to understand that effectively you have to understand that effectively you have to understand how that information is generated. how that information is generated. how that information is generated. >> You have to go very deep into fraud >> You have to go very deep into fraud >> You have to go very deep into fraud patterns you know generative AI for patterns you know generative AI for patterns you know generative AI for disinformation creation. What is the disinformation creation. What is the disinformation creation. What is the motivation of the attacker? What is the motivation of the attacker? What is the motivation of the attacker? What is the motivation of the creator? What motivation of the creator? What motivation of the creator? What techniques are they likely to deploy?

techniques are they likely to deploy? techniques are they likely to deploy? >> If you don't understand that you can't >> If you don't understand that you can't >> If you don't understand that you can't build detection systems for that. So, build detection systems for that. So, build detection systems for that. So, um, you know, a lot of this is is is um, you know, a lot of this is is is um, you know, a lot of this is is is actually a cat-and- mouse game. And I actually a cat-and- mouse game. And I actually a cat-and- mouse game. And I think we're going to still have this think we're going to still have this think we're going to still have this through the internet, right? There's through the internet, right? There's through the internet, right? There's always people who there's this wave of always people who there's this wave of always people who there's this wave of of creation of things. Um, and then of creation of things. Um, and then of creation of things. Um, and then there's this wave of regulation and there's this wave of regulation and there's this wave of regulation and moderation and and making sure that moderation and and making sure that moderation and and making sure that things are are safe. Talking about uh things are are safe. Talking about uh things are are safe. Talking about uh Aden actually, so you said chargeback uh Aden actually, so you said chargeback uh Aden actually, so you said chargeback uh fraud detection and it seems like coming fraud detection and it seems like coming fraud detection and it seems like coming through different companies, you've been through different companies, you've been through different companies, you've been getting in more and more government uh getting in more and more government uh getting in more and more government uh companies. So Aden for example is a companies. So Aden for example is a companies. So Aden for example is a financial corporation and probably financial corporation and probably financial corporation and probably chargeback uh detection is a lot more chargeback uh detection is a lot more chargeback uh detection is a lot more governed question and anti fraud governed question and anti fraud governed question and anti fraud detection than for example what was done detection than for example what was done detection than for example what was done in a fido or before that in a fact in a fido or before that in a fact in a fido or before that in a fact matter so how do you feel about that matter so how do you feel about that matter so how do you feel about that whole governance procedures that are whole governance procedures that are whole governance procedures that are being applied in aden how did it change being applied in aden how did it change being applied in aden how did it change from the other companies and how does it from the other companies and how does it from the other companies and how does it affect the AI capabilities that you are affect the AI capabilities that you are affect the AI capabilities that you are able to use in the company able to use in the company able to use in the company >> yeah interesting yeah I think I think um >> yeah interesting yeah I think I think um >> yeah interesting yeah I think I think um I think your question is a a lot about I think your question is a a lot about I think your question is a a lot about sort of what are the different ranges of sort of what are the different ranges of sort of what are the different ranges of security that you need to have towards security that you need to have towards security that you need to have towards towards AI models. Um towards AI models. Um towards AI models. Um >> the fact is I think in the last kind of >> the fact is I think in the last kind of >> the fact is I think in the last kind of probably and it's accelerated probably and it's accelerated probably and it's accelerated particularly in Europe, right? We've had particularly in Europe, right? We've had particularly in Europe, right? We've had the AI act that's been coming out and I the AI act that's been coming out and I the AI act that's been coming out and I think that's meant to go live. I I don't think that's meant to go live. I I don't think that's meant to go live. I I don't know the exact date, but I think that know the exact date, but I think that know the exact date, but I think that might actually be live now.

might actually be live now. might actually be live now. >> Yeah. >> Yeah. >> Yeah. >> Um but but a lot of the wave has been to >> Um but but a lot of the wave has been to >> Um but but a lot of the wave has been to for AI regulators um and and and for AI regulators um and and and for AI regulators um and and and governments to be sort of trying to governments to be sort of trying to governments to be sort of trying to catch up, right? because AI has moved so catch up, right? because AI has moved so catch up, right? because AI has moved so quickly that um you know you don't want quickly that um you know you don't want quickly that um you know you don't want to be in a regulatory situation where to be in a regulatory situation where to be in a regulatory situation where effectively the regulation is is is effectively the regulation is is is effectively the regulation is is is slowing things down for no reason or the slowing things down for no reason or the slowing things down for no reason or the actual regulation makes zero sense. I actual regulation makes zero sense. I actual regulation makes zero sense. I think one of the the the key things that think one of the the the key things that think one of the the the key things that you typically had with AI is like what you typically had with AI is like what you typically had with AI is like what what data is it trained on? what data is it trained on? what data is it trained on? >> Mhm. >> Mhm. >> Mhm. >> So you know does it contain personal >> So you know does it contain personal >> So you know does it contain personal information? Is it trained on information? Is it trained on information? Is it trained on information that's kind of like a information that's kind of like a information that's kind of like a personally identifiable? personally identifiable? personally identifiable? >> Yeah. Does it have biases? >> Yeah. Does it have biases? >> Yeah. Does it have biases? >> For sure. does it have biases in the >> For sure. does it have biases in the >> For sure. does it have biases in the training data? But then there's the training data? But then there's the training data? But then there's the decision element, right? So if you have decision element, right? So if you have decision element, right? So if you have uh AI models that are taking real uh AI models that are taking real uh AI models that are taking real decisions where they actually affect decisions where they actually affect decisions where they actually affect people's lives, um that's where AI people's lives, um that's where AI people's lives, um that's where AI regulation has has has focused very regulation has has has focused very regulation has has has focused very heavily, right? So whether it's for heavily, right? So whether it's for heavily, right? So whether it's for example getting a loan application example getting a loan application example getting a loan application denied or accepted. Mhm. denied or accepted. Mhm. denied or accepted. Mhm. >> Um, we're going to go through a lot of >> Um, we're going to go through a lot of >> Um, we're going to go through a lot of that now because you're going to have that now because you're going to have that now because you're going to have agents, you know, as you have a world agents, you know, as you have a world agents, you know, as you have a world where the the the drive is to automate where the the the drive is to automate where the the the drive is to automate as many things as possible and manual as many things as possible and manual as many things as possible and manual processes. You got to remember they were processes. You got to remember they were processes. You got to remember they were manual for a reason. You had a human in manual for a reason. You had a human in manual for a reason. You had a human in a loop in for a reason to do that loan a loop in for a reason to do that loan a loop in for a reason to do that loan application or to do the mortgage application or to do the mortgage application or to do the mortgage application. Um and then I think now application. Um and then I think now application. Um and then I think now regulators are sort of looking at um regulators are sort of looking at um regulators are sort of looking at um what you know it is a human in the loop what you know it is a human in the loop what you know it is a human in the loop because you actually need the human because you actually need the human because you actually need the human expertise or actually could a could a expertise or actually could a could a expertise or actually could a could a machine be doing this in a more fair machine be doing this in a more fair machine be doing this in a more fair way. Um I think that's the question that way. Um I think that's the question that way. Um I think that's the question that regulators are are asking. I think

regulators are are asking. I think regulators are are asking. I think another part of your question was around another part of your question was around another part of your question was around um you know how have I seen the um you know how have I seen the um you know how have I seen the different ranges of how AI company or or different ranges of how AI company or or different ranges of how AI company or or tech companies basically open up their tech companies basically open up their tech companies basically open up their data. I think um you have on the one data. I think um you have on the one data. I think um you have on the one hand of the spectrum you have very hand of the spectrum you have very hand of the spectrum you have very locked down closed user groups and user locked down closed user groups and user locked down closed user groups and user permissions where certain parts of the permissions where certain parts of the permissions where certain parts of the codebase are just not accessible to codebase are just not accessible to codebase are just not accessible to other people. Um you have um you know other people. Um you have um you know other people. Um you have um you know limitations on on downloading code or limitations on on downloading code or limitations on on downloading code or downloading files to local files. That's downloading files to local files. That's downloading files to local files. That's another thing that I've seen. Um you've another thing that I've seen. Um you've another thing that I've seen. Um you've had just access permissions denied. uh had just access permissions denied. uh had just access permissions denied. uh you've had sort of trying to ring fence you've had sort of trying to ring fence you've had sort of trying to ring fence access to data versus access to code access to data versus access to code access to data versus access to code versus access to um certain types of versus access to um certain types of versus access to um certain types of tools whether it's your CFKA pipelines tools whether it's your CFKA pipelines tools whether it's your CFKA pipelines or whether it's your um or whether it's your um or whether it's your um >> production databases >> production databases >> production databases >> production databases um I've seen and >> production databases um I've seen and >> production databases um I've seen and then on the other end of the spectrum then on the other end of the spectrum then on the other end of the spectrum when you have an early stage company um when you have an early stage company um when you have an early stage company um you have situations where just everyone you have situations where just everyone you have situations where just everyone has access to everything you don't have has access to everything you don't have has access to everything you don't have time to have that annoying thing where time to have that annoying thing where time to have that annoying thing where you're trying to access some information you're trying to access some information you're trying to access some information you don't have access that is a you don't have access that is a you don't have access that is a >> that's a pain right if you're trying to >> that's a pain right if you're trying to >> that's a pain right if you're trying to move move fast. I think that's basically move move fast. I think that's basically move move fast. I think that's basically the thing that companies are going the thing that companies are going the thing that companies are going through where they put all these access through where they put all these access through where they put all these access controls in place for a reason and now controls in place for a reason and now controls in place for a reason and now we're going to have to question like why we're going to have to question like why we're going to have to question like why are those what is the biggest risk that are those what is the biggest risk that are those what is the biggest risk that we can face by giving people access we can face by giving people access we can face by giving people access >> to that right whether it's a product >> to that right whether it's a product >> to that right whether it's a product manager whether it's a saleserson is the manager whether it's a saleserson is the manager whether it's a saleserson is the risk that they're going to for example risk that they're going to for example risk that they're going to for example interpret the data incorrectly and then interpret the data incorrectly and then interpret the data incorrectly and then that's going to lead to some some that's going to lead to some some that's going to lead to some some outcomes that incorrect outcomes that incorrect outcomes that incorrect >> um when they're you know from a data

>> um when they're you know from a data >> um when they're you know from a data access perspective is it that your access perspective is it that your access perspective is it that your you're scared of someone basically you're scared of someone basically you're scared of someone basically accessing the code, deleting some code. accessing the code, deleting some code. accessing the code, deleting some code. Um, I think you can create I think Um, I think you can create I think Um, I think you can create I think agents actually have a chance to create agents actually have a chance to create agents actually have a chance to create these p permission layers automatically, these p permission layers automatically, these p permission layers automatically, right? Um, whereas in the past it's been right? Um, whereas in the past it's been right? Um, whereas in the past it's been really manual to make really manual to make really manual to make >> you have to have a security expert. >> you have to have a security expert. >> you have to have a security expert. Yeah, Yeah, Yeah, >> you had to security expert, but you also >> you had to security expert, but you also >> you had to security expert, but you also have lots lots of meetings internally of have lots lots of meetings internally of have lots lots of meetings internally of like who gets access to this, who gets like who gets access to this, who gets like who gets access to this, who gets access to that. Different levels of access to that. Different levels of access to that. Different levels of permissioning I've seen. I think it has permissioning I've seen. I think it has permissioning I've seen. I think it has a benefit, right? Um but I think um well a benefit, right? Um but I think um well a benefit, right? Um but I think um well like with any security problem like you like with any security problem like you like with any security problem like you only know uh when things go wrong, only know uh when things go wrong, only know uh when things go wrong, right? So so it's kind of one of those right? So so it's kind of one of those right? So so it's kind of one of those things of like okay do we keep all this things of like okay do we keep all this things of like okay do we keep all this architecture going of these like really architecture going of these like really architecture going of these like really intense user access permissions? What intense user access permissions? What intense user access permissions? What are the risks or is actually is that are the risks or is actually is that are the risks or is actually is that slowing down our progress? Um do you see slowing down our progress? Um do you see slowing down our progress? Um do you see the uh do you see there a parallel with the uh do you see there a parallel with the uh do you see there a parallel with basically an application of agents? So basically an application of agents? So basically an application of agents? So as far as far as I see that uh when you as far as far as I see that uh when you as far as far as I see that uh when you have a small startup you have indeed have a small startup you have indeed have a small startup you have indeed access to everything in a lot of cases access to everything in a lot of cases access to everything in a lot of cases founder is the only one who is able to founder is the only one who is able to founder is the only one who is able to fix production right now and the same is fix production right now and the same is fix production right now and the same is actually happening with the companies actually happening with the companies actually happening with the companies who are trying to become more of a who are trying to become more of a who are trying to become more of a companies. They do see actually influx companies. They do see actually influx companies. They do see actually influx of like tens of thousands of new of like tens of thousands of new of like tens of thousands of new developers agents which actually have developers agents which actually have developers agents which actually have the very similar problems with the very similar problems with the very similar problems with permissions with trying to access the permissions with trying to access the permissions with trying to access the production database or trying to drop production database or trying to drop production database or trying to drop something. Do you see the barrel here?

something. Do you see the barrel here? something. Do you see the barrel here? >> Big time. I think I think one of the >> Big time. I think I think one of the >> Big time. I think I think one of the biggest things that that AI native biggest things that that AI native biggest things that that AI native companies are starting to do is is is companies are starting to do is is is companies are starting to do is is is for example just create MCPs create for example just create MCPs create for example just create MCPs create access to all these different tools that access to all these different tools that access to all these different tools that they were using. they were using. they were using. >> Um and it's magic when you can actually >> Um and it's magic when you can actually >> Um and it's magic when you can actually do that. You know uh you know I know do that. You know uh you know I know do that. You know uh you know I know that uh you know at a for example we that uh you know at a for example we that uh you know at a for example we have an internal team that's built this have an internal team that's built this have an internal team that's built this this this system for UTRA which is our this this system for UTRA which is our this this system for UTRA which is our kind of like task tracking system. kind of like task tracking system. kind of like task tracking system. >> Right. Yeah. So, so that basically, you >> Right. Yeah. So, so that basically, you >> Right. Yeah. So, so that basically, you know, like I was doing this yesterday, know, like I was doing this yesterday, know, like I was doing this yesterday, you know, a PM can basically start you know, a PM can basically start you know, a PM can basically start editing tickets, you know, on their editing tickets, you know, on their editing tickets, you know, on their terminal and and and adjusting them, terminal and and and adjusting them, terminal and and and adjusting them, changing the statuses, doing them in changing the statuses, doing them in changing the statuses, doing them in bulk. I mean, that would have taken so bulk. I mean, that would have taken so bulk. I mean, that would have taken so long, long, long, >> you know, pointing and clicking at at >> you know, pointing and clicking at at >> you know, pointing and clicking at at certain things. So, I think certain things. So, I think certain things. So, I think >> just that magical unlock of of unlocking >> just that magical unlock of of unlocking >> just that magical unlock of of unlocking access to your tools, I think every access to your tools, I think every access to your tools, I think every single um SAS company has to have an MCP single um SAS company has to have an MCP single um SAS company has to have an MCP strategy right now. And because the the strategy right now. And because the the strategy right now. And because the the the magic of of opening that up for like the magic of of opening that up for like the magic of of opening that up for like a new user base is is is huge, right? a new user base is is is huge, right? a new user base is is is huge, right? Like people really want to see, okay, Like people really want to see, okay, Like people really want to see, okay, does this that's how I think, right?

does this that's how I think, right? does this that's how I think, right? Like does this tool have an MCP or not? Like does this tool have an MCP or not? Like does this tool have an MCP or not? Is it integrated? Is it open? Is it integrated? Is it open? Is it integrated? Is it open? >> Um in a way we're in the we're in the >> Um in a way we're in the we're in the >> Um in a way we're in the we're in the same kind of world of APIs, right? Like same kind of world of APIs, right? Like same kind of world of APIs, right? Like okay, does this thing is this is this okay, does this thing is this is this okay, does this thing is this is this promoting an open ecosystem? M um and I promoting an open ecosystem? M um and I promoting an open ecosystem? M um and I think that's where um companies can think that's where um companies can think that's where um companies can really start like actually like the the really start like actually like the the really start like actually like the the the unlock you can get from creating the unlock you can get from creating the unlock you can get from creating toolkits and skills for your tools MCPS toolkits and skills for your tools MCPS toolkits and skills for your tools MCPS you know and access um creating little you know and access um creating little you know and access um creating little use cases and builders for like use cases and builders for like use cases and builders for like interacting with your data information interacting with your data information interacting with your data information um that's going to be a real unlock I um that's going to be a real unlock I um that's going to be a real unlock I think for for for think for for for think for for for >> talking about unlock so uh you've been >> talking about unlock so uh you've been >> talking about unlock so uh you've been before in Uber and there is a very before in Uber and there is a very before in Uber and there is a very well-known nowadays Michelangelo well-known nowadays Michelangelo well-known nowadays Michelangelo platform uh that was created in Uber and platform uh that was created in Uber and platform uh that was created in Uber and that is as far as understand unlock that is as far as understand unlock that is as far as understand unlock talking a lot of EA capabilities inside talking a lot of EA capabilities inside talking a lot of EA capabilities inside of that. Could you tell a bit more about of that. Could you tell a bit more about of that. Could you tell a bit more about >> Yeah, I think I think Uber was is a very >> Yeah, I think I think Uber was is a very >> Yeah, I think I think Uber was is a very advanced company when it comes to and advanced company when it comes to and advanced company when it comes to and I've I've definitely now I've seen I've I've definitely now I've seen I've I've definitely now I've seen different ranges of people on the on the different ranges of people on the on the different ranges of people on the on the stack. Um Uber built Michelangelo where stack. Um Uber built Michelangelo where stack. Um Uber built Michelangelo where effectively the idea was they had they effectively the idea was they had they effectively the idea was they had they had a marketplace real-time algorithm had a marketplace real-time algorithm had a marketplace real-time algorithm where people you know drivers and and where people you know drivers and and where people you know drivers and and and riders are being matched in real and riders are being matched in real and riders are being matched in real time. M time. M time. M >> it's incredibly hard to do that you know >> it's incredibly hard to do that you know >> it's incredibly hard to do that you know you're running models at scale low you're running models at scale low you're running models at scale low latency you know in real time with with latency you know in real time with with latency you know in real time with with constant pinging different times constant pinging different times constant pinging different times different time zones uh you have to keep different time zones uh you have to keep different time zones uh you have to keep the systems up at all times and you have the systems up at all times and you have the systems up at all times and you have to have uh redundancy you have to check to have uh redundancy you have to check to have uh redundancy you have to check around observability at all times this around observability at all times this around observability at all times this this machine learning infrastructure this machine learning infrastructure this machine learning infrastructure they had to build systems to make that they had to build systems to make that they had to build systems to make that that easy and possible

that easy and possible that easy and possible >> and that was the birth of Michelangelo u >> and that was the birth of Michelangelo u >> and that was the birth of Michelangelo u but since then what that enabled was but since then what that enabled was but since then what that enabled was that, you know, as as an organization, that, you know, as as an organization, that, you know, as as an organization, as the organization grew, you had all as the organization grew, you had all as the organization grew, you had all sorts of other teams saying, "Hey, we sorts of other teams saying, "Hey, we sorts of other teams saying, "Hey, we now need to do u marketplace matching of now need to do u marketplace matching of now need to do u marketplace matching of the drivers, but how do we do real-time the drivers, but how do we do real-time the drivers, but how do we do real-time pricing?" pricing?" pricing?" >> Mhm. >> Mhm. >> Mhm. >> You know, um how do we do real-time >> You know, um how do we do real-time >> You know, um how do we do real-time incentives that we give people? How do incentives that we give people? How do incentives that we give people? How do we, you know, run other models that we we, you know, run other models that we we, you know, run other models that we might want to have that do um you know, might want to have that do um you know, might want to have that do um you know, real-time prediction of of where a real-time prediction of of where a real-time prediction of of where a driver should go to predict demand and driver should go to predict demand and driver should go to predict demand and supply? So, the use cases started to supply? So, the use cases started to supply? So, the use cases started to grow. Um, and instead of all of these grow. Um, and instead of all of these grow. Um, and instead of all of these different internal model builders different internal model builders different internal model builders starting to have to compute features for starting to have to compute features for starting to have to compute features for their models again and again their models again and again their models again and again >> or tried to reuse, you know, all the >> or tried to reuse, you know, all the >> or tried to reuse, you know, all the different systems they had to build and different systems they had to build and different systems they had to build and pipelines, whether it was, you know, ML pipelines, whether it was, you know, ML pipelines, whether it was, you know, ML flow or airflow or different DAGs that flow or airflow or different DAGs that flow or airflow or different DAGs that they had to build out, why not reuse they had to build out, why not reuse they had to build out, why not reuse these components? And that was that was these components? And that was that was these components? And that was that was the benefit of Michelangelo. I see that the benefit of Michelangelo. I see that the benefit of Michelangelo. I see that you know the role of an MLOps person you you know the role of an MLOps person you you know the role of an MLOps person you know it's very hard to that was if you know it's very hard to that was if you know it's very hard to that was if you go on LinkedIn now it's actually very go on LinkedIn now it's actually very go on LinkedIn now it's actually very nowadays obviously it's it's a big role nowadays obviously it's it's a big role nowadays obviously it's it's a big role but I think a lot of large corporates but I think a lot of large corporates but I think a lot of large corporates don't sort of don't understand like that don't sort of don't understand like that don't sort of don't understand like that is an actual role it's not a data is an actual role it's not a data is an actual role it's not a data engineer it's an MLOps person um and engineer it's an MLOps person um and engineer it's an MLOps person um and that that goes into the language of that that goes into the language of that that goes into the language of features feature stores um you know uh features feature stores um you know uh features feature stores um you know uh model pipelines uh observability uh model pipelines uh observability uh model pipelines uh observability uh model evaluation metrics retraining model evaluation metrics retraining model evaluation metrics retraining systems, AutoML, it isn't just, you systems, AutoML, it isn't just, you systems, AutoML, it isn't just, you know, data cataloges, data pipelines,

know, data cataloges, data pipelines, know, data cataloges, data pipelines, um, uh, it's it's a different language. um, uh, it's it's a different language. um, uh, it's it's a different language. It's It's It's >> a whole system. >> a whole system. >> a whole system. >> It's a whole system. And I think that >> It's a whole system. And I think that >> It's a whole system. And I think that role, um, a lot of, um, companies I'm role, um, a lot of, um, companies I'm role, um, a lot of, um, companies I'm seeing sort of, okay, let's let's hire seeing sort of, okay, let's let's hire seeing sort of, okay, let's let's hire some scientists to to fix, you know, some scientists to to fix, you know, some scientists to to fix, you know, build this model for us. But the model build this model for us. But the model build this model for us. But the model builders can't build a model if there builders can't build a model if there builders can't build a model if there isn't the architecture to run models isn't the architecture to run models isn't the architecture to run models effectively or or run experiments effectively or or run experiments effectively or or run experiments quickly or test them both offline and quickly or test them both offline and quickly or test them both offline and online um to instrument your your test online um to instrument your your test online um to instrument your your test data sets like how do you have data sets like how do you have data sets like how do you have versioning on your test data sets versioning on your test data sets versioning on your test data sets >> all of these things like are um sort of >> all of these things like are um sort of >> all of these things like are um sort of linked to data but they're a different linked to data but they're a different linked to data but they're a different language and so I think that's really language and so I think that's really language and so I think that's really really important for a big business to really important for a big business to really important for a big business to >> really build the infrastructure so that >> really build the infrastructure so that >> really build the infrastructure so that anyone can go in and say hey I have a anyone can go in and say hey I have a anyone can go in and say hey I have a machine learning problem I'm sitting in machine learning problem I'm sitting in machine learning problem I'm sitting in the sales sales department or I'm the sales sales department or I'm the sales sales department or I'm sitting in the marketing department. I sitting in the marketing department. I sitting in the marketing department. I want to build a model. Where do I start? want to build a model. Where do I start? want to build a model. Where do I start? And and you build this internal And and you build this internal And and you build this internal experience. It's really easy for them experience. It's really easy for them experience. It's really easy for them and how to create that model. So it and how to create that model. So it and how to create that model. So it would be production level already. would be production level already. would be production level already. >> Exactly. Yeah. >> Exactly. Yeah. >> Exactly. Yeah. >> Talking about like product culture and >> Talking about like product culture and >> Talking about like product culture and the fact that you are being amplified a the fact that you are being amplified a the fact that you are being amplified a lot with AI agents. You still have a lot lot with AI agents. You still have a lot lot with AI agents. You still have a lot of like standard product metrics and of like standard product metrics and of like standard product metrics and standard product approaches. Yeah. uh it standard product approaches. Yeah. uh it standard product approaches. Yeah. uh it doesn't probably matter whether you are doesn't probably matter whether you are doesn't probably matter whether you are like uh helping developers to understand like uh helping developers to understand like uh helping developers to understand what they should be doing and what what they should be doing and what what they should be doing and what product we are creating or agents. So product we are creating or agents. So product we are creating or agents. So how do you approach that especially in how do you approach that especially in how do you approach that especially in the era of AI and the agents?

the era of AI and the agents? the era of AI and the agents? >> Yeah. So if you have traditional ML >> Yeah. So if you have traditional ML >> Yeah. So if you have traditional ML right like take let's take kind of you right like take let's take kind of you right like take let's take kind of you know fraud detection is a great example know fraud detection is a great example know fraud detection is a great example right you you have precision and you right you you have precision and you right you you have precision and you have recall right and then you have a have recall right and then you have a have recall right and then you have a curve curve curve >> rc curve where you can kind of tune >> rc curve where you can kind of tune >> rc curve where you can kind of tune those things in different ways. those things in different ways. those things in different ways. Precision is effectively in the case of Precision is effectively in the case of Precision is effectively in the case of fraud, you know, like how good how when fraud, you know, like how good how when fraud, you know, like how good how when you do flag something as a fraud, is it you do flag something as a fraud, is it you do flag something as a fraud, is it correct or not? correct or not? correct or not? >> Recall is of the things that you should >> Recall is of the things that you should >> Recall is of the things that you should be catching, what percentage did you be catching, what percentage did you be catching, what percentage did you catch? Um, you know, you could also have catch? Um, you know, you could also have catch? Um, you know, you could also have a system that flags everything and so a system that flags everything and so a system that flags everything and so your recall is 100%. You caught your recall is 100%. You caught your recall is 100%. You caught everything. Um, but then when you caught everything. Um, but then when you caught everything. Um, but then when you caught it, you're going to have a lot of false it, you're going to have a lot of false it, you're going to have a lot of false positives, right? you're going to be. So positives, right? you're going to be. So positives, right? you're going to be. So these these are just like basic these these are just like basic these these are just like basic frameworks that like I think people have frameworks that like I think people have frameworks that like I think people have to just understand like what are the to just understand like what are the to just understand like what are the core ML model metrics core ML model metrics core ML model metrics >> um that honestly like you can deduce by >> um that honestly like you can deduce by >> um that honestly like you can deduce by logic right like it's it's depending on logic right like it's it's depending on logic right like it's it's depending on the problem that that you have right the problem that that you have right the problem that that you have right predictions have the same thing predictions have the same thing predictions have the same thing classification has the same thing in the classification has the same thing in the classification has the same thing in the world of AI where you're building let's world of AI where you're building let's world of AI where you're building let's say a chat assistant uh or you're say a chat assistant uh or you're say a chat assistant uh or you're building you know let's say chat GPT building you know let's say chat GPT building you know let's say chat GPT >> Mhm. your evals are a little bit >> Mhm. your evals are a little bit >> Mhm. your evals are a little bit different, right? Because you have other different, right? Because you have other different, right? Because you have other types of things you have to to types of things you have to to types of things you have to to understand like, okay, was this understand like, okay, was this understand like, okay, was this conversation a good conversation?

conversation a good conversation? conversation a good conversation? >> How many multi-turn conversations did >> How many multi-turn conversations did >> How many multi-turn conversations did you have versus one-shot conversations you have versus one-shot conversations you have versus one-shot conversations where you just answered it one time? Um, where you just answered it one time? Um, where you just answered it one time? Um, how many time like what was the average how many time like what was the average how many time like what was the average length of a conversation? um in terms of length of a conversation? um in terms of length of a conversation? um in terms of uh you know e we had this phrase of eval uh you know e we had this phrase of eval uh you know e we had this phrase of eval right and in LLM sort of like how do you right and in LLM sort of like how do you right and in LLM sort of like how do you measure that that response was a good measure that that response was a good measure that that response was a good response versus a bad response cuz it's response versus a bad response cuz it's response versus a bad response cuz it's a subjective thing right for some people a subjective thing right for some people a subjective thing right for some people that would have been a good response that would have been a good response that would have been a good response some people that wouldn't have been a some people that wouldn't have been a some people that wouldn't have been a good response so so there's this element good response so so there's this element good response so so there's this element of how do you kind of try to take a very of how do you kind of try to take a very of how do you kind of try to take a very subjective thing which is a conversation subjective thing which is a conversation subjective thing which is a conversation and try to put quantitative metrics on and try to put quantitative metrics on and try to put quantitative metrics on it yeah do you see the same problem it yeah do you see the same problem it yeah do you see the same problem actually with just uh training of large actually with just uh training of large actually with just uh training of large language models with RHLF and so on. language models with RHLF and so on. language models with RHLF and so on. >> Yeah. So, so as in RHLF used as an eval >> Yeah. So, so as in RHLF used as an eval >> Yeah. So, so as in RHLF used as an eval metric and mainly I think talk about the metric and mainly I think talk about the metric and mainly I think talk about the cases like for example in case of OpenAI cases like for example in case of OpenAI cases like for example in case of OpenAI or Antropic they've been tuning their or Antropic they've been tuning their or Antropic they've been tuning their models with GPT 5 5.1 5.2 trying to be models with GPT 5 5.1 5.2 trying to be models with GPT 5 5.1 5.2 trying to be less friendly more friendly and so on.

less friendly more friendly and so on. less friendly more friendly and so on. It's also kind of a product work but for It's also kind of a product work but for It's also kind of a product work but for Language model we're trying to Language model we're trying to Language model we're trying to understand which personality should it understand which personality should it understand which personality should it have. I mean look a lot of those model have. I mean look a lot of those model have. I mean look a lot of those model the way that people are doing this is the way that people are doing this is the way that people are doing this is like like I said it's a subjective thing like like I said it's a subjective thing like like I said it's a subjective thing right so right so right so >> people forget that there's there's >> people forget that there's there's >> people forget that there's there's obviously people who are figuring out obviously people who are figuring out obviously people who are figuring out the model the model the model >> does the model work well and there's a >> does the model work well and there's a >> does the model work well and there's a lot of benchmarks that are built for AI lot of benchmarks that are built for AI lot of benchmarks that are built for AI agents right that that like open source agents right that that like open source agents right that that like open source benchmarks where they're how good are benchmarks where they're how good are benchmarks where they're how good are they on these specific hard-coded tasks they on these specific hard-coded tasks they on these specific hard-coded tasks right right right >> so you can try to close the domain and >> so you can try to close the domain and >> so you can try to close the domain and say okay uh we think that um most of our say okay uh we think that um most of our say okay uh we think that um most of our feedback that we got was that the feedback that we got was that the feedback that we got was that the conversation wasn't very friendly. So if conversation wasn't very friendly. So if conversation wasn't very friendly. So if we make the conversations more in a we make the conversations more in a we make the conversations more in a friendlier tone uh people will be be friendlier tone uh people will be be friendlier tone uh people will be be happier. The challenge is that you have happier. The challenge is that you have happier. The challenge is that you have these internal model metrics but then these internal model metrics but then these internal model metrics but then you have your online actual outcome is you have your online actual outcome is you have your online actual outcome is that is chat GPT growing. that is chat GPT growing. that is chat GPT growing. >> Yeah. >> Yeah. >> Yeah. >> That's what management cares about, >> That's what management cares about, >> That's what management cares about, right? Is is it and and and that may not right? Is is it and and and that may not right? Is is it and and and that may not be at all because of the fact that the be at all because of the fact that the be at all because of the fact that the conversations are friendlier.

conversations are friendlier. conversations are friendlier. >> Yeah. It may be that that actually that >> Yeah. It may be that that actually that >> Yeah. It may be that that actually that the cohort that you're going for that the cohort that you're going for that the cohort that you're going for that the next growth segment if you're not the next growth segment if you're not the next growth segment if you're not against the next billion users maybe against the next billion users maybe against the next billion users maybe they don't want friendly conversations. they don't want friendly conversations. they don't want friendly conversations. Maybe they want more targeted um um Maybe they want more targeted um um Maybe they want more targeted um um rationalized conversations that achieve rationalized conversations that achieve rationalized conversations that achieve certain tasks. uh or maybe it's actually certain tasks. uh or maybe it's actually certain tasks. uh or maybe it's actually because the user experience is not very because the user experience is not very because the user experience is not very good um you know for chat or maybe good um you know for chat or maybe good um you know for chat or maybe actually the users that you're targeting actually the users that you're targeting actually the users that you're targeting they need to access spreadsheets very they need to access spreadsheets very they need to access spreadsheets very well and you know the tool calling for well and you know the tool calling for well and you know the tool calling for that isn't very good on on chatbt but that isn't very good on on chatbt but that isn't very good on on chatbt but it's very good for claude so the the the it's very good for claude so the the the it's very good for claude so the the the the the the the the the complexity and the vastness of the space complexity and the vastness of the space complexity and the vastness of the space of metrics that you can have is so so of metrics that you can have is so so of metrics that you can have is so so big I think most ML AI product building big I think most ML AI product building big I think most ML AI product building has to split find the ML metrics the one has to split find the ML metrics the one has to split find the ML metrics the one or two core ML metrics um that or two core ML metrics um that or two core ML metrics um that >> really drive like okay this is a good ML >> really drive like okay this is a good ML >> really drive like okay this is a good ML product product product >> but then the sort of online metrics you >> but then the sort of online metrics you >> but then the sort of online metrics you know which really know which really know which really >> align to them >> align to them >> align to them >> align and and and ideally and this is >> align and and and ideally and this is >> align and and and ideally and this is where people get it very wrong is like where people get it very wrong is like where people get it very wrong is like connecting that together so if I really connecting that together so if I really connecting that together so if I really make make make if if I was to make the um the responses if if I was to make the um the responses if if I was to make the um the responses perfect for every single conversation it perfect for every single conversation it perfect for every single conversation it answered every single question perfectly answered every single question perfectly answered every single question perfectly would that lead to growth of chat GBT would that lead to growth of chat GBT would that lead to growth of chat GBT But uh what do you see else like in a But uh what do you see else like in a But uh what do you see else like in a such a governed company? What are the such a governed company? What are the such a governed company? What are the main methods and things that you should main methods and things that you should main methods and things that you should be thinking about governing the AI be thinking about governing the AI be thinking about governing the AI inside the organization? Yeah. So AI inside the organization? Yeah. So AI inside the organization? Yeah. So AI inside the organization people have to

inside the organization people have to inside the organization people have to think about access controls. Um you need think about access controls. Um you need think about access controls. Um you need to have a data catalog, right? What data to have a data catalog, right? What data to have a data catalog, right? What data is being spawned? What data already is being spawned? What data already is being spawned? What data already exists? What jobs are running in the exists? What jobs are running in the exists? What jobs are running in the background? What sub metadata sets are background? What sub metadata sets are background? What sub metadata sets are being created? You need to have a good being created? You need to have a good being created? You need to have a good handle of that. You need to be able to handle of that. You need to be able to handle of that. You need to be able to log it and understand what is actually log it and understand what is actually log it and understand what is actually in that data. in that data. in that data. >> Once you've logged that data, you have >> Once you've logged that data, you have >> Once you've logged that data, you have to understand what are the potential use to understand what are the potential use to understand what are the potential use cases that that data could be used for. cases that that data could be used for. cases that that data could be used for. >> Uh I mentioned before, you know, if you >> Uh I mentioned before, you know, if you >> Uh I mentioned before, you know, if you had data on previous loan applications had data on previous loan applications had data on previous loan applications and if they were reject accepted or and if they were reject accepted or and if they were reject accepted or rejected. Um and you also have features rejected. Um and you also have features rejected. Um and you also have features that tell you what race that person is that tell you what race that person is that tell you what race that person is is in or what country they're from. you is in or what country they're from. you is in or what country they're from. you can maybe you're not doing it now but can maybe you're not doing it now but can maybe you're not doing it now but you can possibly understand that that you can possibly understand that that you can possibly understand that that could be used you know as you're could be used you know as you're could be used you know as you're building a model that that learns from building a model that that learns from building a model that that learns from those features in an incorrect way. those features in an incorrect way. those features in an incorrect way. >> Yeah. >> Yeah. >> Yeah. >> So >> So >> So putting safeguards typically what people putting safeguards typically what people putting safeguards typically what people don't is put safeguards on like testing don't is put safeguards on like testing don't is put safeguards on like testing of the model. So you have kind of red of the model. So you have kind of red of the model. So you have kind of red teaming where the models basically are teaming where the models basically are teaming where the models basically are triple checking are they making triple checking are they making triple checking are they making decisions and learning from those decisions and learning from those decisions and learning from those features too much, right?

features too much, right? features too much, right? >> Um or or if you only had those isolated >> Um or or if you only had those isolated >> Um or or if you only had those isolated those features and you just had that as those features and you just had that as those features and you just had that as the feature that that the model was the feature that that the model was the feature that that the model was learning from, would it take what would learning from, would it take what would learning from, would it take what would be the decision boundaries that were be the decision boundaries that were be the decision boundaries that were created from that? created from that? created from that? >> Mhm. Um >> Mhm. Um >> Mhm. Um the the challenge with this is that the the challenge with this is that the the challenge with this is that actually factually it is correct that actually factually it is correct that actually factually it is correct that let's say one race versus another race let's say one race versus another race let's say one race versus another race has been accepted from a loan has been accepted from a loan has been accepted from a loan application but then there was probably application but then there was probably application but then there was probably something wrong in your own system. something wrong in your own system. something wrong in your own system. >> Yeah. Like there is something wrong in >> Yeah. Like there is something wrong in >> Yeah. Like there is something wrong in society basically society basically society basically >> in society. In society but also maybe >> in society. In society but also maybe >> in society. In society but also maybe the way that you had been doing your own the way that you had been doing your own the way that you had been doing your own loan applications as a as a business. So loan applications as a as a business. So loan applications as a as a business. So it's not that you are that the system is it's not that you are that the system is it's not that you are that the system is wrong. it's sort of starting to unlock wrong. it's sort of starting to unlock wrong. it's sort of starting to unlock um changes in the overall actually um changes in the overall actually um changes in the overall actually online system itself. Um online system itself. Um online system itself. Um >> shows basically you something about your >> shows basically you something about your >> shows basically you something about your system. So exactly system. So exactly system. So exactly >> yeah I've actually heard a lot of uh >> yeah I've actually heard a lot of uh >> yeah I've actually heard a lot of uh feedback about some trained systems in feedback about some trained systems in feedback about some trained systems in the big corporations or even trained the big corporations or even trained the big corporations or even trained system in some domains where people are system in some domains where people are system in some domains where people are saying that yeah folks it's like it's saying that yeah folks it's like it's saying that yeah folks it's like it's not correct it should not be working not correct it should not be working not correct it should not be working that way but that way but that way but >> in in the end you find out that it's >> in in the end you find out that it's >> in in the end you find out that it's actually working the way the model is actually working the way the model is actually working the way the model is predicting the problem is not in the predicting the problem is not in the predicting the problem is not in the model the problem in the system model the problem in the system model the problem in the system >> system yeah and and then you can say >> system yeah and and then you can say >> system yeah and and then you can say well look let's even remove that feature well look let's even remove that feature well look let's even remove that feature it's not going to learn from that it's not going to learn from that it's not going to learn from that feature feature feature >> but then then you could then still train >> but then then you could then still train >> but then then you could then still train the system, it would work probably the system, it would work probably the system, it would work probably slightly worse for your task. And so slightly worse for your task. And so slightly worse for your task. And so you've taken a business decision to not you've taken a business decision to not you've taken a business decision to not learn from that feature.

learn from that feature. learn from that feature. >> Mhm. >> Mhm. >> Mhm. >> You've effectively got worse at >> You've effectively got worse at >> You've effectively got worse at automated decisioning. Yeah. For loans automated decisioning. Yeah. For loans automated decisioning. Yeah. For loans because you've decided to ignore that because you've decided to ignore that because you've decided to ignore that feature. But then you can if you did a feature. But then you can if you did a feature. But then you can if you did a correlation correlation correlation and actually like actually it's still and actually like actually it's still and actually like actually it's still like even that system that you trained like even that system that you trained like even that system that you trained that didn't use that feature still that didn't use that feature still that didn't use that feature still correlated on if you had isolated that correlated on if you had isolated that correlated on if you had isolated that feature or not. So I think I think it's feature or not. So I think I think it's feature or not. So I think I think it's basically what I'm what I'm trying to basically what I'm what I'm trying to basically what I'm what I'm trying to say is that um by looking at governance say is that um by looking at governance say is that um by looking at governance and looking at what is actually driving and looking at what is actually driving and looking at what is actually driving a decision for an ML model, a decision for an ML model, a decision for an ML model, >> you can really improve that whole not >> you can really improve that whole not >> you can really improve that whole not just the you you will obviously not not just the you you will obviously not not just the you you will obviously not not just automate the system itself, but just automate the system itself, but just automate the system itself, but you'll start to actually rearchitect the you'll start to actually rearchitect the you'll start to actually rearchitect the way that you're doing loans or the way way that you're doing loans or the way way that you're doing loans or the way that you're doing fraud detection, for that you're doing fraud detection, for that you're doing fraud detection, for example. Um yeah, we we we've I can give example. Um yeah, we we we've I can give example. Um yeah, we we we've I can give you an example. You know, we've you an example. You know, we've you an example. You know, we've chargeback fraud detection. You might chargeback fraud detection. You might chargeback fraud detection. You might have, for example, if you were to dump have, for example, if you were to dump have, for example, if you were to dump all of the features that you had all of the features that you had all of the features that you had available to do this chargeback fraud available to do this chargeback fraud available to do this chargeback fraud dete detection, dete detection, dete detection, >> you might find that actually some >> you might find that actually some >> you might find that actually some features are really really predictive of features are really really predictive of features are really really predictive of that, but they make no explainable sense that, but they make no explainable sense that, but they make no explainable sense whatsoever. They just work. And so whatsoever. They just work. And so whatsoever. They just work. And so there's this decision for companies to there's this decision for companies to there's this decision for companies to be like, well, do I just build the most be like, well, do I just build the most be like, well, do I just build the most high highly precise and high high recall high highly precise and high high recall high highly precise and high high recall system or do I actually put sacrifice system or do I actually put sacrifice system or do I actually put sacrifice some performance on this wider test set some performance on this wider test set some performance on this wider test set to at least make the system more to at least make the system more to at least make the system more explainable?

explainable? explainable? >> Yeah. To inter to make it a more >> Yeah. To inter to make it a more >> Yeah. To inter to make it a more interpretable. Yeah, interpretable. Yeah, interpretable. Yeah, >> that is actually a very important >> that is actually a very important >> that is actually a very important question because interpretability also question because interpretability also question because interpretability also helps you to improve the system. helps you to improve the system. helps you to improve the system. >> Yeah. Yeah. Exactly. >> Yeah. Yeah. Exactly. >> Yeah. Yeah. Exactly. >> Talking about the investments. Uh so you >> Talking about the investments. Uh so you >> Talking about the investments. Uh so you were saying that uh if you're like a big were saying that uh if you're like a big were saying that uh if you're like a big company, middle company and you have a company, middle company and you have a company, middle company and you have a flat budget right now for EI what where flat budget right now for EI what where flat budget right now for EI what where would you invest money? So Uber for would you invest money? So Uber for would you invest money? So Uber for example they've invested in Michelangelo example they've invested in Michelangelo example they've invested in Michelangelo like 10 years ago at least and nowadays like 10 years ago at least and nowadays like 10 years ago at least and nowadays they are able to deliver more and more they are able to deliver more and more they are able to deliver more and more machine learning models uh on any level machine learning models uh on any level machine learning models uh on any level basically both the developers and basically both the developers and basically both the developers and product managers whoever. product managers whoever. product managers whoever. >> Yeah. So what would be your advice to >> Yeah. So what would be your advice to >> Yeah. So what would be your advice to the CTO? My advice to CTO would be like the CTO? My advice to CTO would be like the CTO? My advice to CTO would be like you cannot predict all the use cases for you cannot predict all the use cases for you cannot predict all the use cases for machine learning in an organization. You machine learning in an organization. You machine learning in an organization. You got thousands of people who are who have got thousands of people who are who have got thousands of people who are who have tasks to automate or or jobs to be done tasks to automate or or jobs to be done tasks to automate or or jobs to be done that they take on in on their that they take on in on their that they take on in on their day-to-day. So instead of trying to map day-to-day. So instead of trying to map day-to-day. So instead of trying to map out all those AI use cases, which is out all those AI use cases, which is out all those AI use cases, which is what I see a lot of people do, it's like what I see a lot of people do, it's like what I see a lot of people do, it's like where where can we get value from AI? where where can we get value from AI? where where can we get value from AI? Let's like write it in a document Let's like write it in a document Let's like write it in a document >> and that will change in the next >> and that will change in the next >> and that will change in the next quarter. quarter. quarter. >> That will change in the next quarter.

>> That will change in the next quarter. >> That will change in the next quarter. Build a sandbox environment. Invest for Build a sandbox environment. Invest for Build a sandbox environment. Invest for two quarters in a sandbox environment two quarters in a sandbox environment two quarters in a sandbox environment that enables anyone to build a machine that enables anyone to build a machine that enables anyone to build a machine learning model. learning model. learning model. >> Open up your data so that you have >> Open up your data so that you have >> Open up your data so that you have features that people can access. uh features that people can access. uh features that people can access. uh whether it's of data streams that are whether it's of data streams that are whether it's of data streams that are coming through of what the business is coming through of what the business is coming through of what the business is doing whether it's sales calls whether doing whether it's sales calls whether doing whether it's sales calls whether it's in the case of payments payments it's in the case of payments payments it's in the case of payments payments going through transactions going through transactions going through transactions you know information that's anonymized you know information that's anonymized you know information that's anonymized about shoppers who are doing those about shoppers who are doing those about shoppers who are doing those transa open that up to people transa open that up to people transa open that up to people >> mh make it easy for them to train models >> mh make it easy for them to train models >> mh make it easy for them to train models using sort of template simple xg boost using sort of template simple xg boost using sort of template simple xg boost logistic regression so simple models logistic regression so simple models logistic regression so simple models and just see what use cases come out and just see what use cases come out and just see what use cases come out obviously open up LLMs as well right obviously open up LLMs as well right obviously open up LLMs as well right open access and see what cases people want to do and and see what cases people want to do and create an open channel where those use create an open channel where those use create an open channel where those use cases and the issues that come up are cases and the issues that come up are cases and the issues that come up are fed through to you so you can patch them fed through to you so you can patch them fed through to you so you can patch them and fix them and you will learn so much and fix them and you will learn so much and fix them and you will learn so much about what people want to use AI for in about what people want to use AI for in about what people want to use AI for in your organization. That was a great nice your organization. That was a great nice your organization. That was a great nice discussion. Uh yeah, happy to talk to discussion. Uh yeah, happy to talk to discussion. Uh yeah, happy to talk to you. Uh thank you. So maybe you have I you. Uh thank you. So maybe you have I you. Uh thank you. So maybe you have I don't know any other advices to the don't know any other advices to the don't know any other advices to the folks who are listening to that. Yeah, I folks who are listening to that. Yeah, I folks who are listening to that. Yeah, I would say just yeah it's it's the age of would say just yeah it's it's the age of would say just yeah it's it's the age of building. It's the age of creation. Um I building. It's the age of creation. Um I building. It's the age of creation. Um I think that's a really big opportunity think that's a really big opportunity think that's a really big opportunity for not just people in the workplace but for not just people in the workplace but for not just people in the workplace but also people humanity in general.

also people humanity in general. also people humanity in general. >> Uh and it's this like great opportunity >> Uh and it's this like great opportunity >> Uh and it's this like great opportunity to to recreate that that magic of of of to to recreate that that magic of of of to to recreate that that magic of of of creativity. So I think creativity. So I think creativity. So I think >> even if you are in a job or you are an >> even if you are in a job or you are an >> even if you are in a job or you are an entrepreneur, it doesn't really matter. entrepreneur, it doesn't really matter. entrepreneur, it doesn't really matter. try to bring that and weave that into try to bring that and weave that into try to bring that and weave that into your day in the way that you you you you your day in the way that you you you you your day in the way that you you you you work. So, it's basically be creative, work. So, it's basically be creative, work. So, it's basically be creative, have an agency, and be hands-on. Yeah. have an agency, and be hands-on. Yeah. have an agency, and be hands-on. Yeah. Yeah. Okay, great. Thank you. Thanks for Yeah. Okay, great. Thank you. Thanks for Yeah. Okay, great. Thank you. Thanks for joining. And if you want to know more joining. And if you want to know more joining. And if you want to know more about how AI may fit into your about how AI may fit into your about how AI may fit into your organization, check the link below.