How do you diffuse AI into the real world? — Varun Shenoy, Long Lake

>> Hi everyone. I'm Varun. I'm one of the >> Hi everyone. I'm Varun. I'm one of the co-founders at Long Lake and I'm excited co-founders at Long Lake and I'm excited co-founders at Long Lake and I'm excited to share a little bit about what we've to share a little bit about what we've to share a little bit about what we've been up to for the last 2 years. been up to for the last 2 years. been up to for the last 2 years. It all comes back to a question all of It all comes back to a question all of It all comes back to a question all of us have asked time and time again. us have asked time and time again. us have asked time and time again. The models are getting better, The models are getting better, The models are getting better, but the real question is how do you but the real question is how do you but the real question is how do you actually deploy the AI into the real actually deploy the AI into the real actually deploy the AI into the real world? world? world? How do you get the models to complete How do you get the models to complete How do you get the models to complete economically relevant tasks? Let me start by saying everyone has seen Let me start by saying everyone has seen the demo. Think of the agent the demo. Think of the agent the demo. Think of the agent automatically booking a flight, the automatically booking a flight, the automatically booking a flight, the agent automatically completing a ticket agent automatically completing a ticket agent automatically completing a ticket in some kind of customer service portal. in some kind of customer service portal. in some kind of customer service portal. Think of an agent completing a block of Think of an agent completing a block of Think of an agent completing a block of code ready to commit and go. code ready to commit and go. code ready to commit and go. The reality is we've all seen this and The reality is we've all seen this and The reality is we've all seen this and it feels like magic. 2 years ago any of it feels like magic. 2 years ago any of it feels like magic. 2 years ago any of this would have been complete science this would have been complete science this would have been complete science fiction. The capabilities are real. fiction. The capabilities are real. fiction. The capabilities are real. Now, walk with me into a 200-person Now, walk with me into a 200-person Now, walk with me into a 200-person property management firm.

property management firm. property management firm. Real people, real properties, Real people, real properties, Real people, real properties, real dollars, real customers all across real dollars, real customers all across real dollars, real customers all across the US. the US. the US. You would expect AI to show up by now, You would expect AI to show up by now, You would expect AI to show up by now, but the reality is nothing has changed but the reality is nothing has changed but the reality is nothing has changed at all. Here's the thing. Here's the thing. This is totally normal and maybe in fact This is totally normal and maybe in fact This is totally normal and maybe in fact I'd argue this is what we should expect. I'd argue this is what we should expect. I'd argue this is what we should expect. This is true for every general-purpose This is true for every general-purpose This is true for every general-purpose technology. You know, take electricity technology. You know, take electricity technology. You know, take electricity for example. for example. for example. Electricity was invented in the 1880s Electricity was invented in the 1880s Electricity was invented in the 1880s and it was first demoed at Edison's and it was first demoed at Edison's and it was first demoed at Edison's Pearl Street Station Dynamo Room over in Pearl Street Station Dynamo Room over in Pearl Street Station Dynamo Room over in Manhattan. Manhattan. Manhattan. This was the magic demo of its time. The reality is it took a long time for The reality is it took a long time for electricity to be fully adopted. electricity to be fully adopted. electricity to be fully adopted. Consider a Ford factory. Consider a Ford factory. Consider a Ford factory. It's not enough to just have It's not enough to just have It's not enough to just have electricity. You have to rip out the electricity. You have to rip out the electricity. You have to rip out the existing motors and equipment. You have existing motors and equipment. You have existing motors and equipment. You have to bring in the new equipment. You have to bring in the new equipment. You have to bring in the new equipment. You have to go and train everybody to use that to go and train everybody to use that to go and train everybody to use that very same equipment.

very same equipment. very same equipment. Here's a picture of a Ford electrified Here's a picture of a Ford electrified Here's a picture of a Ford electrified moving assembly in 1924. moving assembly in 1924. moving assembly in 1924. These things take time. These things take time. These things take time. Diffusion of any technology takes a Diffusion of any technology takes a Diffusion of any technology takes a generation. And since everyone here in generation. And since everyone here in generation. And since everyone here in this room today is talking about AI, I this room today is talking about AI, I this room today is talking about AI, I would argue would argue would argue AI diffusion is perhaps the single most AI diffusion is perhaps the single most AI diffusion is perhaps the single most important problem for the next 20 years. important problem for the next 20 years. important problem for the next 20 years. The models are going to keep getting The models are going to keep getting The models are going to keep getting better. The big question is how do we better. The big question is how do we better. The big question is how do we actually get these models to be in the actually get these models to be in the actually get these models to be in the real world, complete real tasks, uh and real world, complete real tasks, uh and real world, complete real tasks, uh and make people more efficient, happier, and make people more efficient, happier, and make people more efficient, happier, and provide better service. provide better service. provide better service. So taking a quick step step back, who So taking a quick step step back, who So taking a quick step step back, who are we? Uh we are Long Lake. Over the are we? Uh we are Long Lake. Over the are we? Uh we are Long Lake. Over the last 2 years, we've raised over $3 last 2 years, we've raised over $3 last 2 years, we've raised over $3 billion from Elad Gil, General Catalyst, billion from Elad Gil, General Catalyst, billion from Elad Gil, General Catalyst, and AlphaWave since our founding. and AlphaWave since our founding. and AlphaWave since our founding. Here's the strange part. We we don't Here's the strange part. We we don't Here's the strange part. We we don't sell software. We actually go out and sell software. We actually go out and sell software. We actually go out and acquire and partner with real services acquire and partner with real services acquire and partner with real services businesses in the world. Uh we've businesses in the world. Uh we've businesses in the world. Uh we've acquired 35 businesses across HOA and acquired 35 businesses across HOA and acquired 35 businesses across HOA and property management, architecture, HR property management, architecture, HR property management, architecture, HR services, and a lot more.

services, and a lot more. services, and a lot more. To give you a little bit more flavor, we To give you a little bit more flavor, we To give you a little bit more flavor, we have roughly a 40% team right now split have roughly a 40% team right now split have roughly a 40% team right now split between technology, finance, and between technology, finance, and between technology, finance, and operations. More than half our team is operations. More than half our team is operations. More than half our team is part of the technology team focused on part of the technology team focused on part of the technology team focused on uh building products, data, and uh building products, data, and uh building products, data, and deploying the core products into the deploying the core products into the deploying the core products into the field. Uh we're in a collected group of field. Uh we're in a collected group of field. Uh we're in a collected group of folks, a bunch of ex-founders who've folks, a bunch of ex-founders who've folks, a bunch of ex-founders who've worked in the services before, worked in the services before, worked in the services before, ex-military, folks from Palantir, Ramp, ex-military, folks from Palantir, Ramp, ex-military, folks from Palantir, Ramp, Glean, uh and from the finance side, Glean, uh and from the finance side, Glean, uh and from the finance side, Blackstone, H.I.G., et cetera. We are we Blackstone, H.I.G., et cetera. We are we Blackstone, H.I.G., et cetera. We are we are not selling them to these companies are not selling them to these companies are not selling them to these companies above from the outside. We're actually above from the outside. We're actually above from the outside. We're actually deploying into these companies and deploying into these companies and deploying into these companies and figuring out how to get the technology figuring out how to get the technology figuring out how to get the technology to work. to work. to work. And just to show you the scale we're And just to show you the scale we're And just to show you the scale we're playing at, we announced recently our playing at, we announced recently our playing at, we announced recently our $6.3 billion take private of American $6.3 billion take private of American $6.3 billion take private of American Express Global Business Travel, the Express Global Business Travel, the Express Global Business Travel, the world's largest corporate travel world's largest corporate travel world's largest corporate travel platform. platform. platform. We own these businesses. We own these businesses. We own these businesses. So, when the AI doesn't work, it's not So, when the AI doesn't work, it's not So, when the AI doesn't work, it's not their problem. We're not the vendor. their problem. We're not the vendor. their problem. We're not the vendor. It's our problem.

It's our problem. It's our problem. Concretely, again, we are not the Concretely, again, we are not the Concretely, again, we are not the vendor. We are the operator owners. And vendor. We are the operator owners. And vendor. We are the operator owners. And we work very closely with our teams we work very closely with our teams we work very closely with our teams within the businesses to drive real within the businesses to drive real within the businesses to drive real outcomes. outcomes. outcomes. Now, I want to step back and get to the Now, I want to step back and get to the Now, I want to step back and get to the concrete about the how. What are the concrete about the how. What are the concrete about the how. What are the lessons we've learned over the last 2 lessons we've learned over the last 2 lessons we've learned over the last 2 and 1/2 years? And what we've learned and 1/2 years? And what we've learned and 1/2 years? And what we've learned from deploying AI into companies we've from deploying AI into companies we've from deploying AI into companies we've owned. Three quick lessons. One, how we move Three quick lessons. One, how we move agents from co-pilots to co-workers. agents from co-pilots to co-workers. agents from co-pilots to co-workers. Two, how we leverage real-world data Two, how we leverage real-world data Two, how we leverage real-world data within these businesses. within these businesses. within these businesses. Remember, we're seeing all of the work Remember, we're seeing all of the work Remember, we're seeing all of the work that's being done in these real services that's being done in these real services that's being done in these real services businesses. There's a lot of interesting businesses. There's a lot of interesting businesses. There's a lot of interesting problems and solutions embedded within problems and solutions embedded within problems and solutions embedded within that. that. that. And then finally, perhaps the most And then finally, perhaps the most And then finally, perhaps the most interesting and exciting is how do you interesting and exciting is how do you interesting and exciting is how do you actually get all of this technology to actually get all of this technology to actually get all of this technology to compound over time by learning loops in compound over time by learning loops in compound over time by learning loops in the enterprise. We'll get to that at the the enterprise. We'll get to that at the the enterprise. We'll get to that at the end over here. end over here. end over here. So, starting off from co-pilots to So, starting off from co-pilots to So, starting off from co-pilots to co-workers. co-workers. co-workers. There's a spectrum of how much autonomy There's a spectrum of how much autonomy There's a spectrum of how much autonomy you can give an agent.

you can give an agent. you can give an agent. On the left here, you see a co-pilot. On the left here, you see a co-pilot. On the left here, you see a co-pilot. This is, you know, your simple rag This is, you know, your simple rag This is, you know, your simple rag chatbot from 2 years ago. It's very chatbot from 2 years ago. It's very chatbot from 2 years ago. It's very quick. You can ask a question. Maybe quick. You can ask a question. Maybe quick. You can ask a question. Maybe it's integrated with some systems. It it's integrated with some systems. It it's integrated with some systems. It can give you information back very, very can give you information back very, very can give you information back very, very quickly. The second step is a synchronous agent. The second step is a synchronous agent. Consider something like Claude code, Consider something like Claude code, Consider something like Claude code, Codex, Claude co-work. It's real-time. Codex, Claude co-work. It's real-time. Codex, Claude co-work. It's real-time. There's this two-way interaction. It's a There's this two-way interaction. It's a There's this two-way interaction. It's a bit more sophisticated than a co-pilot. bit more sophisticated than a co-pilot. bit more sophisticated than a co-pilot. You can go let it run off for 1 to 5 You can go let it run off for 1 to 5 You can go let it run off for 1 to 5 minutes. Uh it'll call tools, maybe use minutes. Uh it'll call tools, maybe use minutes. Uh it'll call tools, maybe use its skills. Uh it's still synchronous. its skills. Uh it's still synchronous. its skills. Uh it's still synchronous. You still need to step in and ask a You still need to step in and ask a You still need to step in and ask a query. So, the next obvious rung of the query. So, the next obvious rung of the query. So, the next obvious rung of the ladder is the asynchronous agent. ladder is the asynchronous agent. ladder is the asynchronous agent. You can come in here, still ask a query. You can come in here, still ask a query. You can come in here, still ask a query. The agent will go off into the The agent will go off into the The agent will go off into the background, do some work, and then come background, do some work, and then come background, do some work, and then come back. Uh and what's really interesting back. Uh and what's really interesting back. Uh and what's really interesting about asynchronous agents is that the about asynchronous agents is that the about asynchronous agents is that the user does not have to be the one that user does not have to be the one that user does not have to be the one that triggers them. triggers them. triggers them. You can have external triggers as well. You can have external triggers as well. You can have external triggers as well. Maybe someone completes a certain task Maybe someone completes a certain task Maybe someone completes a certain task and there is an async job queue uh that and there is an async job queue uh that and there is an async job queue uh that allows the async agent to pull off from allows the async agent to pull off from allows the async agent to pull off from and proactively offer advice to the end and proactively offer advice to the end and proactively offer advice to the end user.

user. user. Then, I'd argue the next step is a Then, I'd argue the next step is a Then, I'd argue the next step is a long-running agent. long-running agent. long-running agent. How do you get these agents to work for How do you get these agents to work for How do you get these agents to work for hours, days, weeks, months, etc.? I hours, days, weeks, months, etc.? I hours, days, weeks, months, etc.? I think this is currently a very core think this is currently a very core think this is currently a very core problem that a lot of the labs are problem that a lot of the labs are problem that a lot of the labs are focused on, as are we. And then finally, at the end, the holy And then finally, at the end, the holy grail, an AI co-worker. grail, an AI co-worker. grail, an AI co-worker. This is where most people start off. This is where most people start off. This is where most people start off. You want a proactive partner that gets You want a proactive partner that gets You want a proactive partner that gets work done just alongside you. work done just alongside you. work done just alongside you. This is what everyone wants to sell you, This is what everyone wants to sell you, This is what everyone wants to sell you, but what we've learned from owning the but what we've learned from owning the but what we've learned from owning the outcomes in this business is outcomes in this business is outcomes in this business is you have to earn the right to do more. you have to earn the right to do more. you have to earn the right to do more. It's it's not enough to jump to the It's it's not enough to jump to the It's it's not enough to jump to the co-worker immediately, co-worker immediately, co-worker immediately, right? For for a bunch of reasons. One, right? For for a bunch of reasons. One, right? For for a bunch of reasons. One, for certain tasks, the models might not for certain tasks, the models might not for certain tasks, the models might not quite be there yet. And two, you quite be there yet. And two, you quite be there yet. And two, you actually have to work with these actually have to work with these actually have to work with these companies in the field, interact and companies in the field, interact and companies in the field, interact and iterate very, very closely, so that they iterate very, very closely, so that they iterate very, very closely, so that they understand that this is the beginning of understand that this is the beginning of understand that this is the beginning of AI, and you can work up the rungs over AI, and you can work up the rungs over AI, and you can work up the rungs over time.

I think a really unique lens to look at I think a really unique lens to look at this problem through is the that of the this problem through is the that of the this problem through is the that of the jagged frontier. We all know that agents jagged frontier. We all know that agents jagged frontier. We all know that agents are incredibly good at writing code. So, are incredibly good at writing code. So, are incredibly good at writing code. So, what does the, for example, synchronous what does the, for example, synchronous what does the, for example, synchronous agent for code generation look like? agent for code generation look like? agent for code generation look like? This is super simple. This is just your This is super simple. This is just your This is super simple. This is just your coding agent, maybe it's Codex, Cloud coding agent, maybe it's Codex, Cloud coding agent, maybe it's Codex, Cloud Code, just running on your desktop. It Code, just running on your desktop. It Code, just running on your desktop. It has access to a file system. You has access to a file system. You has access to a file system. You collaborate within real time. You get collaborate within real time. You get collaborate within real time. You get instant feedback and you iterate. instant feedback and you iterate. instant feedback and you iterate. The next step is, you know, if you look The next step is, you know, if you look The next step is, you know, if you look at code code generation, what is the at code code generation, what is the at code code generation, what is the async agent? This is also fairly async agent? This is also fairly async agent? This is also fairly straightforward and largely solved. You straightforward and largely solved. You straightforward and largely solved. You take the exact same coding agent, you take the exact same coding agent, you take the exact same coding agent, you wrap it in a sandbox, and you just let wrap it in a sandbox, and you just let wrap it in a sandbox, and you just let it go run. It can build, it can test, it go run. It can build, it can test, it go run. It can build, it can test, and once it's done with its work, it can and once it's done with its work, it can and once it's done with its work, it can provide the code in the form of a PR. provide the code in the form of a PR. provide the code in the form of a PR. One thing that's really unique about One thing that's really unique about One thing that's really unique about engineers is folks are incredibly good engineers is folks are incredibly good engineers is folks are incredibly good at already paralyzing their work. at already paralyzing their work. at already paralyzing their work. It's very commonplace to launch 10 jobs It's very commonplace to launch 10 jobs It's very commonplace to launch 10 jobs and be comfortable with the fact that and be comfortable with the fact that and be comfortable with the fact that job seven might finish before job three. job seven might finish before job three. job seven might finish before job three. So, engineers are incredibly good at So, engineers are incredibly good at So, engineers are incredibly good at using these async agents.

using these async agents. using these async agents. Now, when we come to services, the Now, when we come to services, the Now, when we come to services, the equivalent of a synchronous agent, what equivalent of a synchronous agent, what equivalent of a synchronous agent, what we talked about a little bit earlier, we talked about a little bit earlier, we talked about a little bit earlier, it's a co-working agent. It's an agent it's a co-working agent. It's an agent it's a co-working agent. It's an agent that has deep context about your that has deep context about your that has deep context about your enterprise. It interacts potentially enterprise. It interacts potentially enterprise. It interacts potentially with MCPs, custom tools, custom with MCPs, custom tools, custom with MCPs, custom tools, custom integrations, uh and you can chat with integrations, uh and you can chat with integrations, uh and you can chat with it synchronously just like any of these it synchronously just like any of these it synchronously just like any of these other products. other products. other products. I think this is a frontier here in the I think this is a frontier here in the I think this is a frontier here in the bottom right. bottom right. bottom right. What does it mean to build an What does it mean to build an What does it mean to build an asynchronous agent for the services? asynchronous agent for the services? asynchronous agent for the services? What does it mean to paralyze work in What does it mean to paralyze work in What does it mean to paralyze work in industries where work is traditionally industries where work is traditionally industries where work is traditionally done in a very, very serial manner? done in a very, very serial manner? done in a very, very serial manner? This is where we spend a lot of time and This is where we spend a lot of time and This is where we spend a lot of time and this is what I wake up every morning this is what I wake up every morning this is what I wake up every morning really excited thinking about, you know, really excited thinking about, you know, really excited thinking about, you know, we've we've figured out what the async we've we've figured out what the async we've we've figured out what the async and forking mechanism for code is. You and forking mechanism for code is. You and forking mechanism for code is. You just spin up a bunch of sandboxes and do just spin up a bunch of sandboxes and do just spin up a bunch of sandboxes and do work. What does that look like for the work. What does that look like for the work. What does that look like for the rest of the world? So, here's a couple questions we think So, here's a couple questions we think about pretty seriously. One, you know, about pretty seriously. One, you know, about pretty seriously. One, you know, the models are trained on code, they the models are trained on code, they the models are trained on code, they want to write code, they're incredibly want to write code, they're incredibly want to write code, they're incredibly good at writing code. How do we leverage good at writing code. How do we leverage good at writing code. How do we leverage these coding agents for actual knowledge these coding agents for actual knowledge these coding agents for actual knowledge work? You You rather than wait for the work? You You rather than wait for the work? You You rather than wait for the models to catch up on doing services models to catch up on doing services models to catch up on doing services knowledge work, what if we just use that knowledge work, what if we just use that knowledge work, what if we just use that code knowledge and represent knowledge code knowledge and represent knowledge code knowledge and represent knowledge work as code?

work as code? work as code? Two, as I mentioned, engineers are used Two, as I mentioned, engineers are used Two, as I mentioned, engineers are used to paralyzing work. How do you paralyze to paralyzing work. How do you paralyze to paralyzing work. How do you paralyze work that's traditionally serial? You work that's traditionally serial? You work that's traditionally serial? You know, people clean out their inbox one know, people clean out their inbox one know, people clean out their inbox one email by one email, not 10 emails at email by one email, not 10 emails at email by one email, not 10 emails at once. once. once. And finally, how do you move up the And finally, how do you move up the And finally, how do you move up the ladder here both in terms of product and ladder here both in terms of product and ladder here both in terms of product and user enablement? user enablement? user enablement? What are the right form factors? And I'd What are the right form factors? And I'd What are the right form factors? And I'd argue this varies dramatically from argue this varies dramatically from argue this varies dramatically from industry to industry. Just because you industry to industry. Just because you industry to industry. Just because you have one way of launching an async agent have one way of launching an async agent have one way of launching an async agent for code, doesn't mean that same way is for code, doesn't mean that same way is for code, doesn't mean that same way is going to work for architecture or going to work for architecture or going to work for architecture or property management. property management. property management. The second point I want to cover today The second point I want to cover today The second point I want to cover today is leveraging real-world data. is leveraging real-world data. is leveraging real-world data. We all know this. Frontier models have We all know this. Frontier models have We all know this. Frontier models have learned from everything humanity has learned from everything humanity has learned from everything humanity has written down, written down, written down, but the most valuable tasks are not on but the most valuable tasks are not on but the most valuable tasks are not on the internet. the internet. the internet. How do you actually close the books when How do you actually close the books when How do you actually close the books when you're missing receipts? you're missing receipts? you're missing receipts? >> [snorts] >> [snorts] >> [snorts] >> How do you scope a building for >> How do you scope a building for >> How do you scope a building for construction in a blueprint, potentially construction in a blueprint, potentially construction in a blueprint, potentially collaboratively? collaboratively? collaboratively? How do you coordinate vendors for fixing How do you coordinate vendors for fixing How do you coordinate vendors for fixing a broken roof? a broken roof? a broken roof? All of this knowledge lives in people's All of this knowledge lives in people's All of this knowledge lives in people's heads, in 20-year-old software, uh in heads, in 20-year-old software, uh in heads, in 20-year-old software, uh in the way that one senior person on one of the way that one senior person on one of the way that one senior person on one of these teams just knows how to do it. How these teams just knows how to do it. How these teams just knows how to do it. How do you make this information explicit do you make this information explicit do you make this information explicit and create tasks that you can actually and create tasks that you can actually and create tasks that you can actually learn from?

learn from? learn from? So, we've constructed a little bit of a So, we've constructed a little bit of a So, we've constructed a little bit of a flywheel. We get our agents to flywheel. We get our agents to flywheel. We get our agents to collaborate with our employees to do collaborate with our employees to do collaborate with our employees to do real work. And this allows us to real work. And this allows us to real work. And this allows us to generate rich traces of data and generate rich traces of data and generate rich traces of data and information. Tool calls, the hiccups, information. Tool calls, the hiccups, information. Tool calls, the hiccups, the papercuts, everything that goes the papercuts, everything that goes the papercuts, everything that goes wrong with doing real work. wrong with doing real work. wrong with doing real work. This in turn allows us to build This in turn allows us to build This in turn allows us to build real-world evals. real-world evals. real-world evals. There is a ground truth here. In the There is a ground truth here. In the There is a ground truth here. In the case of the roofing example, the case of the roofing example, the case of the roofing example, the question is, did the roof get repaired? question is, did the roof get repaired? question is, did the roof get repaired? Did the books get closed? Did the books get closed? Did the books get closed? And this allows us to hill climb and And this allows us to hill climb and And this allows us to hill climb and build better agents, which leads to more build better agents, which leads to more build better agents, which leads to more and more impact. And what's really and more impact. And what's really and more impact. And what's really exciting is it ratchets up. Every week exciting is it ratchets up. Every week exciting is it ratchets up. Every week our hill climbing benchmarks our hill climbing benchmarks our hill climbing benchmarks become a regression test. So, our agents become a regression test. So, our agents become a regression test. So, our agents get better and better over time. get better and better over time. get better and better over time. Just to drive a little bit deeper here Just to drive a little bit deeper here Just to drive a little bit deeper here on the traces, there's three upshots of on the traces, there's three upshots of on the traces, there's three upshots of being able to collect these rich traces. being able to collect these rich traces. being able to collect these rich traces. One, we get to generate amazing evals One, we get to generate amazing evals One, we get to generate amazing evals that are built and scored automatically. that are built and scored automatically. that are built and scored automatically. Uh and we're able to gather both Uh and we're able to gather both Uh and we're able to gather both implicit and explicit feedback. Explicit implicit and explicit feedback. Explicit implicit and explicit feedback. Explicit feedback in the sense of thumbs ups and feedback in the sense of thumbs ups and feedback in the sense of thumbs ups and thumbs down, maybe people provide a note thumbs down, maybe people provide a note thumbs down, maybe people provide a note telling us whether this response was telling us whether this response was telling us whether this response was good or not. Uh and also implicit good or not. Uh and also implicit good or not. Uh and also implicit feedback. Right? Again, we have the feedback. Right? Again, we have the feedback. Right? Again, we have the ground truth. Maybe there's some data ground truth. Maybe there's some data ground truth. Maybe there's some data that the AI generated and there's a real that the AI generated and there's a real that the AI generated and there's a real diff between the data that the AI diff between the data that the AI diff between the data that the AI generated and what was ultimately generated and what was ultimately generated and what was ultimately submitted. That's rich information that submitted. That's rich information that submitted. That's rich information that almost no one else has.

almost no one else has. almost no one else has. Two, we've started post training models Two, we've started post training models Two, we've started post training models internally on internally on internally on all of the data that these businesses all of the data that these businesses all of the data that these businesses operate on and produce, generally operate on and produce, generally operate on and produce, generally speaking. speaking. speaking. This is all data that is completely out This is all data that is completely out This is all data that is completely out of distribution for most frontier labs. of distribution for most frontier labs. of distribution for most frontier labs. Think of the task I showed at the Think of the task I showed at the Think of the task I showed at the beginning. A lot of the models A lot of beginning. A lot of the models A lot of beginning. A lot of the models A lot of the frontier models today just can't do the frontier models today just can't do the frontier models today just can't do these tasks yet and we're trying to post these tasks yet and we're trying to post these tasks yet and we're trying to post train our own models internally to be train our own models internally to be train our own models internally to be able to do that on the rich source of able to do that on the rich source of able to do that on the rich source of data that we own. data that we own. data that we own. And then finally, the actual agents And then finally, the actual agents And then finally, the actual agents themselves. themselves. themselves. The real world is incredibly hairy and The real world is incredibly hairy and The real world is incredibly hairy and messy and you want customization per messy and you want customization per messy and you want customization per company. Every company does things very company. Every company does things very company. Every company does things very differently. Customization per user. The differently. Customization per user. The differently. Customization per user. The way each user does their work is very way each user does their work is very way each user does their work is very unique. And customization per client. unique. And customization per client. unique. And customization per client. The way you work with every client is The way you work with every client is The way you work with every client is different. It's a services business and different. It's a services business and different. It's a services business and you want to uphold those standards. you want to uphold those standards. you want to uphold those standards. I love this picture I love this picture I love this picture because it's the whole thing in a single because it's the whole thing in a single because it's the whole thing in a single image. Um the the way we usually talk image. Um the the way we usually talk image. Um the the way we usually talk about LLM tasks is the top panel. Right? about LLM tasks is the top panel. Right? about LLM tasks is the top panel. Right? You just It's It's a slope. You got a You just It's It's a slope. You got a You just It's It's a slope. You got a bike. And but there's clear sight to bike. And but there's clear sight to bike. And but there's clear sight to success.

success. success. The reality is most work is not like The reality is most work is not like The reality is most work is not like that. And And you and I both know that. that. And And you and I both know that. that. And And you and I both know that. Uh there are hills and ravines. Uh Uh there are hills and ravines. Uh Uh there are hills and ravines. Uh there's death by a thousand paper cuts. there's death by a thousand paper cuts. there's death by a thousand paper cuts. But But that's what real work looks But But that's what real work looks But But that's what real work looks like. That's the entire job. The like. That's the entire job. The like. That's the entire job. The exceptions are the job. exceptions are the job. exceptions are the job. That's That's the demo. That's That's the demo. That's That's the demo. That's the actual job. Now, on to the final thing I want to Now, on to the final thing I want to chat with you guys today is learning chat with you guys today is learning chat with you guys today is learning loops within the enterprise. loops within the enterprise. loops within the enterprise. I'd argue there's two hot trends I'd argue there's two hot trends I'd argue there's two hot trends everyone's talking about in 2026. One, everyone's talking about in 2026. One, everyone's talking about in 2026. One, it's continual learning. How do you make it's continual learning. How do you make it's continual learning. How do you make an agent better over time with feedback? an agent better over time with feedback? an agent better over time with feedback? I think there are plenty of sessions uh I think there are plenty of sessions uh I think there are plenty of sessions uh this week on how you can use continual this week on how you can use continual this week on how you can use continual learning, whether it's in the prompt or learning, whether it's in the prompt or learning, whether it's in the prompt or in the weights. in the weights. in the weights. And two, enablement. How do you get in And two, enablement. How do you get in And two, enablement. How do you get in these enterprises and actually get them these enterprises and actually get them these enterprises and actually get them to adopt and use AI? to adopt and use AI? to adopt and use AI? Traditionally speaking, Traditionally speaking, Traditionally speaking, these two initiatives are owned by two these two initiatives are owned by two these two initiatives are owned by two separate teams. Right? The continual separate teams. Right? The continual separate teams. Right? The continual learning is owned by your research team, learning is owned by your research team, learning is owned by your research team, your platform engineering team.

your platform engineering team. your platform engineering team. Enablement's owned by growth or Enablement's owned by growth or Enablement's owned by growth or deployment or customer experience. Uh deployment or customer experience. Uh deployment or customer experience. Uh usually pretty siloed, not much usually pretty siloed, not much usually pretty siloed, not much interaction between the two. interaction between the two. interaction between the two. We think these are part of the exact We think these are part of the exact We think these are part of the exact same loop. same loop. same loop. The agent only improves if people The agent only improves if people The agent only improves if people actually use it. actually use it. actually use it. And people only use the agent if it's And people only use the agent if it's And people only use the agent if it's worth adopting. worth adopting. worth adopting. So, here's a little graphic of a So, here's a little graphic of a So, here's a little graphic of a snowball. More usage drives continual snowball. More usage drives continual snowball. More usage drives continual learning, which drives a better agent, learning, which drives a better agent, learning, which drives a better agent, which drives more usage again. which drives more usage again. which drives more usage again. All this to say, there's still a really All this to say, there's still a really All this to say, there's still a really big elephant in the room. big elephant in the room. big elephant in the room. How do you get the initial usage? How do you get the initial usage? How do you get the initial usage? I think a lot of people, you know, will I think a lot of people, you know, will I think a lot of people, you know, will use Claude Code or or give it to their use Claude Code or or give it to their use Claude Code or or give it to their whole enterprise, expect folks to just whole enterprise, expect folks to just whole enterprise, expect folks to just start using it. start using it. start using it. Everyone assumes the usage just shows Everyone assumes the usage just shows Everyone assumes the usage just shows up. up. up. But as we all know, that's simply not But as we all know, that's simply not But as we all know, that's simply not the case. It never does. Right? Getting the case. It never does. Right? Getting the case. It never does. Right? Getting a 100-year-old firm to change its a 100-year-old firm to change its a 100-year-old firm to change its processes is hard. processes is hard. processes is hard. You could have the best AI coworker on You could have the best AI coworker on You could have the best AI coworker on the internet or on Earth. And if the the internet or on Earth. And if the the internet or on Earth. And if the people if the person who's closed the people if the person who's closed the people if the person who's closed the books for the last 20 years continues to books for the last 20 years continues to books for the last 20 years continues to do things the same way, do things the same way, do things the same way, nothing changes. Nothing happens.

nothing changes. Nothing happens. nothing changes. Nothing happens. So, what can you actually do about it? So, what can you actually do about it? So, what can you actually do about it? What you know, this this seems like What you know, this this seems like What you know, this this seems like incredibly hard. What what's the upshot? incredibly hard. What what's the upshot? incredibly hard. What what's the upshot? How do you actually get this stuff to How do you actually get this stuff to How do you actually get this stuff to work? Well, I think a lot about Jensen work? Well, I think a lot about Jensen work? Well, I think a lot about Jensen and how he dominated the market in his and how he dominated the market in his and how he dominated the market in his words with extreme hardware software words with extreme hardware software words with extreme hardware software co-design. Designing the chips and the co-design. Designing the chips and the co-design. Designing the chips and the software together as one system. software together as one system. software together as one system. We look at this through the lens of We look at this through the lens of We look at this through the lens of extreme software service co-design. How extreme software service co-design. How extreme software service co-design. How do you co-design our products with the do you co-design our products with the do you co-design our products with the people and the processes at our people and the processes at our people and the processes at our businesses? And I'd argue this is only businesses? And I'd argue this is only businesses? And I'd argue this is only possible from being within under the possible from being within under the possible from being within under the same roof. same roof. same roof. We need to meet the people within these We need to meet the people within these We need to meet the people within these companies both metaphorically, for companies both metaphorically, for companies both metaphorically, for example, bringing products to their example, bringing products to their example, bringing products to their systems so that the energy required for systems so that the energy required for systems so that the energy required for enablement is kept low, and also enablement is kept low, and also enablement is kept low, and also physically. Get on a plane, show up, say physically. Get on a plane, show up, say physically. Get on a plane, show up, say hi, learn what people actually do. hi, learn what people actually do. hi, learn what people actually do. You know, maybe you build a product You know, maybe you build a product You know, maybe you build a product that's natively embedded into Excel or that's natively embedded into Excel or that's natively embedded into Excel or into their ERP system, maybe their 3D into their ERP system, maybe their 3D into their ERP system, maybe their 3D design software, or or maybe even their design software, or or maybe even their design software, or or maybe even their Microsoft products like Outlook, Gmail, Microsoft products like Outlook, Gmail, Microsoft products like Outlook, Gmail, etc.

etc. etc. Or you show up in person. You do a lunch Or you show up in person. You do a lunch Or you show up in person. You do a lunch and learn with a bunch of folks at one and learn with a bunch of folks at one and learn with a bunch of folks at one of the companies. You go to their of the companies. You go to their of the companies. You go to their conferences and you create cotton candy conferences and you create cotton candy conferences and you create cotton candy and run a stand for them. You go and run a stand for them. You go and run a stand for them. You go mountain biking and ask them about all mountain biking and ask them about all mountain biking and ask them about all the difficulties that they have with the difficulties that they have with the difficulties that they have with their actual day-to-day jobs. their actual day-to-day jobs. their actual day-to-day jobs. Or you show up in person one-on-one or Or you show up in person one-on-one or Or you show up in person one-on-one or sometimes even two-on-one in this case sometimes even two-on-one in this case sometimes even two-on-one in this case and just show them how to use the tools and just show them how to use the tools and just show them how to use the tools and learn from the feedback because this and learn from the feedback because this and learn from the feedback because this is what the rest of the world really is what the rest of the world really is what the rest of the world really looks like. It's not like the folks in looks like. It's not like the folks in looks like. It's not like the folks in this room or in San Francisco. It's a this room or in San Francisco. It's a this room or in San Francisco. It's a lot more like this. You cannot co-design lot more like this. You cannot co-design lot more like this. You cannot co-design software with the services business over software with the services business over software with the services business over Zoom Zoom Zoom or over a support ticket. You you have or over a support ticket. You you have or over a support ticket. You you have to be there. You have to be in person. to be there. You have to be in person. to be there. You have to be in person. And I'd argue this is the part that And I'd argue this is the part that And I'd argue this is the part that actually makes it work. In order to get actually makes it work. In order to get actually makes it work. In order to get AI diffusion to work, you have to touch AI diffusion to work, you have to touch AI diffusion to work, you have to touch some grass. some grass. some grass. Thank you so much. I'll be around for Thank you so much. I'll be around for Thank you so much. I'll be around for the rest of day if there's anything I the rest of day if there's anything I the rest of day if there's anything I can help with. My email is up there. can help with. My email is up there. can help with. My email is up there. And yeah, thank you.